Lock mechanism for scratch_dir creation, small refactoring

[burnout87]

Use a lock and loop to attempt to acquire the lock to check and create the scratch dir, also with a small refactoring

[codecov]

Codecov Report

Attention: Patch coverage is 96.66667% with 1 line in your changes missing coverage. Please review.

Project coverage is 61.98%. Comparing base (2d00c76) to head (4919457).
Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
cdci_data_analysis/flask_app/dispatcher_query.py	96.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #713      +/-   ##
==========================================
+ Coverage   61.90%   61.98%   +0.07%     
==========================================
  Files          50       50              
  Lines        9096     9114      +18     
==========================================
+ Hits         5631     5649      +18     
  Misses       3465     3465              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[burnout87]

This approach has the downside of creating several empty lock files. After discussing with @dsavchenko about it, we agreed on extending the free-up-space to use the same logic to delete the lock files