✨ Add a Distributed Task Queue (using Celery)

[giancarloromeo]

What do these changes do?

Introduce a Distributed Task Queue (using Celery). On the first iteration it will be used by Storage service for file compression.
The Storage now works on 2 modes: STANDARD and WORKER.

In the STANDARD mode, the service uses Celery to submit tasks and get progress/results.
In the WORKER mode, it is executed as Celery worker (with FastAPI intitialized) and it is responsible for tasks execution.

Related issue/s

• UI/UX for Data Handling (Feb.) osparc-issues#1635
• ✨ Add storage dummy rpc endpoints for zipping #7202

How to test

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)

[codecov]

Codecov Report

Attention: Patch coverage is 40.08621% with 139 lines in your changes missing coverage. Please review.

Project coverage is 66.79%. Comparing base (731dd9a) to head (3c8eb4c).

❗ There is a different number of reports uploaded between BASE (731dd9a) and HEAD (3c8eb4c). Click for more details.

HEAD has 30 uploads less than BASE

Flag	BASE (731dd9a)	HEAD (3c8eb4c)
unittests	32	2

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #7214       +/-   ##
===========================================
- Coverage   87.17%   66.79%   -20.39%     
===========================================
  Files        1680      725      -955     
  Lines       65053    34181    -30872     
  Branches     1106      177      -929     
===========================================
- Hits        56709    22830    -33879     
- Misses       8030    11289     +3259     
+ Partials      314       62      -252     

Flag	Coverage Δ		*Carryforward flag
integrationtests	65.45% <ø> (+0.01%)	⬆️	Carriedforward from 8a81d22
unittests	57.91% <40.08%> (-28.27%)	⬇️

*This pull request uses carry forward flags. Click here to find out more.

Components	Coverage Δ
api	∅ <ø> (∅)
pkg_aws_library	∅ <ø> (∅)
pkg_dask_task_models_library	∅ <ø> (∅)
pkg_models_library	∅ <ø> (∅)
pkg_notifications_library	∅ <ø> (∅)
pkg_postgres_database	∅ <ø> (∅)
pkg_service_integration	∅ <ø> (∅)
pkg_service_library	∅ <ø> (∅)
pkg_settings_library	∅ <ø> (∅)
pkg_simcore_sdk	77.25% <ø> (-8.21%)	⬇️
agent	96.46% <ø> (ø)
api_server	∅ <ø> (∅)
autoscaling	∅ <ø> (∅)
catalog	∅ <ø> (∅)
clusters_keeper	∅ <ø> (∅)
dask_sidecar	∅ <ø> (∅)
datcore_adapter	∅ <ø> (∅)
director	∅ <ø> (∅)
director_v2	78.57% <ø> (-12.69%)	⬇️
dynamic_scheduler	∅ <ø> (∅)
dynamic_sidecar	89.74% <ø> (ø)
efs_guardian	∅ <ø> (∅)
invitations	∅ <ø> (∅)
osparc_gateway_server	∅ <ø> (∅)
payments	∅ <ø> (∅)
resource_usage_tracker	∅ <ø> (∅)
storage	47.41% <40.08%> (-39.35%)	⬇️
webclient	∅ <ø> (∅)
webserver	59.16% <ø> (-26.00%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 731dd9a...3c8eb4c. Read the comment docs.

[sonarqubecloud]

Quality Gate failed

Failed conditions
1 Security Hotspot

See analysis details on SonarQube Cloud