Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

♻️Storage: Code refactoring prior to changes (🚨🚨) #7088

Open
wants to merge 160 commits into
base: master
Choose a base branch
from

Conversation

sanderegg
Copy link
Member

@sanderegg sanderegg commented Jan 24, 2025

What do these changes do?

Very noisy PR that refactors storage service from aiohttp- to fastapi-based service.

  • aiohttp --> fastapi, all endpoints were modified, also the long running tasks and the client to the long running tasks that needed a fastapi compatible version
  • aiopg --> asyncpg, this brought a lot of issues regarding the fact that asyncpg does not seemlessly convert between strings and postgres types (along these lines, had to revert to use non time aware types until the database is updated)
  • removed the API entrypoint for syncing the table and S3 since it is completely unused
  • cleanup, refactoring and re-structuring alongside our other fastapi-based services

Important note: @bisgaard-itis @GitHK @mrnicegyu11 @odeimaiz @matusdrobuliak66 storage cannot be run as several replicas as I could not find a way to get the IP of the server via call to request during uploads of files. That means this will need a subsequent PR that will keep the state of the upload in the DB or call into AWS S3 until the completion of the S3 multipart is done. There will be a follow up on that in 1 week.

Related issue/s

How to test

Dev-ops checklist

@sanderegg sanderegg added a:storage issue related to storage service t:maintenance Some planned maintenance work labels Jan 24, 2025
@sanderegg sanderegg added this to the Singularity milestone Jan 24, 2025
@sanderegg sanderegg self-assigned this Jan 24, 2025
Copy link

codecov bot commented Jan 24, 2025

Codecov Report

Attention: Patch coverage is 84.05963% with 139 lines in your changes missing coverage. Please review.

Project coverage is 87.48%. Comparing base (68b2497) to head (18f5f0a).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7088      +/-   ##
==========================================
- Coverage   87.70%   87.48%   -0.23%     
==========================================
  Files        1635     1638       +3     
  Lines       63892    63907      +15     
  Branches     1173     1087      -86     
==========================================
- Hits        56037    55906     -131     
- Misses       7545     7690     +145     
- Partials      310      311       +1     
Flag Coverage Δ
integrationtests 64.64% <72.22%> (-0.03%) ⬇️
unittests 85.77% <83.94%> (-1.03%) ⬇️
Components Coverage Δ
api 76.84% <ø> (ø)
pkg_aws_library 94.02% <100.00%> (ø)
pkg_dask_task_models_library 97.09% <ø> (ø)
pkg_models_library 91.52% <100.00%> (+0.02%) ⬆️
pkg_notifications_library 84.57% <ø> (ø)
pkg_postgres_database 88.39% <ø> (ø)
pkg_service_integration 70.18% <ø> (ø)
pkg_service_library 73.19% <43.62%> (-1.21%) ⬇️
pkg_settings_library 90.71% <ø> (ø)
pkg_simcore_sdk 85.51% <100.00%> (+<0.01%) ⬆️
agent 96.46% <ø> (ø)
api_server 90.55% <ø> (ø)
autoscaling 96.08% <ø> (ø)
catalog 90.35% <100.00%> (+0.02%) ⬆️
clusters_keeper 99.24% <ø> (ø)
dask_sidecar 91.26% <ø> (ø)
datcore_adapter 93.19% <ø> (ø)
director 76.59% <100.00%> (-0.43%) ⬇️
director_v2 91.27% <ø> (+0.01%) ⬆️
dynamic_scheduler 97.21% <ø> (ø)
dynamic_sidecar 89.75% <ø> (ø)
efs_guardian 90.25% <ø> (ø)
invitations 93.28% <ø> (ø)
osparc_gateway_server ∅ <ø> (∅)
payments 92.66% <ø> (ø)
resource_usage_tracker 89.01% <ø> (-0.11%) ⬇️
storage 86.27% <91.89%> (-3.30%) ⬇️
webclient ∅ <ø> (∅)
webserver 86.20% <100.00%> (-0.02%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 68b2497...18f5f0a. Read the comment docs.

@sanderegg sanderegg force-pushed the storage/refactoring branch 6 times, most recently from 2eaa872 to dd968f9 Compare January 29, 2025 21:59
@sanderegg sanderegg force-pushed the storage/refactoring branch 2 times, most recently from 65be5bc to 2c7fcea Compare January 30, 2025 11:19
@sanderegg sanderegg marked this pull request as ready for review January 30, 2025 11:19
@sanderegg sanderegg changed the title ♻️Storage: Code refactoring prior to changes ♻️Storage: Code refactoring prior to changes (🚨🚨) Jan 30, 2025
"/locations/{location_id}/files/{file_id:path}",
response_model=Envelope[FileUploadResponseV1] | Envelope[FileUploadSchema],
)
async def upload_file(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe this should also have cancel on disconnect decorator?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point. will add a middleware actually.

Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, just some more comments

services/storage/tests/fixtures/datcore_adapter.py Outdated Show resolved Hide resolved
@@ -56,10 +55,12 @@ def location_name() -> str:
return SimcoreS3DataManager.get_location_name()


@pytest.mark.skip(reason="legacy service")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: could you ellaborate more why this is disabled, I did not quite get the reason. Maybe it would be helpful to have a note here for the future.
I find these short reasons not that suseful in skipped tests

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually I will re-enable it. I need to find a way to do this as this uses the aiohttp client.
I will probably have to start the real storage in order to run that

services/storage/tests/conftest.py Outdated Show resolved Hide resolved
@sanderegg sanderegg force-pushed the storage/refactoring branch from 182cdc6 to 0c07e27 Compare January 30, 2025 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:storage issue related to storage service t:maintenance Some planned maintenance work
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix openapi specs generation of storage maintenance: long running tasks for FastAPI-based app needs upgrade
7 participants