Skip to content

AWS v2 SDK for S3 #11360

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 26 commits into
base: develop
Choose a base branch
from

Conversation

qqmyers
Copy link
Member

@qqmyers qqmyers commented Mar 20, 2025

What this PR does / why we need it: The v1 AWS SDK Dataverse uses for S3 is in maintenance mode and will reach it's end of life in Dec. 2025. This PR updates Dataverse to use v2.x of the SDK.

Which issue(s) this PR closes:

Special notes for your reviewer:

  • From initial testing, it looks like the older localstack used in testing does not support the new API. The PR updates to localstack/localstack:4.2.0 in the docker-compose-dev file.

  • Sword uses old versions of apache abdera. I've added exclusions to avoid failures with S3. Updating Sword might be a useful thing to do at some point.

Suggestions on how to test this:

  • General S3 testing with direct upload and download on and off, with files below and above the min-part-size set for the store (suggest this get set to just over the 5*1024**2 (~5MB) allows by AWS for testing).
  • ZipDownloader builds but I am not sure how to test it. As it uses S3, it should be included in testing.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?: inc.

Additional documentation:

@coveralls
Copy link

coveralls commented Mar 21, 2025

Coverage Status

coverage: 23.121% (+0.1%) from 23.007%
when pulling 5c655e2 on GlobalDataverseCommunityConsortium:AWSv2
into cd48216 on IQSS:develop.

@qqmyers qqmyers marked this pull request as ready for review April 5, 2025 20:59
@qqmyers qqmyers moved this to Ready for Triage in IQSS Dataverse Project Apr 5, 2025
@qqmyers qqmyers added this to the 6.7 milestone Apr 5, 2025
@qqmyers qqmyers added the Size: 10 A percentage of a sprint. 7 hours. label Apr 5, 2025
@ofahimIQSS ofahimIQSS moved this from Ready for Triage to Ready for Review ⏩ in IQSS Dataverse Project Apr 8, 2025
@cmbz cmbz added the FY25 Sprint 20 FY25 Sprint 20 (2025-03-26 - 2025-04-09) label Apr 9, 2025
@jeromeroucou
Copy link
Contributor

jeromeroucou commented Apr 9, 2025

I have done some quick test on our Rados Ceph storage (S3 compatible) and I can upload file only without direct upload. With direct upload, I have a CORS error on my browser. In addition, on (IQSS) develop branch, I don't have this error with same configuration. I'll look at the JS code another day and try to understand it.

@cmbz cmbz added the FY25 Sprint 21 FY25 Sprint 21 (2025-04-09 - 2025-04-23) label Apr 9, 2025
@jeromeroucou
Copy link
Contributor

custom-endpoint-url parameter seems to be ignored in direct-upload: the URL who valid CORS is https://bucket.s3.region.amazonaws.com/10.5072/FK2/YZ4INU/196204baf01-d80915a27bbb?X-Amz-Algorithm=AWS4-HMAC-SHA256[...]. If I find some time, I'll try to understand why

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FY25 Sprint 20 FY25 Sprint 20 (2025-03-26 - 2025-04-09) FY25 Sprint 21 FY25 Sprint 21 (2025-04-09 - 2025-04-23) Size: 10 A percentage of a sprint. 7 hours.
Projects
Status: Ready for Review ⏩
Development

Successfully merging this pull request may close these issues.

AWS S3 SDK end of support on December 31, 2025
4 participants