Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: S3 compatibility #306

Open
wants to merge 7 commits into
base: releases/v19
Choose a base branch
from
Open

feat: S3 compatibility #306

wants to merge 7 commits into from

Conversation

mjdupont12
Copy link
Contributor

@mjdupont12 mjdupont12 commented Jan 24, 2025

set s3 config for drop-box
Works with the dropbox pr "s3 support" => bento-platform/bento_drop_box_service#63

Need to add some documentation.

But if you want to test =>
If s3 params are set, drop-box will use minio/s3.
if not, drop-box will use the filesystem.

The S3 params to set are =>
BENTO_DROP_BOX_S3_ENDPOINT=''
BENTO_DROP_BOX_S3_BUCKET=''
BENTO_DROP_BOX_S3_ACCESS_KEY=''
BENTO_DROP_BOX_S3_SECRET_KEY=''

@mjdupont12 mjdupont12 changed the base branch from main to releases/v18 January 24, 2025 19:35
@davidlougheed davidlougheed changed the base branch from releases/v18 to releases/v19 February 5, 2025 18:20
Copy link
Contributor

@v-rocheleau v-rocheleau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Just missing docs now.
I think we could have a general docs/object_storage.md document that covers how to configure Bento's services for S3 storage backends. Then we can link to this document from the other docs sections.

@v-rocheleau v-rocheleau marked this pull request as ready for review February 26, 2025 17:39
@v-rocheleau
Copy link
Contributor

v-rocheleau commented Feb 28, 2025

Started work to make WES work with Drop-Box and DRS over the network only for S3 compatibility. Updated the WES version to use the PR.

Hacky solution at the moment, but it works with Katsu workflows when Drop-Box is using S3.
Does not work with Gohan yet, as the Gohan service also relies on a volume mount to read DRS files (working on that next).

NOTE: For the WES PR to work with S3 drop-box, you need to add the view:drop_box grant to the WES client in authz. This allows WES to download files from drop-box

@v-rocheleau v-rocheleau changed the title Feat/drop box s3 feat: S3 compatibility Feb 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants