AWS: Add iceberg-aws-s3-bundle with S3-only AWS clients - #17531
Open
n-dlms wants to merge 3 commits into
Open
Conversation
Add a slim shadow bundle that ships only the S3-related AWS SDK dependencies (s3, auth, http clients, crt, s3 access grants, analytics accelerator), so S3FileIO users no longer need to depend on the full iceberg-aws-bundle with its Glue, DynamoDB, KMS, LakeFormation and SSO clients. Measured on a local build, the S3-only bundle is 48MB vs 65MB for the full bundle. The full iceberg-aws-bundle is left unchanged for Glue catalog and DynamoDB lock users. Add the new distribution to the CVE scan and the AWS labeler glob, and document the S3-only option in the AWS docs. Signed-off-by: Ntokozo Dlamini <ntokozo.dlamini.xyz@gmail.com> Generated-by: Claude Code
Regenerate the runtime dependency baseline after merging the awssdk bom bump (apache#17488); CI resolves 2.49.3 while the initial baseline was generated against 2.47.5. Signed-off-by: Ntokozo Dlamini <ntokozo.dlamini.xyz@gmail.com> Generated-by: Claude Code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Issue #7516 asks that users who only need
S3FileIOnot be forced to ship the full set of AWS SDK clients (Glue, DynamoDB, KMS, etc.). The factory-level split already exists (S3FileIOAwsClientFactory+s3.client-factory-impl, #7590), but the artifact users actually deploy —iceberg-aws-bundle— still shades every client, which is the ~450MB "just dump the bundle jar" problem the issue calls out.What this PR does
Adds a new shadow bundle module
iceberg-aws-s3-bundlethat contains only the S3-path AWS dependencies:s3,auth,apache-client,url-connection-clientcrt-core,http-auth-aws-crt(async/CRT path)s3accessgrants,analyticsaccelerator(S3FileIO features)and excludes the catalog/lock/KMS clients:
glue,dynamodb,kms,lakeformation,sso.The existing
iceberg-aws-bundleis left unchanged for Glue catalog, DynamoDB lock, and KMS users.Measured locally:
iceberg-aws-s3-bundleshadow jar is 48MB vs 65MB for the full bundle (~26% smaller), while:iceberg-aws:test(S3FileIO, S3InputStream, access grants, etc.) passes unchanged.Changes
aws-s3-bundle/— new module (build.gradle mirroringaws-bundlewith the slim dep set, plus LICENSE/NOTICE and the generatedruntime-deps.txtbaseline)settings.gradle— register the module.github/workflows/cve-scan.yml— scan the new distribution.github/labeler.yml— AWS label covers the new moduledocs/docs/aws.md— document the S3-only option for Spark with a REST-catalog exampleCompatibility
Fully additive — no existing artifact or API changes. Users who need Glue/DynamoDB/KMS continue using
iceberg-aws-bundle.Closes #7516
AI Disclosure