URL encoding of bucket name - change in 1.12.460

[debora-ito]

On 05/01/2023, Java SDK version 1.12.460 released a change to the S3 Client to perform URL encoding on the bucket name. This may break customers who currently rely on the behavior of special characters. For example, in a getObject call, if you have the key prefix appended to the bucket name like s3Client.getObject("bucket/prefix", "key"), the URL path of the request sent to S3 is now GET /bucket%2Fprefix/key instead of GET /bucket/prefix/key.

This issue can manifest itself as a 403 SignatureDoesNotMatch error:

com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch;)

How to fix

If you are seeing 403 SignatureDoesNotMatch errors in calls to S3 after upgrading the SDK version to 1.12.460 or later, check your request to make sure that special characters such as "/" are not part of the bucket name.

Note that a key prefix should be part of the key name, and not part of the bucket name:

Don't use: s3Client.getObject("bucket/prefix", "key")
Instead, use: s3Client.getObject("bucket", "prefix/key")

[rhowe]

Since

Bucket names can consist only of lowercase letters, numbers, dots (.), and hyphens (-).

https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html

How about having the SDK apply that check to bucket name parameters and raise a descriptive error?

[debora-ito]

SDKs in general don't make client-side validations. This is to prevent forcing customers to upgrade the SDK version in case the service decides to change the validation in the future.

[prabu-v]

@debora-ito What happens, if the prefix has "/" and is appended with bucket earlier like "bucketName/custId/payments"? How to fix the same after upgrading the sdk version to 1.12.538? I can't able to include this "/custId/banners" in the key as well. I'm getting the access denied error when downloading the uploaded URL object.

[debora-ito]

If you need to upgrade to 1.12.460 or later, you can't have "/custId/payments" as part of the bucket name, you need to include "custId/payments" to the key name. There's no workaround for this.

[hung-tran-aiq]

Is there an equivalent AWS SDK v2 minor version we should be watching out for?

[debora-ito]

Java SDK v2 has always encoded the bucket name.

[MasterDDT]

Btw since this is an API behavior change, shouldnt the minor version be bumped 1.12 -> 1.13? I get this is a bug, but its very old behavior, and old behavior + many years = new supported behavior 😅 .

So fixing the bug should have involved a new minor version and proper changelog notes.

[Thevarun123]

Hi, can the s3client.putObject(putObjectRequest); will work with the version sdk version 1.12.460 or later. The putObjectRequest goes like : PutObjectRequest putObjectRequest = new PutObjectRequest(directory, fileName, inputStream, metadata).withCannedAcl(
CannedAccessControlList.BucketOwnerFullControl);
Please tell.

[Thevarun123]

@debora-ito any idea

[cristiadu]

@Thevarun123 technically (saying like that because I'm still testing) you can fix the same way as the getObject, just do:

String[] directorySegments = directory.split("/")
String rootPath = directorySegments[0]
String remainingPath = directorySegments.size() > 1 ? directorySegments[1..-1] + "/" : ""
PutObjectRequest putObjectRequest = new PutObjectRequest(rootPath, remainingPath + fileName, inputStream, metadata).withCannedAcl(
CannedAccessControlList.BucketOwnerFullControl);