ACK S3 controller - InvalidArgument: Argument format not recognized

[itaiatu]

Describe the bug
ACK S3 controller v1.0.26 throws InvalidArgument: Argument format not recognized on s3 buckets that were created or adopted with ACK S3 controller v1.0.18.

Steps to reproduce
We have this s3 bucket on the cluster that was created with v1.0.18 and is now consumed and reconciled by v1.0.26:

apiVersion: s3.services.k8s.aws/v1alpha1
kind: Bucket
metadata:
  annotations:
    services.k8s.aws/region: us-east-1
  creationTimestamp: "2025-02-19T09:41:52Z"
  finalizers:
  - finalizers.s3.services.k8s.aws/Bucket
  generation: 1
  name: astef-test
  namespace: sbx-clusters
spec:
  encryption:
    rules:
    - applyServerSideEncryptionByDefault:
        sseAlgorithm: AES256
      bucketKeyEnabled: false
  grantFullControl: id=4215a77b61c01ab052e87028ad0fc004204e35bd4127e667e7166ead0cb34a0f
  grantRead: ""
  grantReadACP: ""
  grantWrite: ""
  grantWriteACP: ""
  lifecycle:
    rules:
    - abortIncompleteMultipartUpload:
        daysAfterInitiation: 7
      id: delete expired multi part uploads
      prefix: ""
      status: Enabled
  logging: {}
  name: astef-test
  notification: {}
  ownershipControls:
    rules:
    - objectOwnership: BucketOwnerEnforced
  publicAccessBlock:
    blockPublicACLs: true
    blockPublicPolicy: true
    ignorePublicACLs: true
    restrictPublicBuckets: true
  requestPayment:
    payer: BucketOwner
status:
  ackResourceMetadata:
    ownerAccountID: "258057316678"
    region: us-east-1
  conditions:
  - message: 'api error InvalidArgument: Argument format not recognized'
    status: "True"
    type: ACK.Recoverable
  - lastTransitionTime: "2025-02-25T10:26:01Z"
    message: Unable to determine if desired resource state matches latest observed
      state
    reason: 'operation error S3: CreateBucket, https response error StatusCode: 400,
      RequestID: KX82M956G43Q6Z0Y, HostID: wlzZimwDTxpFypM+1CIfUBnfh5io8JiQg1fuQXmmwEAxVteP7ByET1WN2h298kA9G+Q9xTA01+4=,
      api error InvalidArgument: Argument format not recognized'
    status: Unknown
    type: ACK.ResourceSynced

As we can see in the status of the object, AWS throws InvalidArgument: Argument format not recognized.

In the CloudTrail, we have this

    "eventTime": "2025-02-25T13:38:32Z",
    "eventSource": "s3.amazonaws.com",
    "eventName": "CreateBucket",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "35.162.143.192",
    "userAgent": "[aws-controllers-k8s/s3.services.k8s.aws-1.0.26 (GitCommit/6478c6e5000377cf6393d5d4dddfab3ee1e45134; BuildDate/2025-02-21T23:33; CRDKind/Bucket; CRDVersion/v1alpha1) aws-sdk-go-v2/1.34.0 ua/2.1 os/linux lang/go#1.24.0 md/GOOS#linux md/GOARCH#amd64 api/s3#1.74.1 m/E]",
    "errorCode": "InvalidArgument",
    "errorMessage": "Argument format not recognized",
    "requestParameters": {
        "bucketName": "astef-test",
        "Host": "astef-test.s3.us-east-1.amazonaws.com",
        "accessControlList": {
            "x-amz-grant-write": "",
            "x-amz-grant-read": "",
            "x-amz-grant-full-control": "id=4215a77b61c01ab052e87028ad0fc004204e35bd4127e667e7166ead0cb34a0f",
            "x-amz-grant-read-acp": "",
            "x-amz-grant-write-acp": ""
        }
    },

---

Maybe this is due to the migration to aws-sdk-go-v2 and some parameters are not used anymore.

---

Expected outcome
Correctly consume the resources created with older ACK controller versions.

Environment

• Kubernetes version 1.29
• Using EKS (yes/no), if so version? 1.29
• AWS service targeted (S3, RDS, etc.) S3

[michaelhtm]

Hey @itaiatu, was this resolved?

[itaiatu]

Hi @michaelhtm, I closed it because it wasn't reproducible. But if I recall correctly, everything started from this error.

apiVersion: s3.services.k8s.aws/v1alpha1
kind: Bucket
metadata:
  annotations:
    services.k8s.aws/region: us-east-1
  creationTimestamp: "2025-02-19T09:45:00Z"
  finalizers:
  - finalizers.s3.services.k8s.aws/Bucket
  generation: 1
  name: astef-sbx-va6-k8s-data
  namespace: sbx-clusters
spec:
  accelerate: {}
  acl: private|bucket-owner-read|bucket-owner-full-control
  cors: {}
  encryption:
    rules:
    - applyServerSideEncryptionByDefault:
        sseAlgorithm: AES256
      bucketKeyEnabled: false
  grantFullControl: id=4215a77b61c01ab052e87028ad0fc004204e35bd4127e667e7166ead0cb34a0f
  grantRead: ""
  grantReadACP: ""
  grantWrite: ""
  grantWriteACP: ""
  lifecycle:
    rules:
    - expiration:
        days: 90
      id: backup-deletion
      noncurrentVersionExpiration:
        noncurrentDays: 90
      prefix: backups/
      status: Enabled
    - abortIncompleteMultipartUpload:
        daysAfterInitiation: 7
      id: delete expired multi part uploads
      prefix: ""
      status: Enabled
  logging: {}
  name: astef-sbx-va6-k8s-data
  notification: {}
  ownershipControls:
    rules:
    - objectOwnership: BucketOwnerEnforced
  publicAccessBlock:
    blockPublicACLs: true
    blockPublicPolicy: true
    ignorePublicACLs: true
    restrictPublicBuckets: true
  requestPayment:
    payer: BucketOwner
  versioning:
    status: Enabled
  website: {}
status:
  ackResourceMetadata:
    ownerAccountID: "258057316678"
    region: us-east-1
  conditions:
  - message: 'api error InvalidRequest: Specifying both Canned ACLs and Header Grants
      is not allowed'
    status: "True"
    type: ACK.Recoverable
  - lastTransitionTime: "2025-02-25T10:42:43Z"
    message: Unable to determine if desired resource state matches latest observed
      state
    reason: 'operation error S3: CreateBucket, https response error StatusCode: 400,
      RequestID: NDYEG65D1NN0GS2T, HostID: /NqTyHhTK/VSNXxbIap+C6sbh5whx77nz/bujaJ26OneE8j/fnnCKwbMrr3gGsaTcNCWkVcRvsw=,
      api error InvalidRequest: Specifying both Canned ACLs and Header Grants is not
      allowed'
    status: Unknown
    type: ACK.ResourceSynced

This was a bucket created with created with ACK S3 v1.0.18 and is now consumed and reconciled by v1.0.26.

The interesting part is in the response message from AWS:

api error InvalidRequest: Specifying both Canned ACLs and Header Grants is not allowed

[michaelhtm]

seems like the removal of empty fields we introduced could be messing with the update..
Is this still an issue? we can try rolling back and make a minor release instead of a patch if needed too
what do you think @a-hilaly @rushmash91

[rushmash91]

@michaelhtm are we talking about changes in the release v1.0.21?
@itaiatu are you still encountering the issue/reproducible now?

[itaiatu]

I think, for the moment, let's keep the things as they are, because we are not seeing the issue anymore and we can't replicate it.
If we will get back to this problem, we will definitely reopen this issue.
cc: @michaelhtm @rushmash91