Hello @JonMerlevede,

The rationale behind the proposed changes looks great! I agree that users may not always require the object content when querying the table, so using HeadBucket to populate the column values seems like a good approach.

Could you kindly test these changes with larger objects and share the query execution times (with and without the changes in this PR)?

Here are the steps to measure the query timing:

1. Install the latest AWS plugin build (v0.147.0): steampipe plugin install [email protected]
2. Build the code changes in this PR from the JonMerlevede:feature/s3-head-object branch by running the command make.
3. Create two separate connections in your aws.spc file for the two different AWS plugin versions, like so:

   connection "aws_local_build" {
     plugin = "aws"
     regions = ["*"]
     profile = "default"
   }
   
   connection "aws_current" {
     plugin = "[email protected]"
     regions = ["*"]
     profile = "default"
   }

4. Run steampipe query.
5. Enable query timing by running .timing on.
6. Execute the following queries for both connections:
   • Local build: select key, version_id from aws_local_build.aws_s3_object where bucket_name = '<name of the bucket containing larger objects>'
   • AWS v0.147.0: select key, version_id from aws_current.aws_s3_object where bucket_name = '<name of the bucket containing larger objects>'

Please compare the query execution times and share the differences.

Thanks so much for giving this a try and sharing your observations!