Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle the "not found" error code in the getBucketTagging and getBucketWebsite functions Closes #2334 #2335

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ParthaI
Copy link
Contributor

@ParthaI ParthaI commented Nov 7, 2024

Integration test logs

Logs
Add passing integration test logs here

Example query results

Results

Breaks:

> select * from aws_s3_bucket

Error: aws: operation error S3: GetBucketWebsite, https response error StatusCode: 404, RequestID: 5F42KQGXEHF4XCKR, HostID: RjigpSaozYqUmuu33Jlgl1GdLxm811ewXfw7TvnyMBqdDCplwqJ+Peu8sr4j8iHGCoSiCEZ5gGo=, api error NoSuchWebsiteConfiguration: The specified bucket does not have a website configuration (SQLSTATE HV000)

+------+-----+---------------+-------------------------+--------------------+-----------------------+-------------------+---------------------+--------------------+-------------------------+----------------------->
| name | arn | creation_date | bucket_policy_is_public | versioning_enabled | versioning_mfa_delete | block_public_acls | block_public_policy | ignore_public_acls | restrict_public_buckets | event_notification_con>
+------+-----+---------------+-------------------------+--------------------+-----------------------+-------------------+---------------------+--------------------+-------------------------+----------------------->
+------+-----+---------------+-------------------------+--------------------+-----------------------+-------------------+---------------------+--------------------+-------------------------+----------------------->

Error: aws: operation error S3: GetBucketTagging, https response error StatusCode: 404, RequestID: 3RJ6DYJGY806SNDJ, HostID: 5As1rq9QSVXOG9ng+dteO9h0x168M4kG1rpW8qVMhCJr6nm4xInUFNC0bwQZSNCdjXS+zR+ks7I=, api error NoSuchTagSet: The TagSet does not exist (SQLSTATE HV000)

+------+-----+---------------+-------------------------+--------------------+-----------------------+-------------------+---------------------+--------------------+-------------------------+----------------------->
| name | arn | creation_date | bucket_policy_is_public | versioning_enabled | versioning_mfa_delete | block_public_acls | block_public_policy | ignore_public_acls | restrict_public_buckets | event_notification_con>
+------+-----+---------------+-------------------------+--------------------+-----------------------+-------------------+---------------------+--------------------+-------------------------+----------------------->
+------+-----+---------------+-------------------------+--------------------+-----------------------+-------------------+---------------------+--------------------+-------------------------+----------------------->

Fixed:

> select * from aws_s3_bucket

+-----------------------------------------------------------------+------------------------------------------------------------------------------+---------------------------+-------------------------+------------->
| name                                                            | arn                                                                          | creation_date             | bucket_policy_is_public | versioning_e>
+-----------------------------------------------------------------+------------------------------------------------------------------------------+---------------------------+-------------------------+------------->
| flowpipe-test-8a6db914-9a98-f789-e9d2-d13df22528ee              | arn:aws:s3:::flowpipe-test-8a6db914-9a98-f789-e9d2-d13df22528ee              | 2024-10-09T13:02:36+07:00 | false                   | false       >
| aws-security-data-lake-us-east-1-qeydzsin6buhruyudnvt7gnkpmxck3 | arn:aws:s3:::aws-security-data-lake-us-east-1-qeydzsin6buhruyudnvt7gnkpmxck3 | 2022-12-01T17:06:58+07:00 | false                   | true        >
| aws-cloudtrail-logs-xxxxxxxxxxxx-5edd7283                       | arn:aws:s3:::aws-cloudtrail-logs-xxxxxxxxxxxx-5edd7283                       | 2023-06-22T19:39:07+07:00 | false                   | false       >
| aws-security-data-lake-ap-southeast-2-4i3pnujz7bpnx72x6oheacrcx | arn:aws:s3:::aws-security-data-lake-ap-southeast-2-4i3pnujz7bpnx72x6oheacrcx | 2022-12-01T17:07:29+07:00 | false                   | false       >
| test-s3-bucket-2600a5ba-da1e-d998-32e4-d20fa84662b1             | arn:aws:s3:::test-s3-bucket-2600a5ba-da1e-d998-32e4-d20fa84662b1             | 2024-10-01T23:24:56+07:00 | false                   | false       >
| elasticbeanstalk-us-east-1-xxxxxxxxxxxx                         | arn:aws:s3:::elasticbeanstalk-us-east-1-xxxxxxxxxxxx                         | 2021-08-24T17:09:29+07:00 | false                   | false       >
| test-cloudtrail-s3-bucket-f0117bbb-370d-f1cf-9997-5f56625d347b  | arn:aws:s3:::test-cloudtrail-s3-bucket-f0117bbb-370d-f1cf-9997-5f56625d347b  | 2024-10-01T23:48:39+07:00 | false                   | false       >
| fp-2024-05-23-10-19-02                                          | arn:aws:s3:::fp-2024-05-23-10-19-02                                          | 2024-05-23T17:19:06+07:00 | false                   | false       >
| flowpipe-test-64c48ea5-e9f6-77e6-ae11-0694e9c962bf              | arn:aws:s3:::flowpipe-test-64c48ea5-e9f6-77e6-ae11-0694e9c962bf              | 2024-10-09T13:00:54+07:00 | false                   | false       >
| test-cloudtrail-s3-bucket-3d364669-8ca1-50d5-9e1b-db8ba3fd4289  | arn:aws:s3:::test-cloudtrail-s3-bucket-3d364669-8ca1-50d5-9e1b-db8ba3fd4289  | 2024-10-02T04:25:57+07:00 | false                   | false       >
| aws-security-data-lake-ap-northeast-1-yd5aigzmgod7uadzgou2d2otb | arn:aws:s3:::aws-security-data-lake-ap-northeast-1-yd5aigzmgod7uadzgou2d2otb | 2022-12-01T17:07:10+07:00 | false                   | false       >
| sp-flow-s3bucket-logsink                                        | arn:aws:s3:::sp-flow-s3bucket-logsink                                        | 2022-07-27T19:53:02+07:00 

Comment on lines +31 to +34
// Using IgnoreConfig in the Hydrate function has been observed to increase query execution time significantly.
// Therefore, we have opted not to use IgnoreConfig in Hydrate calls to maintain optimal performance.
// Example: For the query "select * from aws_s3_bucket where region = 'us-east-2'",
// using IgnoreConfig results in a slower execution time of 87.2 seconds, while handling errors manually reduces the time to 25.0 seconds.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a very important notice, thanks!

Are there any solutions to improve this being considered?

@pdecat
Copy link
Contributor

pdecat commented Nov 7, 2024

Can confirm this resolves the NoSuchTagSet: The TagSet does not exist error I was facing with v1.2.0 of the plugin.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Error getting S3 Buckets with No Tags
2 participants