Inconsistent EC2 Instance Listing Across AWS Accounts

[aswin-kevin]

Issue: Inconsistent EC2 Instance Listing Across AWS Accounts

I am using the AWS plugin for Steampipe, directly attached to a PostgreSQL database, and passing three temporary AWS credentials. I have multiple AWS accounts, each with the same IAM role attached via ARN, granting identical permissions.

After generating temporary credentials, I passed them to the PostgreSQL database using the following commands. I am able to successfully list S3 buckets and EC2 instances across most accounts, except for one specific AWS account, where I encounter the following error:

---

Commands to Attach Credentials to PostgreSQL:

CREATE EXTENSION IF NOT EXISTS steampipe_postgres_aws;
CREATE SERVER steampipe_aws FOREIGN DATA WRAPPER steampipe_postgres_aws OPTIONS (
  config 'regions = ["*"]',
  ignore_error_codes = ["AccessDenied", "AccessDeniedException", "NotAuthorized", "UnauthorizedOperation", "UnrecognizedClientException", "AuthorizationError"],
  access_key = "$AWS_ACCESS_KEY_ID",
  secret_key = "$AWS_SECRET_ACCESS_KEY",
  session_token = "$AWS_SESSION_TOKEN"'
);
CREATE SCHEMA aws;
CREATE EXTENSION ltree;
IMPORT FOREIGN SCHEMA aws FROM SERVER steampipe_aws INTO aws;

---

Error Message:

[ERROR] 1526178013893: aws_ec2_instance.listEc2Instance: api_error="operation error EC2: DescribeInstances, https response error StatusCode: 401, RequestID: e32bnb570-9645-43df-1d8h-c0b394b98855, api error AuthFailure: AWS was not able to validate the provided access credentials"

---

Details:

• This issue only occurs with one AWS account, despite all the ARNs having the same role attached, with full permissions to list EC2 instances.
• I have ruled out the possibility of expired credentials, as the error occurs within 5 minutes of generating the credentials, and they are valid for 1 hour.
• I also tried explicitly setting the region to us-west-2 and removing the ignore_error_codes option, but the issue persists.

To ensure the credentials are valid and have sufficient permissions, I created my own Steampipe plugin using the same version of the AWS SDK as the steampipe-plugin-aws. Interestingly, using the same credentials with my custom plugin, I was able to list EC2 instances from all regions, but when using steampipe-plugin-aws, the error occurs.

Here is a link to my custom Steampipe plugin for listing EC2 instances: https://github.com/aswin-kevin/steampipe-plugin-quark

---

AWS Plugin version:
v0.145.0

Could you please provide guidance or help in resolving this inconsistency?

[aswin-kevin]

Hi @ParthaI , just checking in on the status of this issue. Any updates or an ETA would be greatly appreciated, thanks!

[ParthaI]

Apologies for the delay, @aswin-kevin. I haven’t had a chance to take a look at it yet, but I will update you as soon as I begin working on it. Thanks for your patience!

[aswin-kevin]

Thanks for the immediate response @ParthaI. Appreciate it.

[aswin-kevin]

Hey @ParthaI ,

I’ve found a solution to the issue I was facing. It turns out that my keys aren’t able to list EC2 instances from the ap-south-2 region due to a problem on our AWS side (which we’ll resolve soon).

The issue occurs because I’ve set the regions to (*), so Steampipe tries to fetch EC2 instances from all regions. However, when it hits ap-south-2, it encounters an "AuthFailure" error, returning 0 results and breaking the entire flow. The process halts even though only one region is causing the error.

To address this, I added "AuthFailure" to the **ignore_error_code** block in the Steampipe configuration. Now, it bypasses the error in the ap-south-2 region and successfully fetches EC2 instances from the other regions, which is the intended behavior.

I was wondering if there’s a better way to achieve this? Additionally, I’d like to capture both the errors and the results if possible. The reason is, if in the future we encounter permission issues for a specific region or function, I’d like to be aware of the errors so we can enable the necessary permissions on our side.

Thanks!

[ParthaI]

Hi @aswin-kevin, great job identifying the root cause of the issue!

Typically, we don't manage authentication or access-denied errors directly. Steampipe executes parallel API calls based on the regions you've configured in your .spc file. When the API successfully returns responses from the configured regions, the query displays the results. However, if any of the parallel API calls fail to respond correctly, the entire query throws an error.

I was wondering if there’s a better way to achieve this? Additionally, I’d like to capture both the errors and the results if possible. The reason is, if in the future we encounter permission issues for a specific region or function, I’d like to be aware of the errors so we can enable the necessary permissions on our side.

At the moment, there isn't a built-in way to capture both the errors and the results simultaneously. The best option available is the ignore_error_code setting, which allows you to ignore certain error codes. Alternatively, you could configure only the regions where the caller has the necessary permissions to access the resources.

It might be useful to consider support for ignoring errors based on the error message rather than the error code, but this is a bit tricky. For instance, the AccessDeniedException error code can occur for various reasons:

• The user doesn't have access to a particular region.
• The user lacks permissions to a specific resource (e.g., due to an access control list (ACL) restriction).

In such cases, while the error messages may differ, the error code (AccessDeniedException) remains the same, making it challenging to handle error messages specifically.

That said, it looks like your current issue has been resolved. If you think adding support for ignoring the error by the error message might be helpful, feel free to raise a support request, and we’d be happy to assist further.

Thank you!

[aswin-kevin]

Hi @ParthaI I'm using the aws plugin postgres FDW. Looks like the regions and ignore_error_codes fields are not taken by the plugin.

Does the aws postgres fdw supports ignore_error_codes and regions parameter ?

In steampipe CLI everything works as expected after adding the ignore codes block. Currently the issue occurs in postgres FDW.

CREATE SERVER steampipe_aws FOREIGN DATA WRAPPER steampipe_postgres_aws OPTIONS (config ' regions = ["*"] ignore_error_codes = ["AuthFailure", "AccessDenied", "AccessDeniedException", "NotAuthorized", "UnauthorizedOperation", "UnrecognizedClientException", "AuthorizationError"] access_key = "$AWS_ACCESS_KEY_ID" secret_key = "$AWS_SECRET_ACCESS_KEY" session_token = "$AWS_SESSION_TOKEN"');

[ParthaI]

Ah, I see.

Does the AWS Postgres FDW support the ignore_error_codes and regions parameters?

Yes, according to the documentation here, it should support those parameters.

We’ll conduct further investigation and get back to you.

For reference, here’s a related discussion in the community channel: https://turbot-community.slack.com/archives/C01UECB59A7/p1726235163173639

Thanks!

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.