Terraform does not use AWS Task Roles to fetch modules from s3

[repl-david-winiarski]

Terraform Version

Terraform v0.11.11

Terraform Configuration Files

module "s3-module" {
  source                   = "BUCKET.s3-us-west-2.amazonaws.com/1/terraform/2/module.zip"
 somekey                 = "somevalue"
}

Debug Output

2019/02/21 20:58:46 [INFO] Terraform version: 0.11.11 ac4fff416318bf0915a0ab80e062a99ef3724334
2019/02/21 20:58:46 [INFO] Go runtime version: go1.11.1
2019/02/21 20:58:46 [INFO] CLI args: []string{"/bin/terraform", "init", "-no-color"}
2019/02/21 20:58:46 [DEBUG] Attempting to open CLI config file: /root/.terraformrc
2019/02/21 20:58:46 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2019/02/21 20:58:46 [INFO] CLI command args: []string{"init", "-no-color"}
2019/02/21 20:58:46 [DEBUG] command: loading backend config file: /app
Initializing modules...
2019/02/21 20:58:46 [TRACE] module source: "./local-module"
2019/02/21 20:58:46 [TRACE] detected module source "file:///app/local-module"
2019/02/21 20:58:46 [DEBUG] fetching module from file:///app/local-module
2019/02/21 20:58:46 [DEBUG] fetching "file:///app/local-module" with key "1.local_module;./local-module"
- module.local-module
 Getting source "./local-module"
2019/02/21 20:58:46 [DEBUG] found "file:///app/local-module" in ".terraform/modules/7d7d4b57b2fd167574633e7a7d85c9b5": true
2019/02/21 20:58:46 [TRACE] "file:///app/local-module" stored in ".terraform/modules/7d7d4b57b2fd167574633e7a7d85c9b5"
- module.s3-module
2019/02/21 20:58:46 [TRACE] module source: "bucket.s3-us-west-2.amazonaws.com/1/terraform/2/s3-module.zip"
2019/02/21 20:58:46 [TRACE] detected module source "s3::https://s3-us-west-2.amazonaws.com/bucket/1/terraform/2/s3-module.zip"
2019/02/21 20:58:46 [DEBUG] fetching module from s3::https://s3-us-west-2.amazonaws.com/bucket/1/terraform/2/s3-module.zip
2019/02/21 20:58:46 [DEBUG] fetching "s3::https://s3-us-west-2.amazonaws.com/bucket/1/terraform/2/s3-module.zip" with key "1.s3_module;bucket.s3-us-west-2.amazonaws.com/1/terraform/2/s3-module.zip"
 Getting source "bucket.s3-us-west-2.amazonaws.com/1/terraform2/s3-module.zip"
Error downloading modules: Error loading modules: NoCredentialProviders: no valid providers in chain. Deprecated.
2019/02/21 20:59:07 [DEBUG] plugin: waiting for all plugin processes to complete...
 For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Crash Output

Expected Behavior

Terraform should of been able to fetch the module from s3.

Actual Behavior

Terraform init failed to fetch the module and exited with error: Error downloading modules: Error loading modules: NoCredentialProviders: no valid providers in chain. Deprecated.

Steps to Reproduce

Attempt to pull a s3 module with proper IAM permissions to access the s3 location but via a ECS / Codebuild task role. It is specifically accessing it via the Task Role that causes this issue. Using the EC2 role works and passing in he Access ID, Key and Session token work as well.

Additional Context

Terraform is being executed in a codebuild stage using a task role. It can access our remote state in s3 using the same task role with no issues. The only issue is with fetching a module from s3.

References

[tbondarchuk]

Looks like the terraform's code responsible for downloading modules from s3 could work only with aws_access_key_id/aws_secret_access_key/aws_secret_access_key set explicitly or grabbed from EC2 metadata endpoint, but not with assumed role or credentials from EcsContainer metadata endpoint.

Tested by configuring custom profiles in .aws/config and running terraform as AWS_PROFILE=test terraform init.

[solarmosaic-kflorence]

The root issue is here: hashicorp/go-getter#152 in the go-getter library

[solarmosaic-kflorence]

Hm, it seems like hashicorp/go-getter#152 has been resolved by hashicorp/go-getter#218 which made it into this release https://github.com/hashicorp/go-getter/releases/tag/v1.4.2 -- however, I just tested this against terraform 0.13.5 and the problem remains. Terraform seems to be using some variant of 1.4.2: https://github.com/hashicorp/terraform/blob/master/go.mod#L57 -- but I can't tell if it includes this change or not.

[solarmosaic-kflorence]

My guess is that the version used by terraform is some pre-release version of 1.4.2 which does not include the change. Any chance we can update to 1.4.2 stable, or even 1.5.0? @alisdair @jbardin seems like you have made several updates to dependencies lately, perhaps you can provide some guidance.

[alisdair]

@solarmosaic-kflorence Here's a PR upgrading go-getter: #26762. Are you able to build that branch locally and test if it solves the issue?

If not, can you provide detailed reproduction steps so that I can verify this issue is fixed? I'm not familiar with the use of AWS Task Roles.

[solarmosaic-kflorence]

@alisdair oh cool, thanks. I have not built terraform locally before but will try to do so later today if time permits.

This can be easily reproduced by running terraform within AWS CodeBuild which includes a module with an S3 bucket as the source: https://www.terraform.io/docs/modules/sources.html#s3-bucket -- on my case it is the result of executing terratest.

The result currently is a failure that looks like this:

TestUnit_PlanWithDefaults 2020-10-30T17:53:22Z logger.go:66: Downloading s3::https://s3.amazonaws.com/path/to/module.zip for module...
--
119 | TestUnit_PlanWithDefaults 2020-10-30T17:53:22Z logger.go:66:
120 | TestUnit_PlanWithDefaults 2020-10-30T17:53:22Z logger.go:66: Error: Failed to download module
121 | TestUnit_PlanWithDefaults 2020-10-30T17:53:22Z logger.go:66:
122 | TestUnit_PlanWithDefaults 2020-10-30T17:53:22Z logger.go:66: Could not download module "module" (../../../main.tf:142) source code from
123 | TestUnit_PlanWithDefaults 2020-10-30T17:53:22Z logger.go:66: "s3::https://s3.amazonaws.com/path/to/module.zip":
124 | TestUnit_PlanWithDefaults 2020-10-30T17:53:22Z logger.go:66: NoCredentialProviders: no valid providers in chain. Deprecated.
125 | TestUnit_PlanWithDefaults 2020-10-30T17:53:22Z logger.go:66:    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

[alisdair]

@solarmosaic-kflorence I looked into setting up CodeBuild or even just ECS to test this, but was not able to get anything up and running in a reasonable amount of time. The upstream changes really ought to fix it, so I'm marking this closed, but it would be good to get your confirmation if you're able to test the next 0.14 beta when it's released. Thanks!

[solarmosaic-kflorence]

Awesome, really appreciate it @alisdair -- happy to test it when the beta is out.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.