run-all init fails with parallelism

[gnuletik]

Describe the bug

A clear and concise description of what the bug is.

When running the following command:

rm -rf **/.terragrunt-cache && terragrunt run-all init

I get one of the following error on some projects:

│ Error: Backend initialization required, please run "terraform init"
│
│ Reason: Initial configuration of the requested backend "s3"
│
│ The "backend" is the interface that Terraform uses to store state,
│ perform operations, etc. If this message is showing up, it means that the
│ Terraform configuration you're using is using a custom configuration for
│ the Terraform backend.
│
│ Changes to backend configurations require reinitialization. This allows
│ Terraform to set up the new configuration, copy existing state, etc. Please
│ run
│ "terraform init" with either the "-reconfigure" or "-migrate-state" flags
│ to
│ use the current configuration.
│
│ If the change reason above is incorrect, please verify your configuration
│ hasn't changed and try again. At this point, no changes to your existing
│ configuration or state have been made.

╷
│ Error: Required plugins are not installed
│
│ The installed provider plugins are not consistent with the packages
│ selected in the dependency lock file:
│   - registry.terraform.io/hashicorp/aws: there is no package for registry.terraform.io/hashicorp/aws 5.46.0 cached in .terraform/providers
│   - registry.terraform.io/hashicorp/http: there is no package for registry.terraform.io/hashicorp/http 3.4.2 cached in .terraform/providers
│   - registry.terraform.io/datadog/datadog: there is no package for registry.terraform.io/datadog/datadog 3.38.0 cached in .terraform/providers
│
│ Terraform uses external plugins to integrate with a variety of different
│ infrastructure services. To download the plugins required for this
│ configuration, run:
│   terraform init
╵

The workaround was to set TERRAGRUNT_PARALLELISM=1.

To Reproduce

I was not able to extract a small reproducible example.

Expected behavior

To not fail with parallelism enabled.

Versions

• Terragrunt version: v0.57.6
• Terraform version: v1.5.7
• Environment details (Ubuntu 20.04, Windows 10, etc.): Mac OS 14.4.1

[levkohimins]

Hi @gnuletik, most likely you are using the Provider Plugin Cache. Make sure to read Provider Caching.

[gnuletik]

Hi @levkohimins, thanks for the feedback !

I was using the Terraform's provider plugin cache but I disabled it since I migrated to Terragrunt's Provider Caching Server.

I was able to reproduce this issue with:

• Terragrunt's provider caching
• without any caching mechanism enabled

Also, previous releases of Terragrunt didn't have this issue because the parallelism was disabled during init (because of the Terraform's provider plugin cache issue).

[levkohimins]

Hi @gnuletik,
It would be most effective if you make a minimal set of configuration files that will reproduce this issue and upload them to a public repository, because I cannot reproduce the issue from the data, you provided in the description. Thank you!

[gnuletik]

I already tried to reproduce it without success as it only occurs on projects with multiple different modules / configuration and doesn't occurs 100% of the time (seems to be around 70%).
I'll try to find a way to reproduce it though.

[levkohimins]

To clarify, before the implementation of Terragrunt Provider Caching, with Provider Plugin Cache enabled and running run-all init command, Terragrunt automatically set TERRAGRUNT_PARALLELISM=1, but with the advent of Terragrunt Provider Caching, this behavior was removed.

Therefore, I asked you to check whether this feautre is enabled in the Terraform CLI config or in the environment variable

plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"

export TF_PLUGIN_CACHE_DIR="$HOME/.terraform.d/plugin-cache"

[gnuletik]

@levkohimins Yes I can confirm that this configuration is not enabled.
Also, this behavior has been reproduced both locally and on our CI.