Cross stack reference

[rozek1997]

Hi there!

I would like to use StackReference between projects. Currently I have two projects who shares the same s3 bucket. First project can be found under s3://trmz-infrastructure-state/abc-core/core path and second one under s3://trmz-infrastructure-state/abc-core/database. How can takes an output from the first core project and import it to database project. The documentation seems to very unclear to me and still don't know how should I organize the Pulumi project into the folders on the s3 buckets to able to to get outputs from another stack. Any ideas 💯 ?

[nimbinatus]

Hi there!
Generally, to use pulumi.StackReference(), you're calling a fully qualified stack name, meaning something that includes the org name and the project name as well as the stack. So, roughly, your code would read as follows if it's in Python (though I would recommend using variables; I'm hardcoding here to make it a bit clearer):

In the core project, your file would have something like this at the end:

pulumi.export("output_from_core_stack", f"{output_value_from_code}")

In the database project, your file would include something like this:

import pulumi

stack_ref = pulumi.StackReference("trmz-infrastructure-state/abc-core/core")

pulumi.export("output_I_want", stack_ref.get_output("output_from_core_stack"))

Note that the database project's code is referencing the stack with stack_ref.get_output() and is calling the exported output_from_core_stack key to get its value.

Does that help make stack references clearer?

For the organization, are you asking about how to organize a self-service backend? I'm not clear what you're trying to figure out for the s3 bucket organization.

[rozek1997]

Yeap, that is exactly what i was trying to achive. That option is available in Terraform via Terraform remote state data object, but in Pulumi StackReference which should be more or less equivalent, doesn't work that way.
The output of lacking of that feature and many other problems we have met working with Pulumi is that, we have started migrating everything to Terraform.

[rozek1997]

@Frassle i don't agree you need to have multiple service backends. We were using same single s3 bucket for every project/stack but divided that s3 bucket to subfolders: vpc for vpc project, database etc, but we still cannot reference to another subfolder in the same s3 bucket.

[Frassle]

but we still cannot reference to another subfolder in the same s3 bucket.

Right but that's due to the limitation of the file backend. It's not assigned to a bucket, but to a folder in that bucket. So you splitting each project into a different folder is a different backend as far as Pulumi is concerned right now.

There are issues tracking both the supporting of multiple backends for stack references (#3395), and for supporting folders in s3 buckets as one backend (#2522)

[ionut-botizan]

Wow, that's ...unfortunate! 😕

Especially since the docs clearly suggest that you can do that!

Stack names must be fully qualified, including the organization, project, and stack name components, in the format <organization>/<project>/<stack>. For individual accounts, use your account name for the organization component.

To expand on this further, imagine you need to define a cluster’s infrastructure in one project and consume it from another. Perhaps one project, infra, defines a Kubernetes cluster and another, services, deploys services into it. Let’s further imagine you are doing this across three distinct environments: production, staging, and testing. In that case, you will have six distinct stacks that pair up in the following ways:

mycompany/infra/production provides the cluster used by mycompany/services/production
mycompany/infra/staging provides the cluster used by mycompany/services/staging
mycompany/infra/testing provides the cluster used by mycompany/services/testing

[...some code sample]

In the above example, you construct a stack reference to a specific stack in this project which has the same name as your current stack (i.e. when deploying the “staging” stack of the above program, you reference the “staging” stack) from the infra project.

Maybe the docs should be updated to clarify if that's only possible when using the Pulumi Service backend...

[rozek1997]

@ionut-botizan glad you point the docs out. I think that was the point were I really get confused about how to structure the Pulumi projects 👍

[ionut-botizan]

I think this might have been fixed with the release of v3.61. Can anyone confirm?

Confirmed in #3395. 🎉

(We don't use Pulumi on the project I'm currently working on so I can't test, but it would be good to know 😁 )