Namespacing limitations

[artem-nefedov]

While investigating the possibility of moving our current architecture to tf-controller, we've hit a pretty big wall because of the namespacing limitations. Specifically, 2 things in particular:

• runner Pod always starts in the same namespace as Terraform resource, meaning ServiceAccount must exist there (which especially problematic considering we use IRSA)
• varsFrom requires Secret to be in the same namespace (this is used for static credentials, and there's only one such Secret in the cluster)

Now, we could maybe handle it somehow if instead at least output Secret could be written to a different namespace, but right now that isn't possible as well.

Our current solution is Crossplane-based, and it the overall design heavily on a large number of arbitrarily generated namespaces for various reasons. Crossplane works with that because composition claims are namespaced, and will write output Secret to the namespace they exist in, but the ProviderConfig resource referenced in composition is NOT namespaced, and it allows to use static credentials Secret from any namespace (which we only have one of). Also there are no runners.

Before proceeding further, I would like to clarify if current namespace limitations of tf-controller are there to stay, and then decide accordingly.

[chanwit]

Our design is very serious about security. We do not, and will not, violate the namespace boundary to make TF-controller secure and trustworthy. That's our principle.

Every service account and secret must belong to a specific namespace. Runner pods also belong to the namespace too.

• For IRSA, yes, you must annotate your role ARN to each runner service account. It requires once per namespace.
• For sharing Secrets across namespaces, there's a controller to help you do so while not compromising security: https://github.com/vmware-tanzu/carvel-secretgen-controller/blob/develop/docs/secret-export.md

Trust me - it's a bit hard to deal with this namespace scope initially, but it will pay you off when it prevents bad (security) things from happening.

[chanwit]

Terraforms in separate namespaces are ideally the model we would support.
IAM roles would be created separately for each namespace. I think EKSctl allows us to do so here: https://eksctl.io/usage/iamserviceaccounts/.

I would say it's a secure model, but I can feel your pain that a restricted model adds more work.
And I agree in a sense that we would need tooling probably in tfctl to help users set this model up.

[artem-nefedov]

IAM roles would be created separately for each namespace. I think EKSctl allows us to do so here

That don't really work for us, because at cluster creation time we don't know how many namespaces will be created later and what will be their names. Those will be generated later.
And, we only run eksctl once (at cluster creation), afterwards everything is applied declaratively with Flux.

[tomhuang12]

A possible workaround would be deploying tf-controller in the all the namespaces where you want the terraform to run. This is obviously not ideal as you need to manage multiple tf-controller instances.

@chanwit maybe we could think about multi-tenancy feature that Flux has where tf-controller uses tenant-specific service account to create runner pods and access other resources for each tenant namespace?

[chanwit]

We support MT already, but it's just a bit tedious to set it up for each namespace.

tfctl command would be a good way to help solve UX in this MT area.

[artem-nefedov]

Our clusters are single-tenant, which is why we use single SA/credential secret instead of multiple.
Namespace separation is there for a different reason (it's part of operational logic).

The service, which runs in cluster, generates namespaces procedurally at runtime, stores them in git, and flux applies them.
At this point, only flux can make changes to the cluster.