A cluster capacity formula for TF-controller at scale

[chanwit]

Overview

After reconciling 1,500 Terraform modules concurrently using TF-controller v0.10.1 on a 25-node cluster, I would like to share a formula derived from our successful experiment with anyone interested in setting up an EKS cluster and using TF-controller at a similar scale.

Formula

The following formula can be used to determine the number of nodes needed to provision an EKS cluster when using TF-controller at scale:

$$ N = \Bigl\lceil { \alpha \cdot {\sum{max\{\tau_p, \tau_i\}} \cdot O(\bar{w})^{-1}} \over \bar{C_n}} \Bigl\rceil + k$$

In this formula:

• $N$ is the number of nodes required for provisioning.
• $\alpha$ is the factor needed to account for pods being terminated, with a constraint that $\alpha >= 1.0$. If we have 100 running pods and 20 pods are being terminated, then $\alpha = \frac{100+20}{100} = 1.2$
• $\tau_i$ is the .spec.interval of each TF-controller object (infra.contrib.fluxcd.io/v1alpha1.Terraform).
• $\tau_p$ is the processing time expected to be used by each Terraform module.
• $O(\bar{w})$ is the worst-case average wait time.
• $\bar{C_n}$ is the average of each node's pod capacity, which is the maximum number of pods allowed.
• $k$ is a constant representing the additional nodes required for other workloads, such as Flux and the TF-controller itself.

Example

For the given experiment with 1,500 Terraform modules, we can use the formula to calculate the number of nodes needed for provisioning. Given that each module has a .spec.interval ($\tau_i$) of 1m, and the maximum processing time of each module ($\tau_p$) is less than 30 seconds, we can set $\sum{max\{\tau_p, \tau_i\}}$ to 1,500. From the experiment, we found that $\alpha = 1.2$. Assuming an average wait time of 3 minutes for each reconciliation loop of a Terraform module, we can set $O(\bar{w})$ to 3. In our setup, $\bar{C_n}$ is 29, which can be obtained from the information of each Kubernetes node. Finally, we have $k = 4$ for this setup.

By substituting the appropriate values into the formula, we can determine the optimal size of an EKS cluster for managing Terraform modules, similar to the setup used in the experiment.

$$ \begin{aligned} N &= \Bigl\lceil {1.2 \cdot {1500 \cdot 3^{-1}} \over 29} \Bigl\rceil + 4 \\ N &= \Bigl\lceil {1.2 \cdot {500} \over 29} \Bigl\rceil + 4 \\ N &= \Bigl\lceil {600 \over 29} \Bigl\rceil + 4 \\ N &= \Bigl\lceil {20.6896552} \Bigl\rceil + 4 \\ N &= 21 + 4 \\ \therefore N &= 25 \end{aligned} $$

That's it. We got $N = 25$, the number of nodes we need to provision for reconciling 1,500 Terraform modules concurrently, with no more than a 3-minute wait in each reconciliation loop.

Maximum Concurrency

You'll find that a key setting to scale the cluster is in the values of the TF-controller's Helm chart. It's .concurrency, the maximum number of Go routines allowed in the controller. The value of this concurrency setting can be obtained from this term $\sum{max\{\tau_p, \tau_i\}} \cdot O(\bar{w})^{-1}$.

$$ \begin{aligned} concurrency &= \sum{max\{\tau_p, \tau_i\}} \cdot O(\bar{w})^{-1} \\ concurrency &= {1500 \cdot {1 \over 3}} \\ concurrency &= 500 \end{aligned} $$

Setting for a Small Cluster

You may ask: OK, I have a 5-node management cluster. How many Terraform modules can I manage at a time on the cluster?
Let's inverse the equation and round down instead of up.

$$ m = \Bigl\lfloor { {\bar{C_n} \cdot (N-k) \cdot O(\bar{w})} \over {\alpha \cdot \overline{max\{\tau_p, \tau_i\}}} }\Bigl\rfloor $$

$m$ is the number of modules we're finding.

$$ \begin{aligned} m &= \Bigl\lfloor{{29 \cdot (5-4) \cdot 3} \over {1.2 \cdot 1}} \Bigl\rfloor \\ m &= \Bigl\lfloor{72.5}\Bigl\rfloor \\ \therefore m &= 72 \end{aligned} $$

So it's OK to use a 5-node EKS cluster (with the default eksctl config) to manage 72 Terraform modules at a 3-minute wait time.
We should set concurrency to ${72 \cdot 1} / {3}$ = 24 for the TF-controller used by this cluster.

[Nalum]

When you say modules are you referring to a tf-controller Terraform CR or a module in one of those CRs? I assume a Terraform CR but just wanted to be sure.

[Nalum]

Okay so if you have one Terraform CR deployed to the cluster that has 5 modules in it then you want to set $\sum{TF_{interval}}=5$ in the calculation?

[chanwit]

When you say 5 modules, you also need 5 Terraform CR to control each module.

If each Terraform CR has interval: 1m, then it becomes 5 minutes.

[Nalum]

Ah so when you say module here you're referring to a root module and not sub-modules and that root module maps to 1 Terraform CR?

[chanwit]

Exactly, yes.

[Nalum]

Awesome thanks 👍 😄

[tomhuang12]

Will the size of the terraform module affect this? Let's say you have a terraform module that takes a few minutes to vs a small module that only takes a few seconds?

[Nalum]

I think that is what this is:

On average, we want to wait no longer than 3 minutes for each reconciliation loop of a Terraform module, so $O(\bar{w})$ is 3.

[chanwit]

Yes. That's why we can specify the worse waiting time with $O(\bar{w})$. If a module is very fast, say 1m, while another is very slow like 30m, the value of $O(\bar{w})$ can be like 15.

[chanwit]

Thank you, @tomhuang12, that your question triggered me to re-think the semantics of .interval and the processing (provisional) time.
To correct it, I removed TF_interval and used $\tau_i$ and $\tau_p$ instead. The max value between these two represents the more realistic value of time spent by each reconciliation loop.

[artem-nefedov]

I would assume that node size (instance type) should also be accounted for

[chanwit]

Yes, it should. 29 in the example was the pod capacity of the default instance type set by EKSctl.