How to create 1000 worker clusters and check the resource consumption of management cluster.

[mihalycsokas]

I would like to measure the resources consumed by the controller manager while it
manages a thousand clusters. Because of the price it not possible to test it in
the public cloud. My idea is to create clusters on docker provider. But it is not realistic to have a
computer which can handle so many containers.

Is there a way to use docker provider to deploy a cluster on multiple docker
machines (e.g. docker swarm). If it is not possible, how complicated would it be to write
a new controller manager based on docker?

[sbueringer]

/cc @elmiko (kubemark might be able to help here I think)

[enxebre]

@mihalycsokas Is the goal to figure a Cluster footprint and calculate how many clusters a given management cluster can allocate/cost?

[mihalycsokas]

Actually I would like to make sure my management cluster can handle 1000 clusters and I would like to know the controller's resource consumption to set appropriate resource requests and limits.

[elmiko]

thanks for the ping @sbueringer

yeah, we could definitely simulate that, or attempt to, with the kubemark provider. depending on the size of the workload clusters we might need a relatively large instance (or a couple) to run it. kubemark by itself doesn't take up too much resources, but once you get more than 20-30 running on a moderate instance (2 cpu, 16G ram) it will start to get bogged down. but, if you use relatively large instances you can stuff a lot of kubemark in there.

[elmiko]

we do have a slack channel #cluster-api-kubemark on the kubernetes.slack.com instance. that said, i am happy to have the conversation here or there. the nice thing about having it here could be that others will find the discussion when searching the project, but i'm happy to chat where you'd like =)

[mihalycsokas]

Good to hear :)
It is my understanding that to create a simulated cluster with capk I always need a real control plane node. My problem is creating so many control plane nodes.
Is there a way to emulate, simulate control plane nodes? Any recommendation about it?
I am trying with docker provider but I’m not sure it is possible without kind management cluster.

[elmiko]

your understanding is correct. for each capk cluster you will need a "real" control plane. so in your scenario you could do something like this:

1. create a management cluster with kind
2. create a workload cluster with a capd control plane
3. create capk compute nodes in the workload cluster
4. repeat steps 2,3 for as many clusters as you want.

i imagine it will take a fairly large instance to do all this work as all the processes will be running in the same host. but, using something like a 40 core machine with a good amount of ram (eg >=128G), would probably be a good point to start pushing the limits. i'm not sure if you could reach 1000 control planes, but it would be interesting to see.

as far as emulating the control plane, i'm not so sure about that. i haven't seen any projects that emulate a control plane, i suppose with something like kcp it might be possible to create an emulation, but i think it will be more trouble than just spinning up the docker control planes. you will still need the control plane to react in a predictable manner for cluster-api to work properly.

I am trying with docker provider but I’m not sure it is possible without kind management cluster.

could you expand on this a little bit, i'm not sure i follow. you will need some sort of management cluster if you want to test its density, otherwise you will be dealing with "joined" clusters and that won't test your original premise.

[mihalycsokas]

My idea was that I could run docker provider based control planes on different worker nodes if my management cluster is a "real" multi node cluster not a kind based one. Now I see it is not so easy.

[elmiko]

that will be difficult as the capd provider wants to use a local socket file to communicate with the docker daemon. i suppose it might be possible to do some magic with networking, but i have never tried this and i expect it would be frustrating to get working.

so, from a kind management cluster, the docker provider wants to talk on the local socket to the daemon to create the containers for the nodes. i have not explored deeply here.

[mihalycsokas]

Thank you for the quick responses!

I will try Kubemark, and post the results.

[fabriziopandini]

We did this in 1.5 and you can look at this KubeCon talk for reference