[Feature]: ? how to scale our local installation

[tuehlarsen]

What change would you like to see?

Can you give some advises about how to scale our production installation or how to optimize it?

Context

We use k3s as cluster SW and have currently two nodes running in our local production setup.

The master which is virtualized - 32GB RAM, 8 CPUs, it is both master and runs workloads.
We have added the first physical worker node - 64GB RAM 24 CPUs, which only runs workloads and plan to change the master also to a similar physical and add more physical workload servers.

we have using the API to schedule 3 x 800 frontpage jobs without any problems. The last API schedule
was with one hop out. We have in the GUI in the last schedule seen 17 concurrently running jobs. The first 2 schedules took about 2 hours and
we did not see any running jobs in the GUI probaly because they ran so fast that it was not measured in the Dashboard.
We have seen 154 concurrent crawler pods and 478 pods in total during the crawls.

Here a top from the 2 servers:

The Master seems to be maxed out:
CPU %user %nice %system %iowait %steal %idle
11:40:00 all 47.22 11.20 32.08 0.08 0.00 9.43
11:50:01 all 49.38 11.07 31.85 0.07 0.00 7.63
12:00:02 all 46.75 11.14 33.12 0.08 0.00 8.91
12:10:00 all 49.52 10.85 33.26 0.07 0.00 6.30
12:20:00 all 48.00 11.10 32.16 0.08 0.00 8.67
12:30:00 all 50.42 10.78 30.90 0.33 0.00 7.56
12:40:00 all 48.26 11.13 31.63 0.10 0.00 8.88
12:50:00 all 50.89 11.06 30.58 0.13 0.00 7.34
13:00:02 all 49.20 11.07 30.84 0.10 0.00 8.79

The new workernode:
CPU %user %nice %system %iowait %steal %idle
10:40:00 all 0.40 0.00 0.19 0.00 0.00 99.40
10:50:01 all 0.23 0.00 0.17 0.02 0.00 99.59
11:00:00 all 0.36 0.01 0.19 0.00 0.00 99.44
11:10:00 all 18.58 0.00 9.22 0.00 0.00 72.19
11:20:01 all 13.12 0.00 6.82 0.00 0.00 80.06
11:30:00 all 27.21 0.01 12.76 0.08 0.00 59.94
11:40:00 all 28.86 0.01 13.53 0.03 0.00 57.58
11:50:00 all 18.24 0.01 12.78 0.01 0.00 68.97
12:00:00 all 16.38 0.01 12.51 0.01 0.00 71.09
12:10:01 all 19.25 0.01 12.83 0.02 0.00 67.89
12:20:00 all 18.49 0.01 12.96 0.01 0.00 68.53
12:30:00 all 28.97 0.02 13.79 0.04 0.00 57.18
12:40:00 all 23.93 0.01 13.81 0.04 0.00 62.22
12:50:00 all 33.36 0.03 16.04 0.31 0.00 50.26
13:00:00 all 27.66 0.01 15.37 0.04 0.00 56.92