[ISSUE] Client requests are not load balanced by gateway service

[ChrisKujawa]

Describe the issue:

We have encountered that it looks like the gateway service doesn't really load balance the client requests, which makes increasing the replication a bit useless (except for fail over of course).

See related zeebe issue camunda/camunda#11310 and related forum post https://forum.camunda.io/t/under-utilized-cpu-for-gateway-and-broker-pods/42671/3

Actual behavior:

Expected behavior:

The service does a correct load balancing and resources of all deployed gateways are used.

How to reproduce:

Just deploy the helm charts and start sending request to zeebe. You can use our benchmarks for example. https://github.com/zeebe-io/benchmark-helm

Logs:
None.

Environment:

Please note: Without the following info, it's hard to resolve the issue and probably it will be closed.

• Platform: GCP
• Helm CLI version: ALL
• Chart version: ALL
• Values file: https://github.com/zeebe-io/benchmark-helm/blob/main/charts/zeebe-benchmark/values.yaml