Very slow getting metrics from jmx_exporter

[cloud-66]

Using this jmx_exporter config https://github.com/confluentinc/jmx-monitoring-stacks/blob/6.1.0-post/shared-assets/jmx-exporter/kafka_broker.yml
Get metrics from url takes 3 minutes !
How it is possible to use by anyone for any cluster? In what case could it be used?
This is very heavy config.
May I have to disable topic metrics.
Maybe I have to many topics and consumers for this exporter
kafka 2.7.2
2000 topics
40000 partitions
644 consumer groups (not all active)

Which metrics should I disable from this configuration first?

[jeqo]

Yeah, some metrics are labeled by topic and partitions, so multiply 40K * metrics labeled by partitions, and it's a pretty large dataset to load and export.
This tend to be a CPU intensive task. It could help to add some CPU to get this latency down.

I'd suggest to filter out metrics labeled by partition to reduce the amount of metrics to collect.
Another alternative is to enable cache on JMX rules: https://github.com/prometheus/jmx_exporter#configuration, though not sure how much this will reduce this latency.

If you find any workaround, please share it back if possible. It could be possible to add it the repo. Thanks!

[zyxep]

did you find a solution @cloud-66 ?
i am in similar position, the generation of /metrics is so slow the timeout has to be insanely high and then the whole point is gone with metrics. imo.

[cloud-66]

@zyxep No.
It is imposible to use jmx metrics for me because of slow, so i use this exporter which is working using kafka admin api

[zyxep]

@zyxep No. It is imposible to use jmx metrics for me because of slow, so i use this exporter which is working using kafka admin api

what are your times then?

[dhoard]

@cloud-66 as @jeqo pointed out the cardinality is huge for everything. Are using using the JMX Exporter v0.20.0? (latest release in this repo)