ActiveMQ Artemis + JMX Exporter 1.x gives error: -1.0: counters cannot have a negative value

[marcelbil]

For years the organization I work for used Apache ActiveMQ Artemis versions combined with an older version or JMX Exporter: 0.17.0. Worked like a charm, except a little bug we'd like to resolve. Since 0.17.0 is dated over 3 years back the first thing I tried was upgrading to the latest version: 1.5.0. This doesn't work directly. When performing a curl http://localhost:9404/metrics the following error occurs:

An Exception occurred while scraping metrics: java.lang.IllegalArgumentException: -1.0: counters cannot have a negative value
        at io.prometheus.metrics.model.snapshots.CounterSnapshot$CounterDataPointSnapshot.validate(CounterSnapshot.java:78)
        at io.prometheus.metrics.model.snapshots.CounterSnapshot$CounterDataPointSnapshot.<init>(CounterSnapshot.java:64)
        at io.prometheus.metrics.model.snapshots.CounterSnapshot$CounterDataPointSnapshot$Builder.build(CounterSnapshot.java:115)
        at io.prometheus.jmx.MatchedRuleToMetricSnapshotsConverter.convertRulesWithSameName(MatchedRuleToMetricSnapshotsConverter.java:106)
        at io.prometheus.jmx.MatchedRuleToMetricSnapshotsConverter.convert(MatchedRuleToMetricSnapshotsConverter.java:79)
        at io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:795)
        at io.prometheus.metrics.model.registry.MultiCollector.collect(MultiCollector.java:21)
        at io.prometheus.metrics.model.registry.PrometheusRegistry.scrape(PrometheusRegistry.java:84)
        at io.prometheus.metrics.exporter.common.PrometheusScrapeHandler.scrape(PrometheusScrapeHandler.java:135)
        at io.prometheus.metrics.exporter.common.PrometheusScrapeHandler.handleRequest(PrometheusScrapeHandler.java:56)
        at io.prometheus.metrics.exporter.httpserver.MetricsHandler.handle(MetricsHandler.java:33)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Unknown Source)
        at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(Unknown Source)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Unknown Source)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(Unknown Source)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Unknown Source)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

The same error also on version 1.1.0 and 1.0.1. Version 0.20.0 seems to be the latest version that work well.

[dhoard]

This is working as designed.

In both the Prometheus and Open Metrics formats, a counter must start at 0 and be monotonically increasing. A reset may occur, but must be 0.

The previous version of the client_java dependency used by older versions of the JMX Exporter incorrectly allowed a counter value less than 0.

[marcelbil]

Thank you for your quick reponse! Sounds logic. So you're saying I need to update the client_java dependency here? Or is it related to ActiveMQ Artemis configuration and the provided metrics (possible incorrect).

[dhoard]

I think there are two possible solutions:

1. change the JMX Exporter configuration from counter to gauge, which should allow a negative number. Downstream dashboards, etc. most likely would have to be changed.

2. if the value is truly a counter, then an issue needs to be filed against ActiveMQ to return a value >= 0.

[marcelbil]

Ok thank you. I will test your proposed solution(s) and will let you know the result.