Micrometer-bridge breaks Spring Actuator metrics

[santerivairio]

Describe the bug

Using javaagent and spring actuator with micrometer bridge enabled breaks sometimes the /metrics -endpoint.
In those cases it seems like the OpenTelemetryMetricsRegistry is created before PrometheusMetricsRegistry.

In those cases an error stating io.opentelemetry.javaagent.shaded.instrumentation.micrometer.v1_5.OpenTelemetryMeterRegistry - OpenTelemetry metrics bridge does not support reading measurements is shown.

But it some cases the same exact implementation works after restarting the application. In those cases the PrometheusMetricsRegistry is before the OpenTelemetry one in the registries-Set

We're using custom micrometer metrics that need to be bridged for the OpenTelemetry agent too.

Seems like the issue is that spring-actuator's metrics endpoint implementation uses the first registry that has the wanted metric in it. But because OpenTelemetryMeterRegistry is unreadable, the response is empty (no measurement value). Here's the cause of this issue.

We managed to fix this by overriding the MetricsEndpoint-implementation with a one that excludes OpenTelemetryMetricsRegistry. But in my opinion it would be nicer to have it fixed somewhere else, here perhaps?

Steps to reproduce

spring-boot-starter-actuator 3.2.5
opentelemetry-javaagent 2.9.0
micrometer bridge enabled
query metrics through actuator /actuator/metrics/<metric name>

Expected behavior

No errors, wouldn't require booting until the registries are in correct order

Actual behavior

io.opentelemetry.javaagent.shaded.instrumentation.micrometer.v1_5.OpenTelemetryMeterRegistry - OpenTelemetry metrics bridge does not support reading measurements is shown.

But it some cases the same exact implementation works after restarting the application. In those cases the PrometheusMetricsRegistry is before the OpenTelemetry one in the registries-Set

Javaagent or library instrumentation version

v2.9.0

Environment

JDK: Temurin 17.0.12
OS: eclipse-temurin:17-jre-alpine
Spring Boot 3.2.5

Additional context

No response

[trask]

maybe we need to add PrometheusMetricsRegistry here?

opentelemetry-java-instrumentation/instrumentation/spring/spring-boot-actuator-autoconfigure-2.0/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/spring/actuator/v2_0/OpenTelemetryMeterRegistryAutoConfiguration.java

Lines 23 to 26 in 11773aa

	// configure after the SimpleMeterRegistry has initialized; it is normally the last MeterRegistry
	// implementation to be configured, as it's used as a fallback
	// the OTel registry should be added in addition to that fallback and not replace it
	@AutoConfigureAfter(SimpleMetricsExportAutoConfiguration.class)

also I noticed that the Spring Boot Starter uses a different @AutoConfigureAfter below, I'm not sure why it's different, maybe it's better? cc @jeanbisutti @zeitlinger

opentelemetry-java-instrumentation/instrumentation/spring/spring-boot-autoconfigure/src/main/java/io/opentelemetry/instrumentation/spring/autoconfigure/internal/instrumentation/micrometer/MicrometerBridgeAutoConfiguration.java

Line 27 in 11773aa

@AutoConfigureAfter(MetricsAutoConfiguration.class)

[zeitlinger]

I don't know why @AutoConfigureAfter is different

[zeitlinger]

here's the definition of the prometheus auto config: https://github.com/spring-projects/spring-boot/blob/main/spring-boot-project/spring-boot-actuator-autoconfigure/src/main/java/org/springframework/boot/actuate/autoconfigure/metrics/export/prometheus/PrometheusMetricsExportAutoConfiguration.java

@AutoConfiguration(
		before = { CompositeMeterRegistryAutoConfiguration.class, SimpleMetricsExportAutoConfiguration.class },
		after = MetricsAutoConfiguration.class)
@ConditionalOnBean(Clock.class)
@ConditionalOnClass(PrometheusMeterRegistry.class)
@ConditionalOnEnabledMetricsExport("prometheus")
@EnableConfigurationProperties(PrometheusProperties.class)
public class PrometheusMetricsExportAutoConfiguration {

(all registries are defined this way)

compared to what we use

@Configuration
// CompositeMeterRegistryAutoConfiguration configures the "final" composite registry
@AutoConfigureBefore(CompositeMeterRegistryAutoConfiguration.class)
// configure after the SimpleMeterRegistry has initialized; it is normally the last MeterRegistry
// implementation to be configured, as it's used as a fallback
// the OTel registry should be added in addition to that fallback and not replace it
@AutoConfigureAfter(SimpleMetricsExportAutoConfiguration.class)
@ConditionalOnBean(Clock.class)
@ConditionalOnClass(MeterRegistry.class)
public class OpenTelemetryMeterRegistryAutoConfiguration {

I think the issue is that SimpleMetricsExportAutoConfiguration is absent when prometheus is configured - which makes the order upredictable