Use JDK http client (opentelemetry-exporter-sender-jdk) with Java agent

[marco-farinetti-evooq]

Hi all,

We're using Opentelemetry in our application with an extended java agent. The extension is built as described here build.gradle
The actual issue we're facing is with proxy authentication, the same as described here #6819. Since this is not yet supported and with Okhttp we can't just set the Proxy-Authorization header, we thought we could get around it by using the JDK http client.

We tried configuring it in the extension:

    compileOnly("io.opentelemetry:opentelemetry-exporter-sender-jdk")
    compileOnly("io.opentelemetry:opentelemetry-exporter-otlp") {
        exclude(group = "io.opentelemetry", module = "opentelemetry-exporter-sender-okhttp")
    }

but this doesn't affect the java agent jar, which seems to include only the Okhttp sender.

As a last resort (which is really not ideal), we tried forking the instrumentation repo and added the jdk sender to the java agent tooling referring to #5703

  implementation("io.opentelemetry:opentelemetry-exporter-otlp") {
    exclude(group = "io.opentelemetry", module = "opentelemetry-exporter-sender-okhttp")
  }
  implementation("io.opentelemetry:opentelemetry-exporter-sender-jdk")

I believe we're still missing something as when running the application it's still using the Okhttp client:

[otel.javaagent 2025-02-27 16:51:50:774 +0100] [OkHttp https://target-url/...] ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export metrics. The request could not be executed. Full error message: Failed to authenticate with proxy
java.io.IOException: Failed to authenticate with proxy
	at okhttp3.internal.connection.RealConnection.createTunnel(RealConnection.kt:476)
	...
	Suppressed: java.io.IOException: Failed to authenticate with proxy
		... 22 more

Do you have any suggestion? Thank you.

[jack-berg]

I also attempted to use opentelemetry-exporter-sender-jdk with the otel java agent by including it as an extension, but was was unsuccessful. Based on the output I saw, it looks like the SDK is only detecting the OkHttp implementation of the io.opentelemetry.exporter.internal.http.HttpSenderProvider SPI here:

opentelemetry-java/exporters/common/src/main/java/io/opentelemetry/exporter/internal/http/HttpExporterBuilder.java

Lines 259 to 275 in 3c71b79

	Map<String, HttpSenderProvider> httpSenderProviders = new HashMap<>();
	for (HttpSenderProvider spi :
	ServiceLoader.load(HttpSenderProvider.class, serviceClassLoader)) {
	httpSenderProviders.put(spi.getClass().getName(), spi);
	}
	// No provider on classpath, throw
	if (httpSenderProviders.isEmpty()) {
	throw new IllegalStateException(
	"No HttpSenderProvider found on classpath. Please add dependency on "
	+ "opentelemetry-exporter-sender-okhttp or opentelemetry-exporter-sender-jdk");
	}
	// Exactly one provider on classpath, use it
	if (httpSenderProviders.size() == 1) {
	return httpSenderProviders.values().stream().findFirst().get();
	}

@open-telemetry/java-instrumentation-approvers curious what you think about this. I suspect that the io.opentelemetry.exporter.internal.http.HttpSenderProvider SPI is being relocated, which is why the io.opentelemetry.exporter.sender.jdk.internal.JdkHttpSenderProvider implementation from the extension isn't being detected.

[jack-berg]

I didn't try this out but looking at it my guess would be that the issue is that agent loads extensions in a separate class loader

We just merged #7150 which makes the class loader used to load OTLP sender SPIs configurable. Now just need to figure out how to make sure OTLP exporters loaded via autoconfigure use the same class loader autoconfigure is configured with.

Though I would guess that instead of excluding the okhttp sender you could just say that you prefer using the jdk sender by using the configuration option

Yes this is exactly what I was going to say. If you've made it to the point where you get the warning about "Multiple HttpSenderProvider found.", then all you need to to is set a system property / env var to tell the SDK which to use:

-Dio.opentelemetry.exporter.internal.http.HttpSenderProvider=io.opentelemetry.exporter.sender.jdk.internal.JdkHttpSenderProvider

[trask]

I guess we could consider dropping the separate class loader for extensions. @trask wdyt?

if we can solve with #7150 that sounds ideal 👍

[laurit]

I guess we could consider dropping the separate class loader for extensions. @trask wdyt?

if we can solve with #7150 that sounds ideal 👍

There is the same issue with

opentelemetry-java/exporters/common/src/main/java/io/opentelemetry/exporter/internal/compression/CompressorUtil.java

Line 51 in 8efade6

ServiceLoader.load(CompressorProvider.class, CompressorUtil.class.getClassLoader())) {

An alternative would be to use context class loader. Currently we don't set the context loader in the agent, but we used to. Can't recall why we had to stop setting it.

[jack-berg]

There is the same issue with

Indeed!

[jack-berg]

#7181