"Auto" doesn't work with opentelemetry-javaagent v0.9.0

[kewats]

I am trying to use opentelemetry javaagent and export traces to cloud trace. I am running into "No span exporter found" issues:

java -Dotel.exporter.jar=path/to/exporter-trace-0.9.1.jar  -javaagent:path/to/opentelemetry-javaagent-0.9.0-all.jar -jar path/to/myApp.jar
[opentelemetry.auto.trace 2020-11-04 08:08:19:881 +0530] [main] WARN io.opentelemetry.javaagent.tooling.TracerInstaller - No span exporter found in path/to/exporter-trace-0.9.1.jar
[opentelemetry.auto.trace 2020-11-04 08:08:19:882 +0530] [main] WARN io.opentelemetry.javaagent.tooling.TracerInstaller - No valid exporter found. Tracing will run but spans are dropped
[opentelemetry.auto.trace 2020-11-04 08:08:19:901 +0530] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 0.9.0
Hello world
[opentelemetry.auto.trace 2020-11-04 08:08:19:916 +0530] [opentelemetry-exec-shutdown-hook] WARN io.opentelemetry.javaagent.tooling.CommonTaskExecutor - Periodic task scheduler is shutdown. Will not run: cleaner for com.blogspot.mydailyjava.weaklockfree.WeakConcurrentMap@7d775612

Any suggestions on what could I be missing here or ways to investigate this further?

[nilebox]

Hi @kewats! It seems that you're passing the wrong jar in otel.exporter.jar=....
You need to pass path to exporter-auto rather than exporter-trace.

See README

Feel free to reopen if you still have issues with using it.

[kewats]

Hi @nilebox, thanks for your response. I tried with exporter-auto too, but I'm running into the same problem.

java -Dotel.exporter.jar=path/to/exporter-auto-0.9.1.jar  -javaagent:path/to/opentelemetry-javaagent-0.9.0-all.jar -jar path/to/myApp.jar
[opentelemetry.auto.trace 2020-11-04 09:06:32:199 +0530] [main] WARN io.opentelemetry.javaagent.tooling.TracerInstaller - No span exporter found in path/to/exporter-auto-0.9.1.jar
[opentelemetry.auto.trace 2020-11-04 09:06:32:200 +0530] [main] WARN io.opentelemetry.javaagent.tooling.TracerInstaller - No valid exporter found. Tracing will run but spans are dropped
[opentelemetry.auto.trace 2020-11-04 09:06:32:220 +0530] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 0.9.0
Hello world
[opentelemetry.auto.trace 2020-11-04 09:06:32:234 +0530] [opentelemetry-exec-shutdown-hook] WARN io.opentelemetry.javaagent.tooling.CommonTaskExecutor - Periodic task scheduler is shutdown. Will not run: cleaner for com.blogspot.mydailyjava.weaklockfree.WeakConcurrentMap@7d3ec58f

I also looked at https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/blob/master/build.gradle to see if I am using the wrong versions. It seems we need to use 0.8.0 of javaagent with 0.9.1 of exporter-auto. However, that leads to java.lang.ClassNotFoundException: com.google.cloud.opentelemetry.trace.TraceExporter
Any suggestions on this?

[nilebox]

By looking at the code in java-instrumentation, it requires Factory implementing io.opentelemetry.javaagent.spi.exporter.SpanExporterFactory, whereas our code implements io.opentelemetry.javaagent.tooling.exporter.SpanExporterFactory

opentelemetry-operations-java/exporters/auto/src/main/java/com/google/cloud/opentelemetry/auto/GoogleCloudSpanExporterFactory.java

Line 6 in fb7b39a

import io.opentelemetry.javaagent.tooling.exporter.SpanExporterFactory;

This incompatibility is caused by this PR open-telemetry/opentelemetry-java-instrumentation#1187

This is easy to fix:

• Update dependency from 0.8.0 to 0.9.0:

  opentelemetry-operations-java/build.gradle

  Line 27 in fb7b39a

  openTelemetryInstrumentationVersion = '0.8.0'

• Fix compilation by updating the interface.

The bigger problem is that Java agent tooling wasn't released in sync with SDK, that's why we have inconsistent dependency now.

[nilebox]

@kewats feel free to submit a PR fixing this issue if that's urgent.

[kewats]

Thanks @nilebox for the insight. I will submit a PR soon.

[kewats]

@nilebox Even after this change and on using the exporter-auto generated with it, I still get this:

Caused by: java.lang.NoClassDefFoundError: com/google/cloud/opentelemetry/trace/TraceExporter
	at com.google.cloud.opentelemetry.auto.GoogleCloudSpanExporterFactory.fromConfig(GoogleCloudSpanExporterFactory.java:15)
	at io.opentelemetry.javaagent.tooling.TracerInstaller.installExporter(TracerInstaller.java:181)
	at io.opentelemetry.javaagent.tooling.TracerInstaller.installExportersFromJar(TracerInstaller.java:155)
	at io.opentelemetry.javaagent.tooling.TracerInstaller.installAgentTracer(TracerInstaller.java:51)
	... 18 more
Caused by: java.lang.ClassNotFoundException: com.google.cloud.opentelemetry.trace.TraceExporter
	at io.opentelemetry.javaagent.tooling.ExporterClassLoader.findClass(ExporterClassLoader.java:80)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 22 more

Am I missing something more here that needs to be fixed?

[nilebox]

@kewats Looks like the exporter-trace jar is missing in your classpath for some reason.
To be clear, you need both exporter-trace (exporter implementation) and exporter-auto (javaagent integration) available in your classpath.

[kewats]

@nilebox Looks like we need to supply exporter-trace as xbootclasspath for the agent to pick it up. However, it seems this too is not sufficient.

java -Xbootclasspath/a:path/to/opentelemetry-operations-java/exporters/trace/build/libs/exporter-trace-0.10.0-SNAPSHOT.jar -Dotel.exporter.jar=path/to/opentelemetry-operations-java-auto-fix/exporters/auto/build/libs/exporter-auto-0.10.0-SNAPSHOT.jar -javaagent:path/to/opentelemetry-javaagent-0.9.0-all.jar -jar path/to/myApp.jar

[opentelemetry.auto.trace 2020-11-05 11:23:02:018 +0530] [main] ERROR io.opentelemetry.javaagent.bootstrap.AgentInitializer - Throwable thrown while installing the agent tracer
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer.installAgentTracer(AgentInitializer.java:162)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:75)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at io.opentelemetry.javaagent.OpenTelemetryAgent.agentmain(OpenTelemetryAgent.java:64)
	at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:51)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.NoClassDefFoundError: io/opentelemetry/sdk/trace/export/SpanExporter
	at java.lang.ClassLoader.findBootstrapClass(Native Method)
	at java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:1008)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:407)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at com.google.cloud.opentelemetry.auto.GoogleCloudSpanExporterFactory.fromConfig(GoogleCloudSpanExporterFactory.java:15)
	at io.opentelemetry.javaagent.tooling.TracerInstaller.installExporter(TracerInstaller.java:181)
	at io.opentelemetry.javaagent.tooling.TracerInstaller.installExportersFromJar(TracerInstaller.java:155)
	at io.opentelemetry.javaagent.tooling.TracerInstaller.installAgentTracer(TracerInstaller.java:51)
	... 18 more

[nilebox]

NoClassDefFoundError: io/opentelemetry/sdk/trace/export/SpanExporter

@kewats now you're missing opentelemetry-sdk (see SpanExporter.java)

You need all other dependencies in

opentelemetry-operations-java/build.gradle

Lines 33 to 46 in fb7b39a

	auto_service_annotations : "com.google.auto.service:auto-service-annotations:${autoServiceVersion}",
	auto_service : "com.google.auto.service:auto-service:${autoServiceVersion}",
	auto_value_annotations : "com.google.auto.value:auto-value-annotations:${autoValueVersion}",
	auto_value : "com.google.auto.value:auto-value:${autoValueVersion}",
	google_cloud_core : "com.google.cloud:google-cloud-core:${googleCloudVersion}",
	google_cloud_trace : "com.google.cloud:google-cloud-trace:${googleCloudVersion}",
	google_cloud_trace_grpc : "com.google.api.grpc:grpc-google-cloud-trace-v2:${googleCloudVersion}",
	google_cloud_monitoring : "com.google.cloud:google-cloud-monitoring:${cloudMonitoringVersion}",
	google_cloud_monitoring_grpc : "com.google.cloud:grpc-google-cloud-monitoring-v3:${cloudMonitoringVersion}",
	slf4j : "org.slf4j:slf4j-api:${slf4jVersion}",
	opentelemetry_api : "io.opentelemetry:opentelemetry-api:${openTelemetryVersion}",
	opentelemetry_sdk : "io.opentelemetry:opentelemetry-sdk:${openTelemetryVersion}",
	opentelemetry_auto : "io.opentelemetry.instrumentation.auto:opentelemetry-javaagent-tooling:${openTelemetryInstrumentationVersion}",
	opentelemetry_operations_java: "com.google.cloud.opentelemetry:exporter-trace:${opentelemetryOperationsVersion}"

as well, so I would just suggest to build an "uber jar" in your path/to/myApp.jar which will include all dependencies.

[nilebox]

Created an issue #50 for addressing this on our side.