Possibility to re-initialize OpenTelemetry runtime

[bencehornak]

Is your feature request related to a problem? Please describe.
I would like to reconfigure OpenTelemetry during runtime without restarting my process. However, the GlobalOpenTelemetry API is designed to be immutable for the whole lifespan of the application.

opentelemetry-java/api/all/src/main/java/io/opentelemetry/api/GlobalOpenTelemetry.java

Lines 106 to 113 in cb44b2b

	if (globalOpenTelemetry != null) {
	throw new IllegalStateException(
	"GlobalOpenTelemetry.set has already been called. GlobalOpenTelemetry.set must be "
	+ "called only once before any calls to GlobalOpenTelemetry.get. If you are using "
	+ "the OpenTelemetrySdk, use OpenTelemetrySdkBuilder.buildAndRegisterGlobal "
	+ "instead. Previous invocation set to cause of this exception.",
	setGlobalCaller);
	}

Describe the solution you'd like
The GlobalOpenTelemetry class could allow changing the registered OpenTelemetry object, and could provide a reactive interface:

I can register a new OpenTelemetry object every time a user logs in for example:

UserSession userSession = myUserService.login(...);
MyCredentials creds = myService.getOtlpCredentials(userSession);
OpenTelemetrySdk.builder()
  .setTracerProvider(
    SdkTracerProvider.builder()
      .addSpanProcessor(
        OtlpGrpcSpanExporter.builder()
          .addHeader("Authorization", creds.mySecretAuthorizationHeader)
          .build()
      )
      .setResource(resource)
      .build();
  )
  .buildAndRegisterGlobal();

And my services can create their OpenTelemetry derived fields reactively:

class MyService {
  private Tracer myTracer;
  private OpenTelemetryChangedListener listener = new OpenTelemetryChangedListener() {
    public void openTelemetryChanged(OpenTelemetry openTelemetry) {
      myTracer = openTelemetry.getTracer("MyService");
    }
  };

  public void init() {
    GlobalOpenTelemetry.registerListener(listener);
  }
  public void shutdown() {
    GlobalOpenTelemetry.unregisterListener(listener);
  }
}

Describe alternatives you've considered
AFAIK the only way to change the registered OpenTelemetry object on GlobalOpenTelemetry is to restart the application.

Alternatively, I could prevent using the GlobalOpenTelemetry API, as well as the Java Instrumentation library, but that would defeat their purpose, wouldn't it?

Additional context
I stumbled upon two related use-cases, but please let me know, if there would be better ways to do things:

1. Client-side instrumentation, where my Otel backend's basic auth/OIDC credentials are scoped to a user session. I don't want to hardcode the basic auth/OIDC credentials into my distributed application, instead, I would like to securely pass them to my clients, when users have successfully logged in to my service. So on each login, I would like to register a new OpenTelemetry object for GlobalOpenTelemetry, and on each logout, I would like to unregister it.
2. Server-side instrumentation, with legacy long-running applications. Since there is no rolling update or load-balancing available for my legacy service, I would like to be able to change certain OpenTelemetry parameters (such as sampling rate or the list of sampled methods) without restarting my service, in order to prevent downtimes for my users. In a perfect world, the Java Instrumentation library could watch for changes in my configuration file, and would register a new OpenTelemetry object on GlobalOpenTelemetry, when any changes to my configuration occur (which is not possible with the current GlobalOpenTelemetry design).

[jkwatson]

The only case we recommend using the GlobalOpenTelemetry instance is if you're using the OTel javaagent. In that case, the agent itself controls the lifecycle of the instance, so you wouldn't be able to manipulate it, even if the underlying APIs allowed you to.

If you need to replace your instance of OpenTelemetry, it will probably be best to just maintain your own state, and swap it as you need to.

For the configuration of an existing instance (like sampling, etc), you can do some of this today by providing your own implementation of Sampler that allows you to change how it works.

There is some speculative work-in-progress to figure out a more general approach to dynamically reconfiguring an SDK instance at runtime, but it's still a ways off until it's complete and production-ready.

[jack-berg]

Client-side instrumentation, where my Otel backend's basic auth/OIDC credentials are scoped to a user session. I don't want to hardcode the basic auth/OIDC credentials into my distributed application, instead, I would like to securely pass them to my clients, when users have successfully logged in to my service.

I think you need a custom header supplier (i.e. OtlpHttpSpanExporterBuilder#setHeaders(Supplier<Map<String, String>>). The supplier can coordinate with the auth/OIDC logic in the app and adjust the headers which are sent according to user session state. No need to replace GlobalOpenTelemetry. See dynamic header-based authentication docs for more information.

[bencehornak]

Thanks for both of you for the great suggestions.

@jack-berg I hadn't known this function with the header Supplier, that is definitely the way to go for my 1st use-case (OIDC).

@jkwatson can you point me to the discussion around the speculative reconfiguring project? That's going to help with the 2nd one (long-running legacy applications).

[jack-berg]

@jkwatson can you point me to the discussion around the speculative reconfiguring project? That's going to help with the 2nd one (long-running legacy applications).

I'm working a cross-functional project called declarative config. The idea is to define a expressive data model for users to declare what their intended SDK configuration is. This will typically be encoded in a YAML file with a pointer to the file's path provided as an env var at startup.

We're actively working towards stabilizing the core building blocks of that, which would support static YAML based configuration. There's been a decent amount of discussion about followup work where SDKs would support dynamic configuration by using opamp as a mechanism to communicate between clients and servers, and the declarative config data model to communicate config updates. As @jkwatson mentions, its still some way off.

Potentially closer on the horizon, there's interest in prototyping new programmatic APIs for reconfiguring certain bits of the SDK. The opamp + declarative config dynamic config mechanism would eventually leverage these programmatic APIs. See #7021 for an example of the early early proposal for one of these APIs. If such an API landed for updating the Sampler, you'd be able to call it to update SDK config, probably long before the opamp + declarative config mechanism is available.

Available immediately, you could use the JaegerRemoteSampler to have your SDK instances receive sampler updates from a remote server. The portion of the SDK which is dynamically configuration is limited to samplers.