Unable to export metrics directly to o11y

[pellared]

Describe the issue

https://docs.splunk.com/observability/en/gdi/get-data-in/application/go/instrumentation/instrument-go-application.html#send-data-directly-to-splunk-observability-cloud

It is not possible to send metrics directly to o11y Cloud.

Steps to reproduce the issue:
For example:

1. Use the code from main branch.
2. Set (export) SPLUNK_ACCESS_TOKEN SPLUNK_REALM OTEL_SERVICE_NAME OTEL_RESOURCE_ATTRIBUTES env vars
3. Run: cd example; go run .
4. See error:

2023-10-23T16:40:04.069Z        error   distro/otel.go:73       OpenTelemetry error     {"error": "failed to upload metrics: rpc error: code = Unimplemented desc = "}

Attempted solutions and workarounds

Setting OTEL_METRICS_EXPORTER="none" gets rid of the error but we do not have the metrics exported.

Expected behavior

Make it possible to export metrics directly to o11y.

We need to make sure to not make a breaking change when introducing a fix.

We should use OTLP over HTTP/protobuf when SPLUNK_REALM is set. This is what e.g. https://github.com/signalfx/splunk-otel-js/ does.

Additional context

We could also add support for OTEL_EXPORTER_OTLP_METRICS_PROTOCOL and other per signal metrics. Probably it would be best to reuse https://pkg.go.dev/go.opentelemetry.io/contrib/exporters/autoexport (with overridden defaults). This can be done later.

[pellared]

I created: splunk/public-o11y-docs#1008

[pellared]

Currently blocked by:

• otlpmetrichttp otlptracehttp: Handle non-protobuf responses open-telemetry/opentelemetry-go#4717