Validate Connector Usage

[arielvalentin]

Is your feature request related to a problem? Please describe.

I expected the configuration validator to find pipeline issues related to invalid usage of connector components, however the collector fails to start due to invalid usage and exits.

Here is an example that defines a count connector and uses it as a receiver in a metrics pipeline:

---
receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  logging:
connectors:
  count:

service:
  pipelines:
    traces/in:
      receivers: [otlp]
      exporters: [logging]
    metrics/out:
      receivers: [count]
      exporters: [logging]

The config.yaml passes validation:

otel-validations(:|✔) %
🤘 docker run -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.87.0 validate --config=/etc/otelcol-contrib/config.yaml

otel-validations(:|✔) %
🤘 echo $?
0

However upon using this configuration, the collector reports an error and exits:

otel-validations(:|✔) %
🤘 docker run -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.87.0
2023-10-22T21:05:36.612Z	info	service@v0.87.0/telemetry.go:84	Setting up own telemetry...
2023-10-22T21:05:36.612Z	info	service@v0.87.0/telemetry.go:201	Serving Prometheus metrics	{"address": ":8888", "level": "Basic"}
Error: failed to build pipelines: connector "count" used as receiver in metrics pipeline but not used in any supported exporter pipeline
2023/10/22 21:05:36 collector server run finished with error: failed to build pipelines: connector "count" used as receiver in metrics pipeline but not used in any supported exporter pipeline

otel-validations(:|✔) %
🤘 echo $?
1

 
Describe the solution you'd like

I expect the validator to return an error

[mx-psi]

cc @djaglowski

[djaglowski]

This problem is not specific to connectors. The validate command currently does not consider supported data types for any class of component.

That said, I agree there are additional configuration errors, such as this one, which we could surface within the command if we would try to instantiate (but not start) a service.

[ringerc]

I raised this in #8866 after missing this issue in a search (somehow).

As I noted in that issue, it'd also be extremely helpful if the connector error mentioned the specific pipeline name in its complaint, e.g logs/out instead of just logs, like this:

Error: failed to build pipelines: connector "forward/test" used as receiver in "logs/out" pipeline but not used in any supported exporter pipeline

Currently it doesn't name the actual pipeline, so in complex configurations it can be tricky to tell what exactly it's complaining about.

[robincw-gr]

Any eyes on this?
Shouldn't this have the label bug, because the validate command should definitely fail if the config prevents the collector from starting. Fixing is not an enhancement

[mx-psi]

@robincw-gr I don't think there is anyone working on this at the moment. In general there is no way to statically validate all things that would prevent the Collector from starting (think for example about a server using a port that is not available or some permissions issue). But I agree this should be considered a bug since it is statically verifiable.