KafkaReceiver "failed to unmarshal message" "illegal wiretype"

[acar-ctpe]

Component(s)

receiver/kafka

Description

Unable to get kafka receiver to work with a valid endpoint/topic

Steps to Reproduce

• Deploy collector 0.95.0 with kafka receiver

Collector version

0.95.0

Environment information

Environment

Collector: Kubernetes 1.23 helm chart

OpenTelemetry Collector configuration

mode: "deployment"
config:
  receivers:
    kafka/metrics:
      brokers:
        - XYZ:9092
      client_id: ObservabilityOpentelemetryCollector-metrics
      topic: OpenTelemetry_Metrics
      encoding: otlp_proto
      auth:
        sasl:
          username: ${env:XYZ}
          password: ${env:CYZ}
          mechanism: SCRAM-SHA-256
        tls:
          insecure: false
          insecure_skip_verify: true

Log output

2024-03-11T09:27:03.951Z	info	kafkareceiver@v0.95.0/kafka_receiver.go:514	Starting consumer group	{"kind": "receiver", "name": "kafka/metrics", "data_type": "metrics", "partition": 2}
2024-03-11T09:27:03.952Z	info	kafkareceiver@v0.95.0/kafka_receiver.go:514	Starting consumer group	{"kind": "receiver", "name": "kafka/metrics", "data_type": "metrics", "partition": 18}
2024-03-11T09:27:03.952Z	info	kafkareceiver@v0.95.0/kafka_receiver.go:514	Starting consumer group	{"kind": "receiver", "name": "kafka/metrics", "data_type": "metrics", "partition": 20}
2024-03-11T09:27:12.937Z	error	kafkareceiver@v0.95.0/kafka_receiver.go:544	failed to unmarshal message	{"kind": "receiver", "name": "kafka/metrics", "data_type": "metrics", "error": "proto: illegal wireType 6"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kafkareceiver.(*metricsConsumerGroupHandler).ConsumeClaim
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kafkareceiver@v0.95.0/kafka_receiver.go:544
github.com/IBM/sarama.(*consumerGroupSession).consume
	github.com/IBM/sarama@v1.42.2/consumer_group.go:952
github.com/IBM/sarama.newConsumerGroupSession.func2

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Related: #5908, #16078, #22934

It looks like the data coming in must not match specification, but I haven't gotten a chance to see why.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[maxdidkivskyi]

Got the same issue otel/opentelemetry-collector-contrib:0.102.1:

2024-06-10T16:43:37.752Z        error   kafkareceiver@v0.102.0/kafka_receiver.go:643    failed to unmarshal message     {"kind": "receiver", "name": "kafka", "data_type": "logs", "error": "proto: wrong wireType = 2 for field TimeUnixNano"}

What do I do wrong? My configuration is:
Exporter:

  kafka:
    protocol_version: "2.0.0"
    brokers: 
      - kafka.local:9092
    topic: telemetry
    client_id: exporter.local
    encoding: otlp_proto
    partition_traces_by_id: true
    producer: 
      max_message_bytes: 1048576 
      compression: none
    timeout: 5s

Receiver:

  kafka:
    protocol_version: "2.0.0"
    brokers: 
      - kafka.local:9092
    topic: telemetry
    encoding: otlp_proto
    initial_offset: earliest
    client_id: receiver.local
    group_id: otel-collector
    message_marking:
      after: true

Used telemetry signals: spans, metrics.

It seems it's some fluky issue, because sometimes it consumes and parses message successfully.

IDK if it's matter, I use the tracegen tool to generate spans.

[maxdidkivskyi]

Okay, I understand my mistake now. I expected to deliver all signals (metrics, logs, spans) through a single topic, but unfortunately, that doesn't work. By the way, I didn't see this limitation mentioned in the documentation for either the Kafka exporter or the Kafka receiver.

[crobert-1]

Thanks for following up @maxdidkivskyi! That's very helpful to know. I've filed #33678 to hopefully clarify this.