Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Getting Certificate issue with google cloud exporter #49

Closed
dashpole opened this issue Oct 5, 2023 · 4 comments
Closed

Getting Certificate issue with google cloud exporter #49

dashpole opened this issue Oct 5, 2023 · 4 comments

Comments

@dashpole
Copy link
Contributor

dashpole commented Oct 5, 2023

Component(s)

exporter/googlecloud

What happened?

Description

I am running this configuration

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  googlecloud:
    impersonate:
      target_principal: otel-collector@arjun-demo-123.iam.gserviceaccount.com


processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 65
    spike_limit_percentage: 20
  batch:
  resourcedetection:
    detectors: [gcp]
    timeout: 10s
service:
  pipelines:
#    traces:
#      receivers: [otlp]
#      processors: [memory_limiter, batch]
#      exporters: [googlecloud]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [googlecloud]
#    logs:
#      receivers: [otlp]
#      processors: [memory_limiter, batch]
#      exporters: [googlecloud]

getting this in collector logs

2023-10-04T15:41:53.565Z warn zapgrpc/zapgrpc.go:195 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "monitoring.googleapis.com:443", ServerName: "monitoring.googleapis.com:443", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority" {"grpc_log": true}

Steps to Reproduce

Run the custom collector https://github.com/GoogleCloudPlatform/opentelemetry-collector-builder-sample

Expected Result

To send metrics to GCP

Actual Result

2023-10-04T15:41:53.565Z warn zapgrpc/zapgrpc.go:195 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "monitoring.googleapis.com:443", ServerName: "monitoring.googleapis.com:443", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority" {"grpc_log": true}

Collector version

v0.86.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

Copyright 2022 Google LLC

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

FROM golang:latest as build
ARG OTEL_VERSION=0.86.0
WORKDIR /app
COPY . .
RUN go install go.opentelemetry.io/collector/cmd/builder@v${OTEL_VERSION}
RUN builder --config=builder-config.yaml

FROM ubuntu:latest
COPY --from=build /app/bin/otelcol-custom /

4317 - default OTLP receiver

55678 - opencensus (tracing) receiver

55679 - zpages

EXPOSE 4317/tcp 55678/tcp 55679/tcp

#RUN ls
#RUN chmod +x /otelcol-custom

ENTRYPOINT ["/otelcol-custom"]

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  googlecloud:
    impersonate:
      target_principal: otel-collector@arjun-demo-123.iam.gserviceaccount.com


processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 65
    spike_limit_percentage: 20
  batch:
  resourcedetection:
    detectors: [gcp]
    timeout: 10s
service:
  pipelines:
#    traces:
#      receivers: [otlp]
#      processors: [memory_limiter, batch]
#      exporters: [googlecloud]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [googlecloud]
#    logs:
#      receivers: [otlp]
#      processors: [memory_limiter, batch]
#      exporters: [googlecloud]


### Log output

```shell
2023-10-04T15:41:25.156Z        info    memorylimiterprocessor@v0.86.0/memorylimiter.go:102     Memory limiter configured       {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics", "limit_mib": 332, "spike_limit_mib": 102, "check_interval": 1}
2023-10-04T15:41:25.195Z        info    service@v0.86.0/service.go:138  Starting otelcol-custom...      {"Version": "1.0.0", "NumCPU": 4}
2023-10-04T15:41:25.195Z        info    extensions/extensions.go:31     Starting extensions...
2023-10-04T15:41:25.195Z        warn    internal@v0.86.0/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks       {"kind": "receiver", "name": "otlp", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-10-04T15:41:25.196Z        info    otlpreceiver@v0.86.0/otlp.go:83 Starting GRPC server    {"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:4317"}
2023-10-04T15:41:25.196Z        warn    internal@v0.86.0/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks       {"kind": "receiver", "name": "otlp", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-10-04T15:41:25.196Z        info    otlpreceiver@v0.86.0/otlp.go:101        Starting HTTP server    {"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:4318"}
2023-10-04T15:41:25.196Z        info    service@v0.86.0/service.go:161  Everything is ready. Begin running and processing data.
2023-10-04T15:41:25.210Z        warn    zapgrpc/zapgrpc.go:195  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "monitoring.googleapis.com:443", ServerName: "monitoring.googleapis.com:443", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"    {"grpc_log": true}
2023-10-04T15:41:26.225Z        warn    zapgrpc/zapgrpc.go:195  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "monitoring.googleapis.com:443", ServerName: "monitoring.googleapis.com:443", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"    {"grpc_log": true}
2023-10-04T15:41:27.959Z        warn    zapgrpc/zapgrpc.go:195  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "monitoring.googleapis.com:443", ServerName: "monitoring.googleapis.com:443", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"    {"grpc_log": true}
2023-10-04T15:41:31.054Z        warn    zapgrpc/zapgrpc.go:195  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "monitoring.googleapis.com:443", ServerName: "monitoring.googleapis.com:443", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"    {"grpc_log": true}
2023-10-04T15:41:34.595Z        warn    zapgrpc/zapgrpc.go:195  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "monitoring.googleapis.com:443", ServerName: "monitoring.googleapis.com:443", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"    {"grpc_log": true}
2023-10-04T15:41:41.999Z        warn    zapgrpc/zapgrpc.go:195  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "monitoring.googleapis.com:443", ServerName: "monitoring.googleapis.com:443", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"    {"grpc_log": true}
2023-10-04T15:41:53.565Z        warn    zapgrpc/zapgrpc.go:195  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "monitoring.googleapis.com:443", ServerName: "monitoring.googleapis.com:443", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"    {"grpc_log": true}

Additional context

No response
@dashpole
Copy link
Contributor Author

dashpole commented Oct 5, 2023

Originally in open-telemetry/opentelemetry-collector-contrib#27426

@dashpole
Copy link
Contributor Author

dashpole commented Oct 6, 2023

@arjunvijaygoogle can you see if this can be reproduced after #48?

@arjunvijaygoogle
Copy link

arjunvijay@cloudshell:~/opentelemetry-collector-builder-sample/deploy/gke/simple (arjun-demo-123)$ kubectl logs deployment/otel-collector -n otel-collector
2023-10-08T05:45:36.821Z info service@v0.86.0/telemetry.go:84 Setting up own telemetry...
2023-10-08T05:45:36.902Z info service@v0.86.0/telemetry.go:201 Serving Prometheus metrics {"address": ":8888", "level": "Basic"}
2023-10-08T05:45:36.909Z info exporter@v0.86.0/exporter.go:275 Deprecated component. Will be removed in future releases. {"kind": "exporter", "data_type": "logs", "name": "logging"}
2023-10-08T05:45:36.911Z warn common/factory.go:68 'loglevel' option is deprecated in favor of 'verbosity'. Set 'verbosity' to equivalent value to preserve behavior. {"kind": "exporter", "data_type": "logs", "name": "logging", "loglevel": "debug", "equivalent verbosity level": "Detailed"}
2023-10-08T05:45:36.911Z info exporter@v0.86.0/exporter.go:275 Deprecated component. Will be removed in future releases. {"kind": "exporter", "data_type": "traces", "name": "logging"}
2023-10-08T05:45:36.911Z info memorylimiterprocessor@v0.86.0/memorylimiter.go:138 Using percentage memory limiter {"kind": "processor", "name": "memory_limiter", "pipeline": "traces", "total_memory_mib": 512, "limit_percentage": 65, "spike_limit_percentage": 20}
2023-10-08T05:45:36.912Z info memorylimiterprocessor@v0.86.0/memorylimiter.go:102 Memory limiter configured{"kind": "processor", "name": "memory_limiter", "pipeline": "traces", "limit_mib": 332, "spike_limit_mib": 102, "check_interval": 1}
2023-10-08T05:45:36.913Z info exporter@v0.86.0/exporter.go:275 Deprecated component. Will be removed in future releases. {"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-10-08T05:45:36.925Z info service@v0.86.0/service.go:138 Starting otelcol-custom... {"Version": "1.0.0", "NumCPU": 2}
2023-10-08T05:45:36.926Z info extensions/extensions.go:31 Starting extensions...
2023-10-08T05:45:36.926Z info internal/resourcedetection.go:125 began detecting resource information {"kind": "processor", "name": "resourcedetection/gke", "pipeline": "traces"}
2023-10-08T05:45:36.929Z info internal/resourcedetection.go:139 detected resource information {"kind": "processor", "name": "resourcedetection/gke", "pipeline": "traces", "resource": {"cloud.account.id":"arjun-demo-123","cloud.platform":"gcp_kubernetes_engine","cloud.provider":"gcp","cloud.region":"us-east1","host.id":"4919498455003402556","host.name":"gk3-builder-bug-default-pool-3bfbf47f-d00h","k8s.cluster.name":"builder-bug"}}
2023-10-08T05:45:36.930Z warn internal@v0.86.0/warning.go:40 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks {"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-10-08T05:45:36.930Z info otlpreceiver@v0.86.0/otlp.go:83 Starting GRPC server {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4317"}
2023-10-08T05:45:36.930Z warn internal@v0.86.0/warning.go:40 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks {"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-10-08T05:45:36.930Z info otlpreceiver@v0.86.0/otlp.go:101 Starting HTTP server {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4318"}
2023-10-08T05:45:36.931Z info prometheusreceiver@v0.86.0/metrics_receiver.go:239 Starting discovery manager {"kind": "receiver", "name": "prometheus/self", "data_type": "metrics"}
2023-10-08T05:45:36.931Z info prometheusreceiver@v0.86.0/metrics_receiver.go:230 Scrape job added {"kind": "receiver", "name": "prometheus/self", "data_type": "metrics", "jobName": "otel-collector"}
2023-10-08T05:45:36.931Z info service@v0.86.0/service.go:161 Everything is ready. Begin running and processing data.
2023-10-08T05:45:36.932Z info prometheusreceiver@v0.86.0/metrics_receiver.go:281 Starting scrape manager {"kind": "receiver", "name": "prometheus/self", "data_type": "metrics"}

@arjunvijaygoogle
Copy link

@dashpole The issue is fixed. Thanks for the extraordinary support. Really appreciate your help

@dashpole dashpole closed this as completed Oct 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dashpole @arjunvijaygoogle