Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Particular hostmetrics export failure: The start time must be before the end time #3278

Closed
tmodelsk opened this issue Apr 29, 2021 · 3 comments
Labels
bug Something isn't working

Comments

@tmodelsk
Copy link

tmodelsk commented Apr 29, 2021

Describe the bug
Particular metrics (sum metrics) that are from hostmetrics receiver cannot be exported to google monitoring api.
The error message is: "The start time must be before the end time"
Affected hostmetrics are:

  • process.memory.physical_usage
  • process.memory.virtual_usage
  • system.disk.pending_operations
  • system.processes.count
  • system.processes.created
  • system.memory.usage
  • system.filesystem.usage
  • system.filesystem.inodes.usage
  • system.network.connections
  • system.paging.usage
  • system.cpu.load_average.1m
  • system.cpu.load_average.5m
  • system.cpu.load_average.15m

Other hostmetrics (ie: system.cpu.time) are properly exported.

Further investigations (debug & file exporter log) shows that all affected metrics:

  • don't have startTimeUnixNano field (file exporter log)
  • field StartTimestamp is 0 (debugging, pdata.Metrics data structure)
  • simple workaround described at the very bottom

Steps to reproduce
Create metrics pipeline with hostmetrics receiver and googlecloud exporter

What did you expect to see?
All hostmetrics values should be exported to monitoring api, given metrics values should be soon visible in metrics explorer GUI and queryable by rest api https://monitoring.googleapis.com/v3/projects/PROJECT_ID/timeSeries

What did you see instead?
Errors in otel output console are like:
rpc error: code = InvalidArgument desc = Field timeSeries[3].points[0].interval.start_time had an invalid value of \"2021-04-26T04:17:28.120024-07:00\": The start time must be before the end time (2021-04-26T04:17:28.120024-07:00) for the non-gauge metric 'external.googleapis.com/prometheus/process.memory.physical_usage'.;

Values of metrics mentioned above are not exported to google monitoring.
Only some hostmetrics values (ie: system.cpu.time) are there.

What version did you use?
Release versions 0.24.0 both for opentelemetry-collector-builder & opentelemetry-collector-contrib/exporter/googlecloudexporter

What config did you use?
Otel config:

receivers:
  hostmetrics: 
    collection_interval: 30s
    scrapers:
      cpu:
      disk:
      load:
      filesystem:
      memory:
      network:
      paging:
      processes:
      process:
processors: 
  resource: 
    attributes:
      # gke settings   
exporters:
  googlecloud:
    project: PROJECT_ID
    metric:
      prefix: "external.googleapis.com/prometheus"
      skip_create_descriptor: false
service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [resource]
      exporters: [googlecloud]

Additional context
Simple working workaround (assign missing StartTimestamp := Timestamp - 1microSecond) is here:
SabreOSS@d6795c9
With this one all hostmetrics values are exported.

@tmodelsk tmodelsk added the bug Something isn't working label Apr 29, 2021
@bogdandrutu
Copy link
Member

@jsuereth what do you think about this?

@jsuereth
Copy link
Contributor

Sorry I missed this comment. The OpenTelemetry GCP metrics exporter for Go does do the suggested fix, but the current collector exporter is based on OpenCensus where that was unnecessary. This should be considered a bug against the Google Cloud exporters in the collector.

@dashpole
Copy link
Contributor

I think this should be fixed by #3360

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants