Add Prometheus Dashboard

[sorantis]

In an effort to increase our dashboard coverage for all modules (#10594) I've created a Prometheus dashboard that shows the following metrics:

• prometheus_http_requests_total: Counter of HTTP requests.
• prometheus_engine_query_duration_seconds: Query timings for (99th percentile).
• tsdb_wal_truncations_failed_total: Total number of WAL truncations that failed.
• prometheus_tsdb_wal_corruptions_total: Total number of WAL corruptions.
• prometheus_tsdb_reloads_total: Number of times the database reloaded block data from disk.
• prometheus_tsdb_reloads_failures_total: Number of times the database failed to reload block data from disk.
• prometheus_sd_discovered_targets: Current number of discovered targets.
• prometheus_tsdb_head_chunks: Total number of chunks in the head block.
• prometheus_api_remote_read_queries: The current number of remote read queries being executed or waiting.
• prometheus_notifications_queue_capacity: The capacity of the alert notifications queue.
• prometheus_notifications_queue_length: The number of alert notifications in the queue.

The dashboards looks like this:

[exekias]

Thank you for opening this! I'm curious, it looks like you are graphing totals for http requests and head chunks. Would it make sense to calculate the rates?

Can you add a changelog entry to CHANGELOG.next.asciidoc?

Tests are failing because of docs, that should be fixed by running make update in metricbeat folder

[sorantis]

The prometheus_tsdb_head_chunks metric used for showing head chunks is a gauge. AFAIK the rate are not used on gauges. We could show rates for the HTTP total requests though.

Thanks for the tip, I'll update the changelog and build docs.

[exekias]

The prometheus_tsdb_head_chunks metric used for showing head chunks is a gauge. AFAIK the rate are not used on gauges. We could show rates for the HTTP total requests though.

Indeed, sorry I didn't check prometheus_tsdb_head_chunks, that's fine then 👍

[exekias]

Docs were removed on this commit: 48cd11f could you add them back?

[kaiyan-sheng]

travis-ci finally passed after a lot of retries 😂

[kaiyan-sheng]

jenkins, test this please

[kaiyan-sheng]

LGTM, I manually loaded this dashboard and it works.