Add a metric to provide missed events per type #1674

tpapagian · 2023-10-30T12:42:16Z

Example:

$ curl localhost:2112/metrics 2> /dev/null | grep 'sent_events_total\|missed_events_total\|ringbuf_perf_event_lost_total\|ringbuf_queue_lost_total\|msg_op_total\|ringbuf_queue_received_total'
# HELP tetragon_missed_events_total The total number of Tetragon events per type that are failed to sent from the kernel.
# TYPE tetragon_missed_events_total counter
tetragon_missed_events_total{msg_op="13"} 73300
tetragon_missed_events_total{msg_op="23"} 28
tetragon_missed_events_total{msg_op="24"} 606
tetragon_missed_events_total{msg_op="5"} 20
tetragon_missed_events_total{msg_op="7"} 22
# HELP tetragon_msg_op_total The total number of times we encounter a given message opcode. For internal use only.
# TYPE tetragon_msg_op_total counter
tetragon_msg_op_total{msg_op="13"} 4.268532e+06
tetragon_msg_op_total{msg_op="23"} 12444
tetragon_msg_op_total{msg_op="24"} 2110
tetragon_msg_op_total{msg_op="5"} 11908
tetragon_msg_op_total{msg_op="7"} 12447
# HELP tetragon_ringbuf_perf_event_lost_total The total number of Tetragon ringbuf perf events lost.
# TYPE tetragon_ringbuf_perf_event_lost_total counter
tetragon_ringbuf_perf_event_lost_total 73976
# HELP tetragon_ringbuf_queue_lost_total The total number of Tetragon events ring buffer queue lost.
# TYPE tetragon_ringbuf_queue_lost_total counter
tetragon_ringbuf_queue_lost_total 0
# HELP tetragon_ringbuf_queue_received_total The total number of Tetragon events ring buffer queue received.
# TYPE tetragon_ringbuf_queue_received_total counter
tetragon_ringbuf_queue_received_total 4.307441e+06

This PR adds an eBPF map collector for getting metrics directly from a map. This map contains information about the return values of all perf_event_output calls (i.e. if it fails). This provides us the ability to determine missed events per type. Metric tetragon_missed_events_total contains such information.

Using the previous example, we can see that we lost 73976 events from the user-space (tetragon_ringbuf_perf_event_lost_total). This is the same as the sum of all tetragon_missed_events_total metrics gathered from the kernel.

netlify · 2023-10-30T12:42:42Z

✅ Deploy Preview for tetragon ready!

Name	Link
🔨 Latest commit	`097b35f`
🔍 Latest deploy log	https://app.netlify.com/sites/tetragon/deploys/6540b5a1ff11ee0007cad97a
😎 Deploy Preview	https://deploy-preview-1674--tetragon.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

pkg/metrics/metricsconfig/initmetrics.go

pkg/bpfmetrics/metrics.go

lambdanis

I left some comments :) The most important one being that it looks like we want to expose two different metric names here.

bpf/lib/process.h

bpf/process/types/basic.h

pkg/metrics/eventmetrics/collector.go

Signed-off-by: Anastasios Papagiannis <tasos.papagiannnis@gmail.com>

tpapagian · 2023-10-31T08:09:29Z

I left some comments :) The most important one being that it looks like we want to expose two different metric names here.

Thanks for the review! I have made the changes that you proposed.

lambdanis

looks good to me

kkourt

Nice! I think this will help with debugging.
I have some small comments, PTAL.

bpf/cgroup/bpf_cgroup_events.h

bpf/lib/process.h

bpf/process/types/basic.h

bpf/lib/process.h

Example: $ curl localhost:2112/metrics 2> /dev/null | grep 'sent_events_total\|missed_events_total\|ringbuf_perf_event_lost_total\|ringbuf_queue_lost_total\|msg_op_total\|ringbuf_queue_received_total' tetragon_missed_events_total{msg_op="13"} 73300 tetragon_missed_events_total{msg_op="23"} 28 tetragon_missed_events_total{msg_op="24"} 606 tetragon_missed_events_total{msg_op="5"} 20 tetragon_missed_events_total{msg_op="7"} 22 tetragon_msg_op_total{msg_op="13"} 4.268532e+06 tetragon_msg_op_total{msg_op="23"} 12444 tetragon_msg_op_total{msg_op="24"} 2110 tetragon_msg_op_total{msg_op="5"} 11908 tetragon_msg_op_total{msg_op="7"} 12447 tetragon_ringbuf_perf_event_lost_total 73976 tetragon_ringbuf_queue_lost_total 0 tetragon_ringbuf_queue_received_total 4.307441e+06 This PR adds an eBPF map collector for getting metrics directly from a map. This map contains information about the return values of all perf_event_output calls (i.e. if it fails). This provides us the ability to determine missed events per type. Metric tetragon_missed_events_total contains such information. Using the previous example, we can see that we lost 73976 events from the user-space (tetragon_ringbuf_perf_event_lost_total). This is the same as the sum of all tetragon_missed_events_total metrics gathered from the kernel. Signed-off-by: Anastasios Papagiannis <tasos.papagiannnis@gmail.com>

kkourt · 2023-11-02T09:26:12Z

Thanks! I think this will be very useful to have, which is why I added a backport label for 1.0.

tpapagian · 2023-11-02T13:00:00Z

Backport PR: #1702

tpapagian added area/metrics Related to prometheus metrics release-note/minor This PR introduces a minor user-visible change labels Oct 30, 2023

tpapagian force-pushed the pr/apapag/ebpf_metrics branch 2 times, most recently from 1c5ca91 to 0f24125 Compare October 30, 2023 12:50

tpapagian changed the title ~~Add a metric to provide per-event missed events~~ Add a metric to provide missed events per type Oct 30, 2023

lambdanis reviewed Oct 30, 2023

View reviewed changes

pkg/metrics/metricsconfig/initmetrics.go Outdated Show resolved Hide resolved

pkg/bpfmetrics/metrics.go Outdated Show resolved Hide resolved

tpapagian force-pushed the pr/apapag/ebpf_metrics branch from 0f24125 to a2ab538 Compare October 30, 2023 14:35

tpapagian marked this pull request as ready for review October 30, 2023 14:53

tpapagian requested a review from a team as a code owner October 30, 2023 14:53

tpapagian requested a review from kkourt October 30, 2023 14:53

lambdanis requested changes Oct 30, 2023

View reviewed changes

Update gitignore

2a41c07

Signed-off-by: Anastasios Papagiannis <tasos.papagiannnis@gmail.com>

tpapagian force-pushed the pr/apapag/ebpf_metrics branch from a2ab538 to 097b35f Compare October 31, 2023 08:06

lambdanis approved these changes Oct 31, 2023

View reviewed changes

kkourt reviewed Nov 1, 2023

View reviewed changes

bpf/cgroup/bpf_cgroup_events.h Outdated Show resolved Hide resolved

bpf/lib/process.h Outdated Show resolved Hide resolved

bpf/process/types/basic.h Outdated Show resolved Hide resolved

tpapagian force-pushed the pr/apapag/ebpf_metrics branch 2 times, most recently from 1c293ad to f294f5d Compare November 1, 2023 09:13

olsajiri reviewed Nov 1, 2023

View reviewed changes

bpf/lib/process.h Outdated Show resolved Hide resolved

tpapagian force-pushed the pr/apapag/ebpf_metrics branch 3 times, most recently from ad0ea1c to d1c041a Compare November 1, 2023 09:38

tpapagian force-pushed the pr/apapag/ebpf_metrics branch from d1c041a to 19778a4 Compare November 1, 2023 10:30

olsajiri approved these changes Nov 1, 2023

View reviewed changes

kkourt approved these changes Nov 2, 2023

View reviewed changes

kkourt added the needs-backport/1.0 This PR needs backporting to 1.0 label Nov 2, 2023

kkourt merged commit d5a7ee2 into main Nov 2, 2023

kkourt deleted the pr/apapag/ebpf_metrics branch November 2, 2023 09:26

tpapagian added backport-pending/1.0 and removed needs-backport/1.0 This PR needs backporting to 1.0 labels Nov 2, 2023

tpapagian added backport-done/1.0 and removed backport-pending/1.0 labels Nov 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a metric to provide missed events per type #1674

Add a metric to provide missed events per type #1674

tpapagian commented Oct 30, 2023 •

edited

Loading

netlify bot commented Oct 30, 2023 •

edited

Loading

lambdanis left a comment

tpapagian commented Oct 31, 2023

lambdanis left a comment

kkourt left a comment

kkourt commented Nov 2, 2023

tpapagian commented Nov 2, 2023

Add a metric to provide missed events per type #1674

Add a metric to provide missed events per type #1674

Conversation

tpapagian commented Oct 30, 2023 • edited Loading

netlify bot commented Oct 30, 2023 • edited Loading

✅ Deploy Preview for tetragon ready!

lambdanis left a comment

Choose a reason for hiding this comment

tpapagian commented Oct 31, 2023

lambdanis left a comment

Choose a reason for hiding this comment

kkourt left a comment

Choose a reason for hiding this comment

kkourt commented Nov 2, 2023

tpapagian commented Nov 2, 2023

tpapagian commented Oct 30, 2023 •

edited

Loading

netlify bot commented Oct 30, 2023 •

edited

Loading