Add support for USDT probes

[vchuravy]

Based on a previous exploration with @NHDaly in #31616, this PR adds a infrastructure support for USDT,
supporting dtrace on MacOS and bpftrace and related tools on Linux. Support is gate by a new Make flag.

As an example I added some GC probes that allow one to measure GC latencies, as an example in #41616 I was interested
in the latency distribution of stop-the-world, e.g. how long it takes until GC starts with productive work.

I hope this can provide visibilty into issues like #41586 and #33097

Inspired by https://docs.python.org/3/howto/instrumentation.html

[vchuravy]

@jpsamaroo might be a cool example for BPFnative.jl?

[Sacha0]

Exciting! Thanks Valentin! :)

[NHDaly]

Yeah agreed, thanks for picking this up again, Valentin!!

[jpsamaroo]

These are working on Linux (musl) with bpftrace and BPFnative!

[Sacha0]

This looks terrific! Thanks Valentin! :)

[jpsamaroo]

Any chance we could also get probes on task switching? This could be a good, temporary alternative to #39994, where hook execution can be disabled via Preferences or some other mechanism per-package.

[vchuravy]

For Julia code side probes there is https://github.com/JuliaPerf/UProbes.jl, but that is hamstrung by the lack of JIT support in tools bpftrace/bpftrace#425

[vchuravy]

Any chance we could also get probes on task switching

I think adding probes on task switching shouldn't be too hard.

One thing I haven't tested in a while is ustack support e.g. if we get a correctly demangled stacktraces inside the probe.

This could be a good, temporary alternative to #39994, where hook execution can be disabled via Preferences or some other mechanism per-package.

Uhm... I don't quite understand what you are thinking about, but let's not go down the path of using eBPF programs to modify user-space on task-switches.

[jpsamaroo]

Uhm... I don't quite understand what you are thinking about, but let's not go down the path of using eBPF programs to modify user-space on task-switches.

I'm thinking of using it to collect metrics about executing tasks, like being able to calculate the actual CPU execution time of a task (since clock_gettimeis per-thread, we need to exclude other concurrently executing tasks).

[vchuravy]

Wanna give it a go to try adding probes for that?

[carlocab]

By the way, building from master with WITH_DTRACE=1 fails for me on macOS with the error

 dtrace -h -s /private/tmp/julia-20210907-69372-bfig8y/src/uprobes.d -o uprobes.h.gen
dtrace: failed to compile script /private/tmp/julia-20210907-69372-bfig8y/src/uprobes.d: line 5: syntax error near "probe"
make[2]: *** [uprobes.h.gen] Error 1

I guess uprobes.d needs to be slightly different for macOS dtrace. Apologies for digging up this PR to post this, but I wasn't sure if this was worth tracking in a separate issue. Happy to open a new one if it is.