[Sketch] OS-level tracing

[eugeneia]

This is my working branch for exploring OS-level tracing capabilities. So far it includes:

• USDT probe support for Linux (via SystemTap)

This adds a build dependency (SystemTap) on Linux systems, so it might make sense to put this behind a flag. IMHO the (build-only) dependency is worth its weight and USDT probes should be there by default. What this gives you is that you can attach to probes (these were already defined for use with DTrace on Darwin) of running CCL processes via BCC’s trace:

# trace -p 2570 \
      'u:ccl:egc__start "used %d, gen %d", arg1, arg2' \
      'u:ccl:egc__finish "freed %d, gen %d", arg1, arg2' \
      'u:ccl:gc__start "used %d", arg1' \
      'u:ccl:gc__finish "freed %d", arg1'
> PID    TID    COMM         FUNC             -
2570   3469   lx86cl64     egc__start       used 2097152, gen 0
2570   3469   lx86cl64     egc__finish      freed 1514896, gen 0
2570   3484   lx86cl64     egc__start       used 2170256, gen 0
2570   3484   lx86cl64     egc__finish      freed 2065552, gen 0
2570   3499   lx86cl64     egc__start       used 2196624, gen 0
2570   3499   lx86cl64     egc__finish      freed 2090240, gen 0
2570   3514   lx86cl64     egc__start       used 2221312, gen 0
2570   3514   lx86cl64     egc__finish      freed 2116432, gen 0
...
2570   2571   lx86cl64     gc__start        used 55574528
2570   2571   lx86cl64     gc__finish       freed 32111648
2570   2571   lx86cl64     egc__start       used 2227232, gen 0
2570   2571   lx86cl64     egc__finish      freed 0, gen 0
...

• PERF-HELPER, a mechanism to export symbol map files for use with Linux’s perf(1). The code/magic for this module was provided by Gary Byers, and worked out of the box, credit to him! In a nutshell it allows you to dump a map of the current heap:

(require "PERF-HELP")
(ccl:create-perf-map) ; will create "/tmp/perf-<PID>.map"

which allows perf(1) to reconstruct stack traces of a running CCL process. Now when you profile CCL via perf:

perf record -F 999 -p <PID> -g -- sleep 20 # Sample at 999 Hz for 20 seconds
perf report # labels stack frames using /tmp/perf-<PID>.map

Or instead of using perf report generate fancy SVG flame graphs, see http://www.brendangregg.com/flamegraphs.html :-)

If you look at the flame graph you can see that there are still a bunch of “unknown” frames. Some of these are kernel frames, and merely unresolved because I ran the example as a regular user, but some are indeed frames that point into the CCL process but can’t be resolved via the perf map. That’s a FIXME.

TODO

• Add more USDT probes: there already exist probes for GC/EGC events as well as thread startup. There should be more events in CCL that make for interesting probes, suggestions welcome.