USDT/BPF Tracing Tools: Java, Python, Ruby, Node, MySQL, PostgreSQL

A lot of high-level languages, runtimes, and libraries people use on linux haveUSDT probes embedded in them. In some cases, you have to compile with a specific flag to get the probes embedded (e.g. in Node.js), and in other cases they are part of the default package on most major distributions (e.g. in OpenJDK).Some examples of information these probes can provide include:

Garbage collection events and latenciesin Java and Node Method invocations and latencies in python and Ruby Object allocations in Ruby and Java Thread start and stop events in Java Class load events in Java and Ruby

This information can be used for a variety of investigations in the field, as well as performance monitoring of a running production system.

If you havean application or runtime instrumented with USDT probes, there are a bunch of tools in the BCC project that you can now use to get low-overhead,safe, andpowerfulinformation from a production system (granted that you have a recent enough kernel to use BPF). A couple of days ago I was able to get several new tools merged , and it’s time to describe a few of them along with examples and usage scenarios. Let’s go!

Raw Probes: tplist, trace, and argdist

If you know exactly which probes you care about, and are willing to do the legwork of correlating trace output yourself, you can use the multi-tooltracing trio: tplist, trace, and argdist. All three tools support USDT probes. For example, you can use tplist to get alist of the probes embedded into a running Node process:

# tplist -p $(pgrep -n node)
../node node:gc__start
../node node:gc__done
../node node:http__server__request
../node node:http__server__response
<...more output omitted for brevity...>

To figure out the format of the probe, you can apply tplist again to see the number and arguments and their types, but you will often need to consult the runtime’s source to see what the arguments stand for. Specifically, for the http__server__request probe in Node, the probe’s arguments include the requested URL, the client’s IP address, the HTTP method, and some other details.

Next, you can instrument probes you’re interested in, for example to monitor URLs that are frequently accessed by clients, with the argdist multi-tool:

# argdist -C 'u:/opt/bin/node:http__server__request():char*:arg6' -p $(pgrep -n node)
[13:51:38]
u:/opt/bin/node:http__server__request():char*:arg6
COUNT EVENT
123 arg6 = /index.html
11 arg6 = /favicon.ico
1 arg6 = /
^C

… or to print a trace message whenever there is a garbage collection in the Node process with the trace multi-tool:

# trace 'u:/opt/bin/node:gc__start' -p $(pgrep -n node) -T
TIME PID TID COMM FUNC
13:56:54 7333 7333 node gc__start
^C

I especially love trace because it’s essentially a dynamic logger. I can attach it to arbitrary locations in the kernel or in user-space processes, and get live logs when interesting things happen ― with minimaloverhead. But I digress.

Java, Python, Ruby, Node: u* Tools

Learning the different kinds of probes that are available in different runtimes and then tracing and correlating this information can be a bit tedious. This is why I wrote a set of tools (inspired by Brendan Gregg’s workon the DTrace Toolkit for Solaris) that help trace GC latency, method calls andflow (including syscalls), thread start and stop events, and other interesting statistics and metrics for high-levellanguage runtimes.These tools have just recently been merged into the BCC project,and should work on all Linux kernels after 4.3.

For example, let’s trace garbage collections in a Java process:

# ugc -m java $(pidof java)
Tracing garbage collections in java process 1241... Ctrl-C to quit.
START DESCRIPTION TIME (ms)
1.831 PS Scavenge PS Eden Space ... 50.00
2.005 PS Scavenge PS Old Gen ... 19.00
^C

Or, trace method calls in a Python script:

# ucalls -L -l python $(pidof python)
Tracing calls in process 4049 (language: python)... Ctrl-C to quit.
^C
METHOD # CALLS TIME (us)
./runner.py.something_else 42 2017394.35
./runner.py.something 17 1011.38

Or, count syscall frequencies and latencies inthe same process:

# ucalls -LS 4923
Attached 742 kernel probes for syscall tracing.
Tracing calls in process 4923 (language: none)... Ctrl-C to quit.
^C
METHOD # CALLS TIME (us)
sys_select 42 1998941.02

Or, trace Java thread creation and destruction events:

# uthreads -l java 27420
Tracing thread events in process 27420 (language: java)... Ctrl-C to quit.
TIME ID TYPE DESCRIPTION
18.596 R=9/N=0 start SIGINT handler
18.596 R=4/N=0 stop Signal Dispatcher
^C

Or, trace Ruby object allocations in a REPL process:

# uobjnew ruby 27245
Tracing allocations in process 27245 (language: ruby)... Ctrl-C to quit.
^C
TYPE # ALLOCS # BYTES
NameError 1 0
RubyToken::TkSPACE 1 0
RubyToken::TkSTRING 1 0
String 7 0
RubyToken::TkNL 2 0
RubyToken::TkIDENTIFIER 2 0
array 55 129
string 344 1348

Or even trace method flow in a Rubyprocess, including filtering capabilities (this also works for Python OOTB and for Java with the -XX:+ExtendedDTraceProbes flag):

# uflow ruby 27245
Tracing method calls in ruby process 27245... Ctrl-C to quit.
CPU PID TID TIME(us) METHOD
3 27245 27245 4.536 <- IO.gets
3 27245 27245 4.536 <- IRB::StdioInputMethod.gets
3 27245 27245 4.536 -> IRB::Context.verbose?
3 27245 27245 4.536 -> NilClass.nil?
3 27245 27245 4.536 <- NilClass.nil?
3 27245 27245 4.536 -> IO.tty?
3 27245 27245 4.536 <- IO.tty?
3 27245 27245 4.536 -> Kernel.kind_of?
3 27245 27245 4.536 <- Kernel.kind_of?
3 27245 27245 4.536 <- IRB::Context.verbose?
3 27245 27245 4.536 <- IRB::Irb.signal_status
3 27245 27245 4.536 -> String.chars
3 27245 27245 4.536 <- String.chars
^C

Or just monitor interesting events across all processes onthe system that have one of the supported runtimes:

# ustat -C 5
Tracing... Output every 5 secs. Hit Ctrl-C to end
12:18:26 loadavg: 0.27 0.11 0.04 2/336 26455
PID CMDLINE METHOD/s GC/s OBJNEW/s