Efficient Dynamic Tracepoints in Kernel and User-SpaceRafik Fahem
With in kernel static tracepoints through TRACE_EVENT, and user-space tracepoints through the UST library, it is now possible to obtain a low overhead trace of the whole system execution. There are cases, however, where dynamic tracepoints are required to hunt an elusive problem possibly performance related. A natural interface to navigate through the source code and dynamically add a tracepoint is the debugger in an integrated development environment. GDB was recently enhanced to support user-space dynamic tracepoints, with support from the Debugger Service Framework in Eclipse. GDB can also list and activate the static user-space tracepoints normally used with the UST library. Once started, the trace is collected with minimal overhead in buffers without intervention from GDB or the operating system. With each tracepoint may be associated a condition expression and action statements to collect tracing data. The condition and action are converted by GDB into a simple bytecode to be interpreted in the target program. Furthermore, the bytecode for the condition may be converted to native code for more performance.
More recently, a new kernel module for Kernel GDB TracePoints (KGTP) was submitted as contribution to the kernel. It uses kprobes to insert dynamic tracepoints, implements the GDB remote serial protocol for tracepoints, and can interpret the bytecode used by GDB to define conditions and actions. We have extended the bytecode to native code converter from user-space conditions to conditions and actions in kernel and in user-space. It is now also possible to list and activate the kernel static tracepoints from within GDB, just like in user-space. In the proposed paper and presentation, these new capabilities and in particular our extensions will be detailed. The performance of these static and dynamic tracepoints, with bytecode or native code conditions and actions, has been measured and will be presented and discussed.