Locating system problems using dynamic instrumentation

It is often difficult to diagnose complex problems without multiple rebuilds and reboots. Even in a simple setup, the problem can touch various layers of the application and operating system. Diagnosis is even more difficult in complex, multi-tiered systems. As Linux is deployed in these environments, it is becoming more important to have facilities to locate and identify such problems.

Using the kprobes infrastructure, SystemTAP is being developed to dynamically instrument the kernel and user applications. SystemTAP instrumentation incurs low overhead when enabled, and zero overhead when disabled. SystemTAP provides facilities to define instrumentation points in a high-level language, and to aggregate and analyze the instrumentation data. Details of the SystemTAP architecture and implementation are presented, along with examples of solving problems in the production environments.