Improving the Approach to Linux Performance Analysis

In-depth Linux kernel performance analysis and debugging historically has been a focus which required resident kernel hackers. This effort not only required deep kernel knowledge to analyze the code, but also required programming skills in order to modify the code, re-build the kernel, re-boot and extract and format the information from the system. A variety of powerful Linux system tools are emerging which provide significantly more flexibility for the analyst and the system owner. This paper highlights an example set of performance problems which are often seen on a system, and focuses on the methodology, approach, and steps that can be used to address each problem. The paper will also provide examples of a set of pre-defined SystemTAP "tapsets" which make it easier for the non-programmer to extract information about kernel events from a running system, effectively tracing some of the kernel behavior. Comparing and contrasting available tools such as SystemTAP, kprobes, oprofile, and hardware counter performance tools will help provide system owners and developers with real-life examples which can be used on their systems today, in addition to providing insights into how current tools can improve and evolve in the future.