Scaling Linux to the Extreme

Ray Bryant, Jesse Barnes, John Hawkes, Jeremy Higdon, and Jack Steiner, Silicon Graphics, Inc.

In January 2003, SGI announced the SGI® Altix(tm) 3000 family of servers. As announced, the SGI Altix 3000 system supported up to 64 Intel Itanium® 2 processors and 512GB of main memory in a single Linux image. At OLS 2003, we discussed many of the changes to Linux that allowed us to scale Linux to 64 processors for our HPC workloads.

Since then, we have continued our scalability work, and we now support up to 256 processors in a single Linux image, and we have a few early-adopter customers who are running 512 processors or as much as 4 terabytes of memory in a single system image. As can be imagined, the type of changes necessary to get a single Linux system to scale on a 512 processor, 4 terabyte system are of a different nature than those necessary to get Linux to scale up to a 64 processor system, and the majority of this talk will describe such problems. Examples of problems that had to be solved include global variables that were updated at each timer tick or generic interrupt, resulting in lots of useless cache thrashing; accesses to certain per-CPU and per-node data structures that result in TLB thrashing; lots of fixes to the page-cache allocation code; scalability problems in mmap(); I/O for very large configurations; changes in support for hugetlb pages; and the ever popular CPU scheduler issues.

While much of this work has been done in the context of a Linux 2.4 kernel, Altix is now a supported platform in the Linux 2.6 series, and our plan is to port many of these changes to Linux 2.6 and propose them as enhancments to the community kernel. While some of these fixes will be unique to the Linux kernel for Altix, many of the changes we propose will also improve performance on smaller SMP systems, so should be of general interest to the Linux scalability community.

In this talk, we will provide a brief review of the SGI Altix 3000 hardware and discuss the Linux changes that were necessary to achieve good (near-linear) scalability for high-performance computing (HPC) applications on this system. We will describe changes that were required to get Altix to boot using a standard Linux 2.6 kernel. We will also discuss changes that improved scalability for more general workloads including changes for high-performance I/O. Plans for submitting these changes to the Linux community for incorporation in standard Linux kernels will be discussed.