1999 Linux Symposium

hoser ftpd


1999 home

Ninja Adventures in the Stratosphere, or Two Attempts at Server Optimization

Zach Brown, Red Hat Software, Ltd.
Philip Schwan, The Puffin Group, Inc.

hftpd is a lightweight FTP server that performs well on low-end hardware and scales well on high class machines. It follows a heavily threaded model and was designed to minimize the amount of load it puts on a system. It succeeded in that regard, sometimes performing an order of magnitude better than current freely available servers.

In spite of its age, FTP is still an important resource and a useful means of distributing data. To some people it is the cornerstone of their distribution system: Walnut Creek runs what is regarded as the busiest FTP server on the planet. It recently set records by sending out peaks of almost 1.4 TB/second with 10,000 active connections over gigabit ether. Red Hat Software also relies heavily on FTP to distribute their wares, large hardware is deployed to distribute their software via FTP.

The threaded model was chosen so hftpd will perform well on the widest breadth of systems. The threads are lightweight enough to allow a modest amount of work to be done by aging systems, and by the same token lots of clients may be served off of larger hardware. Attention was also paid to optimizing typical operations an FTP server must perform; features like the generation of file listings were implemented such that CPU usage is greatly reduced.

Predictably, there were complications with deciding to use a native thread implementation. We ended up implementing our own memory allocator to alleviate the pains of tracking thread's memory and managed to rid ourselves of one of the main points of locking contention in the server. With all the threads sharing control of the entire server, great care was required when implementing features like 'SITE EXEC' that could expose the state of all the clients to external programs.

Future direction will be discussed as well. There is interesting potential in moving the FTP server to a single thread of execution, further removing the overhead of threads. This requires moderately more capable hardware to deal with the additional CPU use associated with abstracting the client credentials into user level code, but that is acceptable given today's glut of CPU time. Kernel level changes that relate to efficient network serving, such as using SIGIO in the single execution style server and using intelligent interfaces to provide a zero copy TCP sendfile() implementation will also be discussed.

© 2000 Linux Symposium.  All Rights Reserved.