Flow based network accounting with Linux

Many networking scenarios require some form of network accounting that goes beyond some simple packet and byte counters as available from the 'ifconfig' output.

When people want to do network accouting, the past and current Linux kernel didn't provide them with any reasonable mechanism for doing so.

Network accounting can generally be done in a number of different ways. The traditional way is to capture all packets by some userspace program. Capturing can be done via a number of mechanisms such as PF_PACKET sockets, mmap()ed PF_PACKET, ipt_ULOG, or ip_queue. This userspace program then analyzes the packets and aggregates the result into per-flow data structures.

Whatever mechanism used, this scheme has a fundamental performance limitation, since all packets need to be copied and analyzed by a userspace process.

The author has implemented a different approach, by which the accounting information is stored in the in-kernel connection tracking table of the ip_conntrack stateful firewall state machine. On all firewalls, that state table has to be kept anyways - the additional overhead introduced by accounting is minimal.

Once a connection is evicted from the state table, its accounting relevant data is transferred to userspace to a special accounting daemon for further processing, aggregation and finally storage in the accounting log/database.