2014 Linux Symposium, July 14-16

Stride based Transparent Compression

Abhijeet Pawar (304abhijeet@gmail.com)

In recent years, the data is growing at a very high rate compared to the rate at which available storage is increasing. Although, cost of storage devices is reducing, the overall expenditure on storage is increasing. Also, device I/O is still the slowest component in the storage stack. There is a need of efficient storage mechanism. In this paper, we present Transparent Compression, a file-system layer compression module aimed at optimizing storage entropy.

Unlike Conventional I/O, which stores data uncompressed on the disk, Transparent Compression, stores data in compressed format. Implementation at the file-system layer instead of application layer provides compression on-the-fly making it transparent to applications. Our Stride-based design compresses and stores data in chunks of compression strides, in turn, improving random rewrite performance. We propose Threshold-based Compression, which will enable/disable compression for stride, based on computed compression ratio. This eliminates the overhead of compressing strides having poor compression ratio. Our design is capable of supporting multiple compression algorithms. Our benchmarks on TUX 3 file-system using LZO compression algorithm show substantial improvement in storage as well as time performance.

Policies   |   Media Archives