Hi there, I am using gzip 1.6 to compress large files >10 GiB in parallel (Kubuntu 14.04, 12 cores). The underlying disk system (RAID 10) is able to deliver read speeds >1 GB/s (measured with flushed file caches, iostat -mx 1 100).
Here are some numbers when running gzip in parallel: 1 gzip process: the CPU is the bottleneck in compressing things and utilisation is 100%. 2 gzips in parallel: the disk throughput drops to a meagre 70MB/s and the CPU utilisation per process is at ~60%. 6 gzips in parallel: the disk throughput fluctuates between 50 and 60 MB/s and the CPU utilisation per process is at ~18-20%. Running 6 gzips in parallel on the same data residing on a SSD: 100% CPU utilisation per process Googling a bit I found this thread on SuperUser where someone saw the same behaviour already with a single disk doing normally 125 MB/s and running 4 gzips drops it to 25 MB/s: http://superuser.com/questions/599329/why-is-gzip-slow-despite-cpu-and-hard-drive-performance-not-being-maxed-out The posts there propose a workaround like this: buffer -s 100000 -m 10000000 -p 100 < bigfile.dat | gzip > bigfile.dat.gz And indeed, using "buffer" resolves trashing problems when working on a disk system. However, using "buffer" is pretty arcane (it isn't even installed per default on most Unix/Linux installations) and pretty counterintuitive. Would it be possible to have bigger buffers by default (1 MB? 10 MB?) or have an automatism in gzip like "if file to compress >10 MB and free RAM >500MB, setup the file buffer to use 1 (10?) MB" ? Alternatively, a command line option to manually set the buffer size? Best, Bastien -- DSM Nutritional Products Microbia Inc | Bioinformatics 60 Westview Street | Lexington, MA 02421 | United States Phone +1 781 259 7613 | Fax +1 781 259 0615 ________________________________ DISCLAIMER: This e-mail is for the intended recipient only. If you have received it by mistake please let us know by reply and then delete it from your system; access, disclosure, copying, distribution or reliance on any of it by anyone else is prohibited. If you as intended recipient have received this e-mail incorrectly, please notify the sender (via e-mail) immediately.
