Re: [zfs-discuss] compression at zfs filesystem creation

[Haudy Kazemi]

Bob Friesenhahn wrote: On Mon, 15 Jun 2009, Bob Friesenhahn wrote: On Mon, 15 Jun 2009, Rich Teer wrote: You actually have that backwards.  :-)  In most cases, compression is very desirable.  Performance studies have shown that today's CPUs can compress data faster than it takes for the uncompressed data to be read or written. Do you have a reference for such an analysis based on ZFS?  I would be interested in linear read/write performance rather than random access synchronous access. Perhaps you are going to make me test this for myself. Ok, I tested this for myself on a Solaris 10 system with 4 3GHz AMD64 cores and see that we were both right.  I did an iozone run with compression and do see a performance improvement.  I don't know what the data iozone produces looks like, but it clearly must be quite compressable.  Testing was done with a 64GB file:                      KB  reclen   write rewrite    read    reread uncompressed:  67108864     128  359965  354854   550869   554271 lzjb:          67108864     128  851336  924881  1289059  1362625 Unfortunately, during the benchmark run with lzjb the system desktop was essentially unusable with misbehaving mouse and keyboard as well as reported 55% CPU consumption.  Without the compression the system is fully usable with very little CPU consumed. If the system is dedicated to serving files rather than also being used interactively, it should not matter much what the CPU usage is.  CPU cycles can't be stored for later use.  Ultimately, it (mostly*) does not matter if one option consumes more CPU resources than another if those CPU resources were otherwise going to go unused.  Changes (increases) in latencies are a consideration but probably depend more on process scheduler choice and policies. *Higher CPU usage will increase energy consumption, heat output, and cooling costs...these may be important considerations in some specialized dedicated file server applications, depending on operational considerations. The interactivity hit may pose a greater challenge for any other processes/databases/virtual machines run on hardware that also serves files.  The interactivity hit may also be evidence that the process scheduler is not fairly or effectively sharing CPU resources amongst the running processes.  If scheduler tweaks aren't effective, perhaps dedicating a processor core(s) to interactive GUI stuff and the other cores to filesystem duties would help smooth things out.  Maybe zones be used for that? With a slower disk subsystem the CPU overhead would surely be less since writing is still throttled by the disk. It would be better to test with real data rather than iozone. There are 4 sets of articles with links and snippets from their test data below.  Follow the links for the full discussion: First article: http://blogs.sun.com/dap/entry/zfs_compression#comments Hardware: Sun Storage 7000 # The server is a quad-core 7410 with 1 JBOD (configured with mirrored storage) and 16GB of RAM. No SSD. # The client machine is a quad-core 7410 with 128GB of DRAM. Summary: text data set Compression Ratio Total Write Read off 1.00x 3:30 2:08 1:22 lzjb 1.47x 3:26 2:04 1:22 gzip-2 2.35x 6:12 4:50 1:22 gzip 2.52x 11:18 9:56 1:22 gzip-9 2.52x 12:16 10:54 1:22 Summary: media data set Compression Ratio Total Write Read off 1.00x 3:29 2:07 1:22 lzjb 1.00x 3:31 2:09 1:22 gzip-2 1.01x 6:59 5:37 1:22 gzip 1.01x 7:18 5:57 1:21 gzip-9 1.01x 7:37 6:15 1:22 Second article/discussion: http://ekschi.com/technology/2009/04/28/zfs-compression-a-win-win/ http://blogs.sun.com/observatory/entry/zfs_compression_a_win_win Third article summary: ZFS and MySQL/InnoDB shows that gzip is often cpu-bound on current processors; lzjb improves performance. http://blogs.smugmug.com/don/2008/10/13/zfs-mysqlinnodb-compression-update/ Hardware: SunFire X2200 M2 w/64GB of RAM and 2 x dual-core 2.6GHz Opterons Dell MD3000 w/15 x 15K SCSI disks and mirrored 512MB battery-backed write caches "Also note that this is writing to two DAS enclosures with 15 x 15K SCSI disks apiece (28 spindles in a striped+mirrored configuration) with 512MB of write cache apiece." TABLE1 compression size ratio time uncompressed 172M 1 0.207s lzjb 79M 2.18X 0.234s gzip-1 50M 3.44X 0.24s gzip-9 46M 3.73X 0.217s Notes on TABLE1: This dataset seems to be small enough that much of time is probably spent in system internals, rather than actually reading, compressing, and writing data, so I view this as only an interesting size datapoint, rather than size and time. Feel free to correct me, though. TABLE2 compression size ratio time ratio uncompressed 631M 1 1.064s 1 lzjb 358M 1.76X 0.668 1.59X gzip-1 253M 2.49X 1.302 0.82X gzip-9 236M 3.73X 11.1s 0.10X Notes on TABLE2: gzip-9 is massively slower on this particular hunk of data. I’m no expert on gzip, so I have no idea why this would be, but you can see the tradeoff is probably rarely worth it, even if were using precious storage commodities (say, flash or RAM rather than hard disks). I ran this one extra times just to make sure. Seems valid (or a bug). TABLE3 compression size ratio time ratio uncompressed 2675M 1 15.041s 1 lzjb 830M 3.22X 5.274 2.85X gzip-1 246M 10.87X 44.287 0.34X gzip-9 220M 12.16X 52.475 0.29X Notes on TABLE3: LZJB really shines here, performance wise. It delivers roughly 3X faster performance while also chewing up roughly 3X less bytes. Awesome. gzip’s compression ratios are crazy great on this hunk of data, but the performance is pretty awful. Definitely CPU-bound, not IO-bound. TABLE4 compression size ratio time ratio uncompressed 2828M 1 17.09s 1 lzjb 1814M 1.56X 14.495s 1.18X gzip-1 1384M 2.04X 48.895s 0.35X gzip-9 1355M 2.09X 54.672s 0.31X Notes on TABLE4: Again, LZJB performs quite well. 1.5X bytes saved while remaining faster. Nice! gzip is again very obviously CPU bound, rather than IO-bound. Dang. Fourth article: zfs-fuse on Linux: http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg08219.html Hardware: "this test was done an an ancient suse 9.1 box, fuse 2.5.3, 1,25 GB RAM (1 gb in use for other apps), 2x80gb 3ware raid1" linux kernel source, (239M) time tar xf linux-2.6.20.3.tar |compression |time-real |time-user |time-sys |compressratio -------------------------------------------------------------- |lzo |6m39.603s |0m1.288s |0m6.055s |2.99x |gzip |7m46.875s |0m1.275s |0m6.312s |3.41x |lzjb |7m7.600s |0m1.227s |0m6.033s |1.79x |off |7m26.735s |0m1.348s |0m6.230s |1.00x

_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss