On 06/26/2017 11:28 AM, Kevin Wolf wrote: > [ Cc: qemu-devel; don't post to qemu-block only! ] > > Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben: >> Hi, >> >> I am currently working on optimizing speed for compressed QCOW2 >> images. We use them for templates and would also like to use them for >> backups, but the latter is almost infeasible because using gzip for >> compression is horribly slow. I tried to experiment with different >> options to deflate, but in the end I think its better to use a >> different compression algorithm for cases where speed matters. As we >> already have probing for it in configure and as it is widely used I >> would like to use LZO for that purpose. I think it would be best to >> have a flag to indicate that compressed blocks use LZO compression, >> but I would need a little explaination which of the feature fields I >> have to use to prevent an older (incompatible) Qemu opening LZO >> compressed QCOW2 images. >> >> I also have already some numbers. I converted a fresh Debian 9 Install >> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a >> compressed QCOW2. With GZIP compression the result is 356MB whereas >> the LZO version is 452MB. However, the current GZIP variant uses 35 >> seconds for this operation where LZO only needs 4 seconds. I think is >> is a good trade in especially when its optional so the user can >> choose. >> >> What are your thoughts? > We had a related RFC patch by Den earlier this year, which never > received many comment and never got out of RFC: > > https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html > > So he chose a different algorithm (zstd). When I asked, he posted a > comparison of algorithms (however a generic one and not measured in the > context of qemu) that suggests that LZO would be slightly faster, but > have a considerable worse compression ratio with the settings that were > benchmarked. > > I think it's clear that if there is any serious interest in compression, > we'll want to support at least one more algorithm. What we still need to > evaluate is which one(s) to take, and whether a simple incompatible flag > in the header like in Den's patch is enough or whether we should add a > whole new header field for the compression algorithm (like we already > have for encryption). > > Kevin I have been contacted today Yann Collet who is ZSTD maintainer, he has dropped nowadays status of ZSTD, which could be useful for the discussion:
"_1. zstd package availability_ We have been tracking distribution availability since Zstandard official release, in September 2016 : https://github.com/facebook/zstd/issues/320 There is also this tool which tracks availability of packages : https://repology.org/metapackage/zstd/versions zstd seems now available as a package in most recent distributions. It’s even part of “core” for recent BSD releases. Zstandard v1.0 is still less than 1 year old, so older distributions typically do not have it (or support a development version). That’s the main limitation currently. We expect things to improve over time. 2. _Compression speed is good but does not matter _For such scenarios, it’s possible to trade speed for more compression. At its maximum compression level (--ultra -22), zstd compression ratio (and speed) is close to lzma. A nice property though is that decompression speed remains roughly the same at all compression levels, about 10x faster than lzma decompression speed (about 1 GB/s on modern CPU). 3. _zstd is multi-threaded, and it’s dangerous_ libzstd is single-threaded. There is a multi-thread extension, which is enabled in the CLI, but not in the library. There is also an experimental target which makes it possible to produce a MT-enabled library. Even in this case, the API remains single-threaded by default. It’s necessary to use dedicated entry points to enable multi-threading. TL;DR : zstd supports multithreading, but is single threaded by default. 4. _How to identify gz format from zstd one ? _Many implementations assume they require to add some custom header in order to identify gz from zstd. That’s not the case: well-formed compression format already provide a header with enough information to guarantee their identity. Such “good” compression format include gz, zstd, xz, lz4-frame, to name a few. For zstd, the identifier is a 4-bytes value, documented in the compression format : https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#zstandard-frames As an example, zstd project provides a zlib-wrapper which is able to dynamically recognize an input as gz or zstd, and route to appropriate decoder, without any special header : https://github.com/facebook/zstd/tree/dev/zlibWrapper Unfortunately, not all compression algorithm do provide unambiguous standard header. LZO, for example, does not by default. Behind a single name, lzo effectively groups multiple incompatible variants, which must be correctly identified for proper decoding." Den
