On 06/26/2017 11:54 PM, Peter Lieven wrote: > >> Am 26.06.2017 um 22:30 schrieb Denis V. Lunev <d...@openvz.org>: >> >>> On 06/26/2017 11:28 AM, Kevin Wolf wrote: >>> [ Cc: qemu-devel; don't post to qemu-block only! ] >>> >>> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben: >>>> Hi, >>>> >>>> I am currently working on optimizing speed for compressed QCOW2 >>>> images. We use them for templates and would also like to use them for >>>> backups, but the latter is almost infeasible because using gzip for >>>> compression is horribly slow. I tried to experiment with different >>>> options to deflate, but in the end I think its better to use a >>>> different compression algorithm for cases where speed matters. As we >>>> already have probing for it in configure and as it is widely used I >>>> would like to use LZO for that purpose. I think it would be best to >>>> have a flag to indicate that compressed blocks use LZO compression, >>>> but I would need a little explaination which of the feature fields I >>>> have to use to prevent an older (incompatible) Qemu opening LZO >>>> compressed QCOW2 images. >>>> >>>> I also have already some numbers. I converted a fresh Debian 9 Install >>>> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a >>>> compressed QCOW2. With GZIP compression the result is 356MB whereas >>>> the LZO version is 452MB. However, the current GZIP variant uses 35 >>>> seconds for this operation where LZO only needs 4 seconds. I think is >>>> is a good trade in especially when its optional so the user can >>>> choose. >>>> >>>> What are your thoughts? >>> We had a related RFC patch by Den earlier this year, which never >>> received many comment and never got out of RFC: >>> >>> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html >>> >>> So he chose a different algorithm (zstd). When I asked, he posted a >>> comparison of algorithms (however a generic one and not measured in the >>> context of qemu) that suggests that LZO would be slightly faster, but >>> have a considerable worse compression ratio with the settings that were >>> benchmarked. >>> >>> I think it's clear that if there is any serious interest in compression, >>> we'll want to support at least one more algorithm. What we still need to >>> evaluate is which one(s) to take, and whether a simple incompatible flag >>> in the header like in Den's patch is enough or whether we should add a >>> whole new header field for the compression algorithm (like we already >>> have for encryption). >>> >>> Kevin >> I have been contacted today Yann Collet who is ZSTD maintainer, he has >> dropped >> nowadays status of ZSTD, which could be useful for the discussion: >> >> "_1. zstd package availability_ >> >> We have been tracking distribution availability since Zstandard official >> release, in September 2016 : >> https://github.com/facebook/zstd/issues/320 >> There is also this tool which tracks availability of packages : >> https://repology.org/metapackage/zstd/versions >> >> zstd seems now available as a package in most recent distributions. >> It’s even part of “core” for recent BSD releases. >> Zstandard v1.0 is still less than 1 year old, so older distributions >> typically do not have it (or support a development version). >> That’s the main limitation currently. We expect things to improve over time. >> >> 2. >> >> _Compression speed is good but does not matter >> _For such scenarios, it’s possible to trade speed for more compression. >> At its maximum compression level (--ultra -22), zstd compression >> ratio (and speed) is close to lzma. >> A nice property though is that decompression speed remains roughly >> the same at all compression levels, >> about 10x faster than lzma decompression speed (about 1 GB/s on >> modern CPU). >> >> 3. >> >> _zstd is multi-threaded, and it’s dangerous_ >> >> libzstd is single-threaded. >> There is a multi-thread extension, which is enabled in the CLI, but not >> in the library. >> There is also an experimental target which makes it possible to produce >> a MT-enabled library. >> Even in this case, the API remains single-threaded by default. >> It’s necessary to use dedicated entry points to enable multi-threading. >> TL;DR : zstd supports multithreading, but is single threaded by default. >> >> >> >> 4. >> >> _How to identify gz format from zstd one ? >> _Many implementations assume they require to add some custom header >> in order to identify gz from zstd. >> That’s not the case: well-formed compression format already provide >> a header with enough information to guarantee their identity. >> Such “good” compression format include gz, zstd, xz, lz4-frame, to >> name a few. >> For zstd, the identifier is a 4-bytes value, documented in the >> compression format : >> >> https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#zstandard-frames >> As an example, zstd project provides a zlib-wrapper which is able to >> dynamically recognize an input as gz or zstd, and route to >> appropriate decoder, without any special header : >> https://github.com/facebook/zstd/tree/dev/zlibWrapper >> >> >> Unfortunately, not all compression algorithm do provide unambiguous >> standard header. >> LZO, for example, does not by default. >> Behind a single name, lzo effectively groups multiple incompatible >> variants, which must be correctly identified for proper decoding." >> >> Den >> > Hi Den, > > thanks for the update. > > I am about to have an RFC patchset ready for addition of the compression > algorithm header. It will be easy to add support for zstd on top of that. > > I would prefer to have the same algorithm for all compressed clusters and > avoid detecting for each cluster. If you want to change the algorithm you > would have to recode. This way you can also easily detect at open time of the > image if you support the compression algorithm and fail directly if necessary. > > Peter completely agree.
Den
