with an additional commit adding '--rsyncable'. thanks!
On April 28, 2020 3:58 pm, Alwin Antreich wrote: > Zstandard (zstd) [0] is a data compression algorithm, in addition to > gzip, lzo for our backup/restore. It can utilize multiple core CPUs. But > by default it has one compression and one writer thread. > > > Here some quick tests I made on my workstation. The files where placed > on a ram disk. And with dd filled from /dev/urandom and /dev/zero. > > __Compression__ > file size: 1073741824 bytes > = urandom = = zero = > 995ms 1073766414 328ms 98192 zstd -k > 732ms 1073766414 295ms 98192 zstd -k -T4 > 906ms 1073791036 562ms 4894779 lzop -k > 31992ms 1073915558 5594ms 1042087 gzip -k > 30832ms 1074069541 5776ms 1171491 pigz -k -p 1 > 7814ms 1074069541 1567ms 1171491 pigz -k -p 4 > > __Decompression__ > file size: 1073741824 bytes > = urandom = = zero = > 712ms 869ms zstd -d > 685ms 872ms zstd -k -d -T4 > 841ms 2462ms lzop -d > 5417ms 4754ms gzip -k -d > 1248ms 3118ms pigz -k -d -p 1 > 1236ms 2379ms pigz -k -d -p 4 > > > And I used the same ramdisk to move a VM onto it and run a quick > backup/restore. > > __vzdump backup__ > INFO: transferred 34359 MB in 69 seconds (497 MB/s) zstd -T1 > INFO: transferred 34359 MB in 37 seconds (928 MB/s) zstd -T4 > INFO: transferred 34359 MB in 51 seconds (673 MB/s) lzo > INFO: transferred 34359 MB in 1083 seconds (31 MB/s) gzip > INFO: transferred 34359 MB in 241 seconds (142 MB/s) pigz -n 4 > > __qmrestore__ > progress 100% (read 34359738368 bytes, duration 36 sec) > total bytes read 34359738368, sparse bytes 8005484544 (23.3%) zstd -d -T4 > > progress 100% (read 34359738368 bytes, duration 38 sec) > total bytes read 34359738368, sparse bytes 8005484544 (23.3%) lzo > > progress 100% (read 34359738368 bytes, duration 175 sec) > total bytes read 34359738368, sparse bytes 8005484544 (23.3%) pigz -n 4 > > > v4 -> v5: > * fixup, use File::stat directly without overwriting CORE::stat, > thanks Dietmar for pointing this out > https://pve.proxmox.com/pipermail/pve-devel/2020-April/043134.html > * rebase to current master > > v3 -> v4: > * fixed styling issues discovered by f.ebner (thanks) > * incorporated tests of d.jaeger into patches > * added fixes discovered by tests (f.ebner thanks) > > v2 -> v3: > * split archive_info into decompressor_info and archive_info > * "compact" regex pattern is now a constant and used in > multiple modules > * added tests for regex matching > * bug fix for ctime of backup files > > v1 -> v2: > * factored out the decompressor info first, as Thomas suggested > * made the regex pattern of backup files more compact, easier to > read (hopefully) > * less code changes for container restores > > Thanks for any comment or suggestion in advance. > > [0] https://facebook.github.io/zstd/ > > Alwin Antreich (17): > __pve-storage__ > storage: test: split archive format/compressor > storage: replace build-in stat with File::stat > test: parse_volname > test: list_volumes > Fix: backup: ctime was from stat not file name > test: path_to_volume_id > Fix: path_to_volume_id returned wrong content > Fix: add missing snippets subdir > backup: compact regex for backup file filter > Fix: #2124 storage: add zstd support > test: get_subdir > test: filesystem_path > > test/Makefile | 5 +- > PVE/Diskmanage.pm | 9 +- > PVE/Storage.pm | 91 ++++-- > PVE/Storage/Plugin.pm | 47 ++- > test/archive_info_test.pm | 127 ++++++++ > test/filesystem_path_test.pm | 91 ++++++ > test/get_subdir_test.pm | 44 +++ > test/list_volumes_test.pm | 537 +++++++++++++++++++++++++++++++++ > test/parse_volname_test.pm | 253 ++++++++++++++++ > test/path_to_volume_id_test.pm | 274 +++++++++++++++++ > test/run_plugin_tests.pl | 18 ++ > 11 files changed, 1440 insertions(+), 56 deletions(-) > create mode 100644 test/archive_info_test.pm > create mode 100644 test/filesystem_path_test.pm > create mode 100644 test/get_subdir_test.pm > create mode 100644 test/list_volumes_test.pm > create mode 100644 test/parse_volname_test.pm > create mode 100644 test/path_to_volume_id_test.pm > create mode 100755 test/run_plugin_tests.pl > > > __guest_common__ > Fix: #2124 add zstd support > > PVE/VZDump/Common.pm | 9 ++++++++- > 1 file changed, 8 insertions(+), 1 deletion(-) > > > __qemu-server__ > restore: replace archive format/compression > Fix #2124: Add support for zstd > > PVE/QemuServer.pm | 38 +++++++------------------------------- > 1 file changed, 7 insertions(+), 31 deletions(-) > > > __pve-container__ > Fix: #2124 add zstd > > src/PVE/LXC/Create.pm | 1 + > 1 file changed, 1 insertion(+) > > > __pve-manager__ > Fix #2124: Add support for zstd > > PVE/VZDump.pm | 11 +++++++++-- > debian/control | 1 + > www/manager6/form/CompressionSelector.js | 3 ++- > 3 files changed, 12 insertions(+), 3 deletions(-) > -- > 2.20.1 > > > _______________________________________________ > pve-devel mailing list > pve-devel@pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > _______________________________________________ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
