Hi,
series looks mostly good to me. Some comments on individual patches.
Backup/restore seems to work, also still with the other compression
formats. The tests for path_to_volume_id actually uncover a bug and a
missing feature in the implementation in Storage.pm, which should be fixed.
For the patches that won't change (much):
Reviewed-By: Fabian Ebner <f.eb...@proxmox.com>
Tested-By: Fabian Ebner <f.eb...@proxmox.com>
For a potential follow-up, I feel like the parsing of the backup
filename might need its own method, rather than have all those pattern
matchings in different places.
On 08.04.20 12:25, Alwin Antreich wrote:
Zstandard (zstd) [0] is a data compression algorithm, in addition to
gzip, lzo for our backup/restore. It can utilize multiple core CPUs. But
by default it has one compression and one writer thread.
Here some quick tests I made on my workstation. The files where placed
on a ram disk. And with dd filled from /dev/urandom and /dev/zero.
__Compression__
file size: 1073741824 bytes
= urandom = = zero =
995ms 1073766414 328ms 98192 zstd -k
732ms 1073766414 295ms 98192 zstd -k -T4
906ms 1073791036 562ms 4894779 lzop -k
31992ms 1073915558 5594ms 1042087 gzip -k
30832ms 1074069541 5776ms 1171491 pigz -k -p 1
7814ms 1074069541 1567ms 1171491 pigz -k -p 4
__Decompression__
file size: 1073741824 bytes
= urandom = = zero =
712ms 869ms zstd -d
685ms 872ms zstd -k -d -T4
841ms 2462ms lzop -d
5417ms 4754ms gzip -k -d
1248ms 3118ms pigz -k -d -p 1
1236ms 2379ms pigz -k -d -p 4
And I used the same ramdisk to move a VM onto it and run a quick
backup/restore.
__vzdump backup__
INFO: transferred 34359 MB in 69 seconds (497 MB/s) zstd -T1
INFO: transferred 34359 MB in 37 seconds (928 MB/s) zstd -T4
INFO: transferred 34359 MB in 51 seconds (673 MB/s) lzo
INFO: transferred 34359 MB in 1083 seconds (31 MB/s) gzip
INFO: transferred 34359 MB in 241 seconds (142 MB/s) pigz -n 4
__qmrestore__
progress 100% (read 34359738368 bytes, duration 36 sec)
total bytes read 34359738368, sparse bytes 8005484544 (23.3%) zstd -d -T4
progress 100% (read 34359738368 bytes, duration 38 sec)
total bytes read 34359738368, sparse bytes 8005484544 (23.3%) lzo
progress 100% (read 34359738368 bytes, duration 175 sec)
total bytes read 34359738368, sparse bytes 8005484544 (23.3%) pigz -n 4
v2 -> v3:
* split archive_info into decompressor_info and archive_info
* "compact" regex pattern is now a constant and used in
multiple modules
* added tests for regex matching
* bug fix for ctime of backup files
v1 -> v2:
* factored out the decompressor info first, as Thomas suggested
* made the regex pattern of backup files more compact, easier to
read (hopefully)
* less code changes for container restores
Thanks for any comment or suggestion in advance.
[0] https://facebook.github.io/zstd/
Alwin Antreich (13):
__pve-storage__
storage: test: split archive format/compressor
test: parse_volname
test: list_volumes
Fix: backup: ctime taken from stat not file name
test: path_to_volume_id
backup: more compact regex for backup file filter
Fix: #2124 storage: add zstd support
test/Makefile | 5 +-
PVE/Storage.pm | 85 ++++++---
PVE/Storage/Plugin.pm | 9 +-
test/run_parser_tests.pl | 17 ++
test/test_archive_info.pm | 57 ++++++
test/test_list_volumes.pm | 313 +++++++++++++++++++++++++++++++++
test/test_parse_volname.pm | 98 +++++++++++
test/test_path_to_volume_id.pm | 104 +++++++++++
8 files changed, 661 insertions(+), 27 deletions(-)
create mode 100755 test/run_parser_tests.pl
create mode 100644 test/test_archive_info.pm
create mode 100644 test/test_list_volumes.pm
create mode 100644 test/test_parse_volname.pm
create mode 100644 test/test_path_to_volume_id.pm
__pve-container__
Fix: #2124 add zstd
src/PVE/LXC/Create.pm | 1 +
1 file changed, 1 insertion(+)
__qemu-server__
restore: replace archive format/compression regex
Fix #2124: Add support for zstd
PVE/QemuServer.pm | 38 +++++++-------------------------------
1 file changed, 7 insertions(+), 31 deletions(-)
__pve-manager__
Fix #2124: Add support for zstd
PVE/VZDump.pm | 11 +++++++++--
debian/control | 1 +
www/manager6/form/CompressionSelector.js | 3 ++-
3 files changed, 12 insertions(+), 3 deletions(-)
__pve-guest-common__
Fix: #2124 add zstd support
PVE/VZDump/Common.pm | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
_______________________________________________
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
