On Wednesday, 28 August 2019 02:37:25 UTC+2, Nigel Tao wrote: > > On Mon, Aug 26, 2019 at 8:29 PM Klaus Post <klau...@gmail.com > <javascript:>> wrote: > > This package is aimed at replacing Snappy as a high speed compression > package. If you are mainly looking for better compression zstandard gives > better compression, but typically at speeds slightly below "better" mode in > this package. > > Do you know how S2 compares with LZ4? https://github.com/lz4/lz4 > suggests that LZ4 is faster (for both compression and decompression) > than Snappy, for a comparable compression ratio. >
To be fair to LZ4, here is a single core comparison. Only the fastest mode of LZ4 is really 'competitive' - this is level 0 in the chart. S2 level 1 is default, level 2 is "better". It is against master of `github.com/pierrec/lz4` to which I also sent in some nice speedups. | file | out | level | insize | outsize | millis | mb/s | |-----------------------------|-----|-------|-------------|------------|--------|---------| | rawstudio-mint14.tar | lz4 | 0 | 8558382592 | 4553246910 | 17942 | 454.90 | | rawstudio-mint14.tar | s2 | 1 | 8558382592 | 4461314089 | 12935 | 630.95 | | rawstudio-mint14.tar | s2 | 2 | 8558382592 | 4285178801 | 25292 | 322.7 | | github-june-2days-2019.json | lz4 | 0 | 6273951764 | 1274316041 | 9441 | 633.75 | | github-june-2days-2019.json | s2 | 1 | 6273951764 | 1086090092 | 7425 | 805.76 | | github-june-2days-2019.json | s2 | 2 | 6273951764 | 1055384443 | 10936 | 547.1 | | github-ranks-backup.bin | lz4 | 0 | 1862623243 | 618233345 | 3655 | 485.89 | | github-ranks-backup.bin | s2 | 1 | 1862623243 | 624632852 | 3002 | 591.58 | | github-ranks-backup.bin | s2 | 2 | 1862623243 | 564891142 | 4990 | 355.95 | | consensus.db.10gb | lz4 | 0 | 10737418240 | 5065051629 | 23608 | 433.75 | | consensus.db.10gb | s2 | 1 | 10737418240 | 4610183941 | 14913 | 686.63 | | consensus.db.10gb | s2 | 2 | 10737418240 | 4612959122 | 32878 | 311.45 | | adresser.json | lz4 | 0 | 7983034785 | 470399106 | 5572 | 1366.27 | | adresser.json | s2 | 1 | 7983034785 | 457342320 | 4546 | 1674.7 | | adresser.json | s2 | 2 | 7983034785 | 438421980 | 5320 | 1431 | | gob-stream | lz4 | 0 | 1911399616 | 377393110 | 2904 | 627.56 | | gob-stream | s2 | 1 | 1911399616 | 356137611 | 2439 | 747.21 | | gob-stream | s2 | 2 | 1911399616 | 345184546 | 3392 | 537.28 | | 10gb.tar | lz4 | 0 | 10065157632 | 5852284245 | 22916 | 418.87 | | 10gb.tar | s2 | 1 | 10065157632 | 5936329234 | 20340 | 471.91 | | 10gb.tar | s2 | 2 | 10065157632 | 5725266005 | 29463 | 325.79 | | sharnd.out.2gb | lz4 | 0 | 2147483647 | 2147485710 | 442 | 4632.35 | | sharnd.out.2gb | s2 | 1 | 2147483647 | 2147500041 | 246 | 8323.35 | | sharnd.out.2gb | s2 | 2 | 2147483647 | 2147500041 | 272 | 7527.72 | | enwik9 | lz4 | 0 | 1000000000 | 469015635 | 3681 | 259.02 | | enwik9 | s2 | 1 | 1000000000 | 489432671 | 2973 | 320.71 | | enwik9 | s2 | 2 | 1000000000 | 435069175 | 4733 | 201.49 | | silesia.tar | lz4 | 0 | 211947520 | 95265393 | 600 | 336.75 | | silesia.tar | s2 | 1 | 211947520 | 97205820 | 484 | 417.52 | | silesia.tar | s2 | 2 | 211947520 | 90759543 | 794 | 254.51 | On a single core, it seems like LZ4 on the fastest setting is between the two modes. Some cases (the 2 JSON cases, the VM image and the gob stream) are strictly worse on LZ4. TLDR; LZ4 is typically between the default and "better" mode of s2. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a5cb9e65-c04b-4a43-87fa-dc54d9d4c979%40googlegroups.com.
