Nice work, Klaus!
On Mon, Aug 26, 2019 at 8:29 PM Klaus Post <klausp...@gmail.com> wrote: > Concurrent stream compression - several GB/sec. > Faster decompression A number of modern compression formats and implementations allow for concurrent *compression*. Coincidentally, I've been working on a format that allows for concurrent *decompression*. Well, actually, my primary motivation is random access (de)compression: being able to reconstruct part of the decompressed file, starting at a given offset into the decompressed file, without always having to first decompress all of the preceding data. But being able to decompress parts of a compressed file independently means that we can decompress in parallel. In this example below, RAC + ZLIB decompressed about 8x faster (wall clock time) than ZIP, even though (1) they're both based on DEFLATE, and (2) the Go DEFLATE decoder isn't quite as fast as the C one. ---- $ ls -l -rw-r--r-- 1 tao tao 100000000 Jun 2 2011 enwik8 -rw-r--r-- 1 tao tao 36445475 Sep 2 2011 enwik8.zip -rw-r--r-- 1 tao tao 37320896 Aug 9 19:16 shared.rac $ cat enwik8 | sha256sum 2b49720ec4d78c3c9fabaee6e4179a5e997302b3a70029f30f2d582218c024a8 - $ unzip -p enwik8.zip | sha256sum 2b49720ec4d78c3c9fabaee6e4179a5e997302b3a70029f30f2d582218c024a8 - $ ractool -decode shared.rac | sha256sum 2b49720ec4d78c3c9fabaee6e4179a5e997302b3a70029f30f2d582218c024a8 - $ time unzip -p enwik8.zip > /dev/null real 0m0.737s user 0m0.713s sys 0m0.025s $ time ractool -decode shared.rac > /dev/null real 0m0.095s user 0m1.316s sys 0m0.069s ---- More details are at https://godoc.org/github.com/google/wuffs/cmd/ractool In particular, RAC is a 'container format', not tied to any one underlying compression codec like DEFLATE / GZIP / ZLIB. We could easilly have a "RAC + Snappy(2)" flavor. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOeFMNXzKrm8YPz6h0UvddmQuAcEWhnt3qpGNA3GYo3Xwu3sBg%40mail.gmail.com.
