> > Could you share the benchmark code/how the benchmark was run (does this > account for JIT warm-up time)?
I just used the benchmark by the aircompressor project. They run the benchmark for a lot of algorithms on a lot of datasets so I commented out some to get faster results. You can find my version here: https://github.com/HedgehogCode/aircompressor/blob/lz4-benchmark/src/test/java/io/airlift/compress/benchmark/CompressionBenchmark.java Yes, the benchmark accounts for warmup. Best, Benjamin On Mon, Mar 22, 2021 at 5:13 PM Micah Kornfield <emkornfi...@gmail.com> wrote: > > > > I executed some of the benchmarks in the airlift/aircompressor project. I > > found that aircompressior achieves on average only about 72% > > throughput compared to the current version of the lz4-java JNI bindings > > when compressing. When decompressing the gap is even bigger with around > 56% > > throughout. See the following google sheet for the benchmark results. > > > > > https://docs.google.com/spreadsheets/d/1mT1qmpvV25YcRmPz4IYxXyPSzUsMsdovN7gc_vmac5U/edit?usp=sharing > > > I think it is fine if we want to use lz4-java bindings for compression as > this will be a proper subset of LZ4 compression. > > Could you share the benchmark code/how the benchmark was run (does this > account for JIT warm-up time)? > > -Micah > > On Mon, Mar 22, 2021 at 7:46 AM Antoine Pitrou <anto...@python.org> wrote: > > > > > Le 22/03/2021 à 15:29, Benjamin Wilhelm a écrit : > > > Also, I would like to resume the discussion about the Frame format vs > the > > > Block format. There were 3 points for the Frame format by Antoine: > > > > > > - it allows streaming compression and decompression (meaning you can > > >> avoid loading a huge compressed buffer at once) > > >> > > > It seems like this is not used anywhere. Doesn't it make more sense to > > use > > > more record batches if one buffer in a record batch gets too big? > > > > It does. But that depends who emitted the record batches. Perhaps you're > > receiving data written out by a large machine and trying to process it > > on a small embedded client? I'm not sure this example makes sense or is > > interesting at all. > > > > >> - it embeds the decompressed size, allowing exact allocation of the > > >> decompressed buffer > > >> > > > Micah pointed out that this is already part of the IPC specification. > > > > Ah, indeed. Then this point is moot. > > > > >> - it has an optional checksum > > >> > > > Wouldn't it make sense to have a higher level checksum (as already > > > mentioned by Antoine) if we want to have checksums at all? Just having > a > > > checksum in case of one specific compression does not make a lot of > sense > > > to me. > > > > Yes, it would certainly make sense. > > > > > Given these points, I think that the Frame format is not more useful > for > > > compressing Arrow Buffers than the Block format. It only adds > unnecessary > > > overhead in metadata. > > > Are there any other points for the Frame format that I missed? If not, > > what > > > would it mean to switch to the Block format? (Or add the Block format > as > > an > > > option??) > > > > Well, the problem is that by now this spec is public and has been > > implemented since Arrow C++ 2.0. > > > > Fortunately, we named the compression method "LZ4_FRAME" so we could add > > "LZ4_RAW". But it seems a bit wasteful and confusing to allow both even > > though we agree there is no advantage to either one. > > > > Regards > > > > Antoine. > > >
