Dongjin, The KIP states:
"I compared the compressed size and compression time of 3 1kb-sized messages (3102 bytes in total), with the Draft-implementation of ZStandard Compression Codec and all currently available CompressionCodecs. All elapsed times are the average of 20 trials." But doesn't give any details of how this was implemented. Is the source code available somewhere? Micro-benchmarking in the JVM is pretty tricky so it needs verification before numbers can be trusted. A performance test with kafka-producer-perf-test.sh would be nice to have as well, if possible. Thanks, Ismael On Tue, Jan 10, 2017 at 7:44 AM, Dongjin Lee <dong...@apache.org> wrote: > Ismael, > > 1. Is the benchmark in the KIP page not enough? You mean we need a whole > performance test using kafka-producer-perf-test.sh? > > 2. It seems like no major project is relying on it currently. However, > after reviewing the code, I concluded that at least this project has a good > test coverage. And for the problem of upstream tracking - although there is > no significant update on ZStandard to judge this problem, it seems not bad. > If required, I can take responsibility of the tracking for this library. > > Thanks, > Dongjin > > On Tue, Jan 10, 2017 at 7:09 AM, Ismael Juma <ism...@juma.me.uk> wrote: > > > Thanks for posting the KIP, ZStandard looks like a nice improvement over > > the existing compression algorithms. A couple of questions: > > > > 1. Can you please elaborate on the details of the benchmark? > > 2. About https://github.com/luben/zstd-jni, can we rely on it? A few > > things > > to consider: are there other projects using it, does it have good test > > coverage, are there performance tests, does it track upstream closely? > > > > Thanks, > > Ismael > > > > On Fri, Jan 6, 2017 at 2:40 AM, Dongjin Lee <dong...@apache.org> wrote: > > > > > Hi all, > > > > > > I've just posted a new KIP "KIP-110: Add Codec for ZStandard > Compression" > > > for > > > discussion: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 110%3A+Add+Codec+for+ZStandard+Compression > > > > > > Please have a look when you are free. > > > > > > Best, > > > Dongjin > > > > > > -- > > > *Dongjin Lee* > > > > > > > > > *Software developer in Line+.So interested in massive-scale machine > > > learning.facebook: www.facebook.com/dongjin.lee.kr > > > <http://www.facebook.com/dongjin.lee.kr>linkedin: > > > kr.linkedin.com/in/dongjinleekr > > > <http://kr.linkedin.com/in/dongjinleekr>github: > > > <http://goog_969573159/>github.com/dongjinleekr > > > <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr > > > <http://www.twitter.com/dongjinleekr>* > > > > > > > > > -- > *Dongjin Lee* > > > *Software developer in Line+.So interested in massive-scale machine > learning.facebook: www.facebook.com/dongjin.lee.kr > <http://www.facebook.com/dongjin.lee.kr>linkedin: > kr.linkedin.com/in/dongjinleekr > <http://kr.linkedin.com/in/dongjinleekr>github: > <http://goog_969573159/>github.com/dongjinleekr > <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr > <http://www.twitter.com/dongjinleekr>* >
