Hi Dongjin, A couple of comments: I would vote for option b. in the "backward compatibility" section. My reasoning for this is that users upgrading to a zstd compatible version won't start to use it automatically, so manual reconfiguration is required. Therefore an upgrade won't mess up the cluster. If not all the clients are upgraded but just some of them and they'd start to use zstd then it would cause errors in the cluster. I'd like to presume though that this is a very obvious failure case and nobody should be surprised if it didn't work. I wouldn't choose a. as I think we should bump the fetch and produce requests if it's a change in the message format. Moreover if some of the producers and the brokers are upgraded but some of the consumers are not, then we wouldn't prevent the error when the old consumer tries to consume the zstd compressed messages. I wouldn't choose c. either as I think binding the compression type to an API is not so obvious from the developer's perspective.
I would also prefer to use the existing binding, however we must respect the licenses: "The code for these JNI bindings is licenced under 2-clause BSD license. The native Zstd library is licensed under 3-clause BSD license and GPL2" Based on the FAQ page https://www.apache.org/legal/resolved.html#category-a we may use 2- and 3-clause BSD licenses but the Apache license is not compatible with GPL2. I'm hoping that the "3-clause BSD license and GPL2" is really not an AND but an OR in this case, but I'm no lawyer, just wanted to make the point that we should watch out for licenses. :) Regards, Viktor On Sun, Jun 10, 2018 at 3:02 AM Ivan Babrou <ibob...@gmail.com> wrote: > Hello, > > This is Ivan and I still very much support the fact that zstd compression > should be included out of the box. > > Please think about the environment, you can save quite a lot of hardware > with it. > > Thank you. > > On Sat, Jun 9, 2018 at 14:14 Dongjin Lee <dong...@apache.org> wrote: > > > Since there are no responses for a week, I decided to reinitiate the > > discussion thread. > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-110%3A+Add+Codec+for+ZStandard+Compression > > > > This KIP is about to introduce ZStandard Compression into Apache Kafka. > > The reason why it is posted again has a story: It was originally posted > to > > the dev mailing list more than one year ago but since it has no > performance > > report included, it was postponed later. But Some people (including Ivan) > > reported excellent performance report with the draft PR, this work is now > > reactivated. > > > > The updated KIP document includes some expected problems and their > > candidate alternatives. Please have a look when you are free, and give > me a > > feedback. All kinds of participating are welcome. > > > > Best, > > Dongjin > > > > -- > > *Dongjin Lee* > > > > *A hitchhiker in the mathematical world.* > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > <http://github.com/dongjinleekr>linkedin: > kr.linkedin.com/in/dongjinleekr > > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > www.slideshare.net/dongjinleekr > > <http://www.slideshare.net/dongjinleekr>* > > >
