I greatly appreciate your comprehensive reasoning. so: +1 for b until now. For the license issues, I will have a check on how the over projects are doing and share the results.
Best, Dongjin On Mon, Jun 11, 2018 at 10:08 PM Viktor Somogyi <viktorsomo...@gmail.com> wrote: > Hi Dongjin, > > A couple of comments: > I would vote for option b. in the "backward compatibility" section. My > reasoning for this is that users upgrading to a zstd compatible version > won't start to use it automatically, so manual reconfiguration is required. > Therefore an upgrade won't mess up the cluster. If not all the clients are > upgraded but just some of them and they'd start to use zstd then it would > cause errors in the cluster. I'd like to presume though that this is a very > obvious failure case and nobody should be surprised if it didn't work. > I wouldn't choose a. as I think we should bump the fetch and produce > requests if it's a change in the message format. Moreover if some of the > producers and the brokers are upgraded but some of the consumers are not, > then we wouldn't prevent the error when the old consumer tries to consume > the zstd compressed messages. > I wouldn't choose c. either as I think binding the compression type to an > API is not so obvious from the developer's perspective. > > I would also prefer to use the existing binding, however we must respect > the licenses: > "The code for these JNI bindings is licenced under 2-clause BSD license. > The native Zstd library is licensed under 3-clause BSD license and GPL2" > Based on the FAQ page > https://www.apache.org/legal/resolved.html#category-a > we may use 2- and 3-clause BSD licenses but the Apache license is not > compatible with GPL2. I'm hoping that the "3-clause BSD license and GPL2" > is really not an AND but an OR in this case, but I'm no lawyer, just wanted > to make the point that we should watch out for licenses. :) > > Regards, > Viktor > > > On Sun, Jun 10, 2018 at 3:02 AM Ivan Babrou <ibob...@gmail.com> wrote: > > > Hello, > > > > This is Ivan and I still very much support the fact that zstd compression > > should be included out of the box. > > > > Please think about the environment, you can save quite a lot of hardware > > with it. > > > > Thank you. > > > > On Sat, Jun 9, 2018 at 14:14 Dongjin Lee <dong...@apache.org> wrote: > > > > > Since there are no responses for a week, I decided to reinitiate the > > > discussion thread. > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-110%3A+Add+Codec+for+ZStandard+Compression > > > > > > This KIP is about to introduce ZStandard Compression into Apache Kafka. > > > The reason why it is posted again has a story: It was originally posted > > to > > > the dev mailing list more than one year ago but since it has no > > performance > > > report included, it was postponed later. But Some people (including > Ivan) > > > reported excellent performance report with the draft PR, this work is > now > > > reactivated. > > > > > > The updated KIP document includes some expected problems and their > > > candidate alternatives. Please have a look when you are free, and give > > me a > > > feedback. All kinds of participating are welcome. > > > > > > Best, > > > Dongjin > > > > > > -- > > > *Dongjin Lee* > > > > > > *A hitchhiker in the mathematical world.* > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > <http://github.com/dongjinleekr>linkedin: > > kr.linkedin.com/in/dongjinleekr > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > > www.slideshare.net/dongjinleekr > > > <http://www.slideshare.net/dongjinleekr>* > > > > > > -- *Dongjin Lee* *A hitchhiker in the mathematical world.* *github: <http://goog_969573159/>github.com/dongjinleekr <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr <http://kr.linkedin.com/in/dongjinleekr>slideshare: www.slideshare.net/dongjinleekr <http://www.slideshare.net/dongjinleekr>*
