Here is the short conclusion about the license problem: *We can use zstd and zstd-jni without any problem, but we need to include their license, e.g., BSD license.*
Both of BSD 2 Clause License & 3 Clause License requires to include the license used, and BSD 3 Clause License requires that the name of the contributor can't be used to endorse or promote the product. That's it <http://www.mikestratton.net/2011/12/is-bsd-license-compatible-with-apache-2-0-license/> - They are not listed in the list of prohibited licenses <https://www.apache.org/legal/resolved.html#category-x> also. Here is how Spark did for it <https://issues.apache.org/jira/browse/SPARK-19112>: - They made a directory dedicated to the dependency license files <https://github.com/apache/spark/tree/master/licenses> and added licenses for Zstd <https://github.com/apache/spark/blob/master/licenses/LICENSE-zstd.txt> & Zstd-jni <https://github.com/apache/spark/blob/master/licenses/LICENSE-zstd-jni.txt>. - Added a link to the original license files in LICENSE. <https://github.com/apache/spark/pull/18805/files> If needed, I can make a similar update. Thanks for pointing out this problem, Viktor! Nice catch! Best, Dongjin On Mon, Jun 11, 2018 at 11:50 PM Dongjin Lee <dong...@apache.org> wrote: > I greatly appreciate your comprehensive reasoning. so: +1 for b until now. > > For the license issues, I will have a check on how the over projects are > doing and share the results. > > Best, > Dongjin > > On Mon, Jun 11, 2018 at 10:08 PM Viktor Somogyi <viktorsomo...@gmail.com> > wrote: > >> Hi Dongjin, >> >> A couple of comments: >> I would vote for option b. in the "backward compatibility" section. My >> reasoning for this is that users upgrading to a zstd compatible version >> won't start to use it automatically, so manual reconfiguration is >> required. >> Therefore an upgrade won't mess up the cluster. If not all the clients are >> upgraded but just some of them and they'd start to use zstd then it would >> cause errors in the cluster. I'd like to presume though that this is a >> very >> obvious failure case and nobody should be surprised if it didn't work. >> I wouldn't choose a. as I think we should bump the fetch and produce >> requests if it's a change in the message format. Moreover if some of the >> producers and the brokers are upgraded but some of the consumers are not, >> then we wouldn't prevent the error when the old consumer tries to consume >> the zstd compressed messages. >> I wouldn't choose c. either as I think binding the compression type to an >> API is not so obvious from the developer's perspective. >> >> I would also prefer to use the existing binding, however we must respect >> the licenses: >> "The code for these JNI bindings is licenced under 2-clause BSD license. >> The native Zstd library is licensed under 3-clause BSD license and GPL2" >> Based on the FAQ page >> https://www.apache.org/legal/resolved.html#category-a >> we may use 2- and 3-clause BSD licenses but the Apache license is not >> compatible with GPL2. I'm hoping that the "3-clause BSD license and GPL2" >> is really not an AND but an OR in this case, but I'm no lawyer, just >> wanted >> to make the point that we should watch out for licenses. :) >> >> Regards, >> Viktor >> >> >> On Sun, Jun 10, 2018 at 3:02 AM Ivan Babrou <ibob...@gmail.com> wrote: >> >> > Hello, >> > >> > This is Ivan and I still very much support the fact that zstd >> compression >> > should be included out of the box. >> > >> > Please think about the environment, you can save quite a lot of hardware >> > with it. >> > >> > Thank you. >> > >> > On Sat, Jun 9, 2018 at 14:14 Dongjin Lee <dong...@apache.org> wrote: >> > >> > > Since there are no responses for a week, I decided to reinitiate the >> > > discussion thread. >> > > >> > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-110%3A+Add+Codec+for+ZStandard+Compression >> > > >> > > This KIP is about to introduce ZStandard Compression into Apache >> Kafka. >> > > The reason why it is posted again has a story: It was originally >> posted >> > to >> > > the dev mailing list more than one year ago but since it has no >> > performance >> > > report included, it was postponed later. But Some people (including >> Ivan) >> > > reported excellent performance report with the draft PR, this work is >> now >> > > reactivated. >> > > >> > > The updated KIP document includes some expected problems and their >> > > candidate alternatives. Please have a look when you are free, and give >> > me a >> > > feedback. All kinds of participating are welcome. >> > > >> > > Best, >> > > Dongjin >> > > >> > > -- >> > > *Dongjin Lee* >> > > >> > > *A hitchhiker in the mathematical world.* >> > > >> > > *github: <http://goog_969573159/>github.com/dongjinleekr >> > > <http://github.com/dongjinleekr>linkedin: >> > kr.linkedin.com/in/dongjinleekr >> > > <http://kr.linkedin.com/in/dongjinleekr>slideshare: >> > www.slideshare.net/dongjinleekr >> > > <http://www.slideshare.net/dongjinleekr>* >> > > >> > >> > -- > *Dongjin Lee* > > *A hitchhiker in the mathematical world.* > > *github: <http://goog_969573159/>github.com/dongjinleekr > <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > www.slideshare.net/dongjinleekr > <http://www.slideshare.net/dongjinleekr>* > -- *Dongjin Lee* *A hitchhiker in the mathematical world.* *github: <http://goog_969573159/>github.com/dongjinleekr <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr <http://kr.linkedin.com/in/dongjinleekr>slideshare: www.slideshare.net/dongjinleekr <http://www.slideshare.net/dongjinleekr>*
