Sounds reasonable to me. Ismael
On Sat, 18 Aug 2018, 12:20 Jason Gustafson, <ja...@confluent.io> wrote: > Hey Ismael, > > Your summary looks good to me. I think it might also be a good idea to add > a new UNSUPPORTED_COMPRESSION_TYPE error code to go along with the version > bumps. We won't be able to use it for old api versions since the clients > will not understand it, but we can use it going forward so that we're not > stuck in a similar situation with a new message format and a new codec to > support. Another option is to use UNSUPPORTED_FOR_MESSAGE_FORMAT, but it is > not as explicit. > > -Jason > > On Fri, Aug 17, 2018 at 5:19 PM, Ismael Juma <ism...@juma.me.uk> wrote: > > > Hi Dongjin and Jason, > > > > I would agree. My summary: > > > > 1. Support zstd with message format 2 only. > > 2. Bump produce and fetch request versions. > > 3. Provide broker errors whenever possible based on the request version > and > > rely on clients for the cases where the broker can't validate efficiently > > (example message format 2 consumer that supports the latest fetch version > > but doesn't support zstd). > > > > If there's general agreement on this, I suggest we update the KIP to > state > > the proposal and to move the rejected options to its own section. And > then > > start a vote! > > > > Ismael > > > > On Fri, Aug 17, 2018 at 4:00 PM Jason Gustafson <ja...@confluent.io> > > wrote: > > > > > Hi Dongjin, > > > > > > Yes, that's a good summary. For clients which support v2, the client > can > > > parse the message format and hopefully raise a useful error message > > > indicating the unsupported compression type. For older clients, our > > options > > > are probably (1) to down-convert to the old format using no compression > > > type, or (2) to return an error code. I'm leaning toward the latter as > > the > > > simpler solution, but the challenge is finding a good error code. Two > > > possibilities might be INVALID_REQUEST or CORRUPT_MESSAGE. The downside > > is > > > that old clients probably won't get a helpful message. However, at > least > > > the behavior will be consistent in the sense that all clients will fail > > if > > > they do not support zstandard. > > > > > > What do you think? > > > > > > Thanks, > > > Jason > > > > > > On Fri, Aug 17, 2018 at 8:08 AM, Dongjin Lee <dong...@apache.org> > wrote: > > > > > > > Thanks Jason, I reviewed the down-converting logic following your > > > > explanation.[^1] You mean the following routines, right? > > > > > > > > - > > > > https://github.com/apache/kafka/blob/trunk/core/src/ > > > > main/scala/kafka/server/KafkaApis.scala#L534 > > > > - > > > > https://github.com/apache/kafka/blob/trunk/clients/src/ > > > > main/java/org/apache/kafka/common/record/LazyDownConversionRecords. > > > > java#L165 > > > > - > > > > https://github.com/apache/kafka/blob/trunk/clients/src/ > > > > main/java/org/apache/kafka/common/record/RecordsUtil.java#L40 > > > > > > > > It seems like your stance is like following: > > > > > > > > 1. In principle, Kafka does not change the compression codec when > > > > down-converting, since it requires inspecting the fetched data, which > > is > > > > expensive. > > > > 2. However, there are some cases the fetched data is inspected > anyway. > > In > > > > this case, we can provide compression conversion from Zstandard to > > > > classical ones[^2]. > > > > > > > > And from what I understand, the cases where the client without > > ZStandard > > > > support receives ZStandard compressed records can be organized into > two > > > > cases: > > > > > > > > a. The 'compression.type' configuration of given topic is 'producer' > > and > > > > the producer compressed the records with ZStandard. (that is, using > > > > ZStandard implicitly.) > > > > b. The 'compression.type' configuration of given topic is 'zstd'; > that > > > is, > > > > using ZStandard explicitly. > > > > > > > > As you stated, we don't have to handle the case b specially. So, It > > seems > > > > like we can narrow the focus of the problem by joining case 1 and > case > > b > > > > like the following: > > > > > > > > > Given the topic with 'producer' as its 'compression.type' > > > configuration, > > > > ZStandard compressed records and old client without ZStandard, is > there > > > any > > > > case we need to inspect the records and can change the compression > > type? > > > If > > > > so, can we provide compression type converting? > > > > > > > > Do I understand correctly? > > > > > > > > Best, > > > > Dongjin > > > > > > > > [^1]: I'm sorry, I found that I was a little bit misunderstanding how > > API > > > > version works, after reviewing the downconvert logic & the protocol > > > > documentation <https://kafka.apache.org/protocol>. > > > > [^2]: None, Gzip, Snappy, Lz4. > > > > > > > > On Tue, Aug 14, 2018 at 2:16 AM Jason Gustafson <ja...@confluent.io> > > > > wrote: > > > > > > > > > > > > > > > > But in my opinion, since the client will fail with the API > version, > > > so > > > > we > > > > > > don't need to down-convert the messages anyway. Isn't it? So, I > > think > > > > we > > > > > > don't care about this case. (I'm sorry, I am not familiar with > > > > > down-convert > > > > > > logic.) > > > > > > > > > > > > > > > Currently the broker down-converts automatically when it receives > an > > > old > > > > > version of the fetch request (a version which is known to predate > the > > > > > message format in use). Typically when down-converting the message > > > > format, > > > > > we use the same compression type, but there is not much point in > > doing > > > so > > > > > when we know the client doesn't support it. So if zstandard is in > > use, > > > > and > > > > > we have to down-convert anyway, then we can choose to use a > different > > > > > compression type or no compression type. > > > > > > > > > > From my perspective, there is no significant downside to bumping > the > > > > > protocol version and it has several potential benefits. Version > bumps > > > are > > > > > cheap. The main question mark in my mind is about down-conversion. > > > > Figuring > > > > > out whether down-conversion is needed is hard generally without > > > > inspecting > > > > > the fetched data, which is expensive. I think we agree in principle > > > that > > > > we > > > > > do not want to have to pay this cost generally and prefer the > clients > > > to > > > > > fail when they see an unhandled compression type. The point I was > > > making > > > > is > > > > > that there are some cases where we are either inspecting the data > > > anyway > > > > > (because we have to down-convert the message format), or we have an > > > easy > > > > > way to tell whether zstandard is in use (the topic has it > configured > > > > > explicitly). In the latter case, we don't have to handle it > > specially. > > > > But > > > > > we do have to decide how we will handle down-conversion to older > > > formats. > > > > > > > > > > -Jason > > > > > > > > > > On Sun, Aug 12, 2018 at 5:15 PM, Dongjin Lee <dong...@apache.org> > > > wrote: > > > > > > > > > > > Colin and Jason, > > > > > > > > > > > > Thanks for your opinions. In summarizing, the Pros and Cons of > > > bumping > > > > > > fetch API version are: > > > > > > > > > > > > Cons: > > > > > > > > > > > > - The Broker can't know whether a given message batch is > compressed > > > > with > > > > > > zstd or not. > > > > > > - Need some additional logic for the topic explicitly configured > to > > > use > > > > > > zstd. > > > > > > > > > > > > Pros: > > > > > > > > > > > > - The broker doesn't need to conduct expensive down-conversion. > > > > > > - Can message the users to update their client. > > > > > > > > > > > > So, opinions for the backward-compatibility policy by far: > > > > > > > > > > > > - A: bump the API version - +2 (Colin, Jason) > > > > > > - B: leave unchanged - +1 (Viktor) > > > > > > > > > > > > Here are my additional comments: > > > > > > > > > > > > @Colin > > > > > > > > > > > > I greatly appreciate your response. In the case of the dictionary > > > > > support, > > > > > > of course, this issue should be addressed later so we don't need > it > > > in > > > > > the > > > > > > first version. You are right - it is not late to try it after > some > > > > > > benchmarks. What I mean is, we should keep in mind on that > > potential > > > > > > feature. > > > > > > > > > > > > @Jason > > > > > > > > > > > > You wrote, > > > > > > > > > > > > > Similarly, if we have to down-convert anyway because the client > > > does > > > > > not > > > > > > understand the message format, then we could also use a different > > > > > > compression type. > > > > > > > > > > > > But in my opinion, since the client will fail with the API > version, > > > so > > > > we > > > > > > don't need to down-convert the messages anyway. Isn't it? So, I > > think > > > > we > > > > > > don't care about this case. (I'm sorry, I am not familiar with > > > > > down-convert > > > > > > logic.) > > > > > > > > > > > > Please give more opinions. Thanks! > > > > > > > > > > > > - Dongjin > > > > > > > > > > > > > > > > > > On Wed, Aug 8, 2018 at 6:41 AM Jason Gustafson < > ja...@confluent.io > > > > > > > > wrote: > > > > > > > > > > > > > Hey Colin, > > > > > > > > > > > > > > The problem for the fetch API is that the broker does not > > generally > > > > > know > > > > > > if > > > > > > > a batch was compressed with zstd unless it parses it. I think > the > > > > goal > > > > > > here > > > > > > > is to avoid the expensive down-conversion that is needed to > > ensure > > > > > > > compatibility because it is only necessary if zstd is actually > in > > > > use. > > > > > > But > > > > > > > as long as old clients can parse the message format, they > should > > > get > > > > a > > > > > > > reasonable error if they see an unsupported compression type in > > the > > > > > > > attributes. Basically the onus is on users to ensure that their > > > > > consumers > > > > > > > have been updated prior to using zstd. It seems like a > reasonable > > > > > > tradeoff > > > > > > > to me. There are a couple cases that might be worth thinking > > > through: > > > > > > > > > > > > > > 1. If a topic is explicitly configured to use zstd, then we > don't > > > > need > > > > > to > > > > > > > check the fetched data for the compression type to know if we > > need > > > > > > > down-conversion. If we did bump the Fetch API version, then we > > > could > > > > > > handle > > > > > > > this case by either down-converting using a different > compression > > > > type > > > > > or > > > > > > > returning an error. > > > > > > > 2. Similarly, if we have to down-convert anyway because the > > client > > > > does > > > > > > not > > > > > > > understand the message format, then we could also use a > different > > > > > > > compression type. > > > > > > > > > > > > > > For the produce API, I think it's reasonable to bump the api > > > version. > > > > > > This > > > > > > > can be used by clients to check whether a broker supports zstd. > > For > > > > > > > example, we might support a list of preferred compression types > > in > > > > the > > > > > > > producer and we could use the broker to detect which version to > > > use. > > > > > > > > > > > > > > -Jason > > > > > > > > > > > > > > On Tue, Aug 7, 2018 at 1:32 PM, Colin McCabe < > cmcc...@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > > > Thanks for bumping this, Dongjin. ZStd is a good compression > > > codec > > > > > > and I > > > > > > > > hope we can get this support in soon! > > > > > > > > > > > > > > > > I would say we can just bump the API version to indicate that > > > ZStd > > > > > > > support > > > > > > > > is expected in new clients. We probably need some way of > > > > indicating > > > > > to > > > > > > > the > > > > > > > > older clients that they can't consume the partitions, as > well. > > > > > Perhaps > > > > > > > we > > > > > > > > can use the UNSUPPORTED_FOR_MESSAGE_FORMAT error? > > > > > > > > > > > > > > > > The license thing seems straightforward -- it's just a matter > > of > > > > > adding > > > > > > > > the text to the right files as per ASF guidelines. > > > > > > > > > > > > > > > > With regard to the dictionary support, do we really need that > > in > > > > the > > > > > > > first > > > > > > > > version? Hopefully message batches are big enough that this > > > isn't > > > > > > > needed. > > > > > > > > Some benchmarks might help here. > > > > > > > > > > > > > > > > best, > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 7, 2018, at 08:02, Dongjin Lee wrote: > > > > > > > > > As Kafka 2.0.0 was released, let's reboot this issue, > KIP-110 > > > > > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > > > > 110%3A+Add+Codec+for+ZStandard+Compression> > > > > > > > > > . > > > > > > > > > > > > > > > > > > For newcomers, Here is some summary of the history: KIP-110 > > was > > > > > > > > originally > > > > > > > > > worked for the issue KAFKA-4514 but, it lacked benchmark > > > results > > > > to > > > > > > get > > > > > > > > the > > > > > > > > > agreement of the community. Later, Ivan Babrou and some > other > > > > users > > > > > > who > > > > > > > > > adopted the patch provided their excellent performance > report > > > > which > > > > > > is > > > > > > > > now > > > > > > > > > included in the KIP, but it postponed again because of the > > > > > community > > > > > > > was > > > > > > > > > busy for 2.0.0 release. It is why I now reboot this issue. > > > > > > > > > > > > > > > > > > The following is the current status of the feature: You can > > > check > > > > > the > > > > > > > > > current draft implementation here > > > > > > > > > <https://github.com/apache/kafka/pull/2267>. It is based > on > > > zstd > > > > > > 1.3.5 > > > > > > > > and > > > > > > > > > periodically rebased onto the latest trunk[^1]. > > > > > > > > > > > > > > > > > > The issues that should be addressed is like following: > > > > > > > > > > > > > > > > > > *1. Backward Compatibility* > > > > > > > > > > > > > > > > > > To support old consumers, we need to take a strategy to > > handle > > > > the > > > > > > old > > > > > > > > > consumers. Current candidates are: > > > > > > > > > > > > > > > > > > - Bump API version > > > > > > > > > - Leave unchanged: let the old clients fail. > > > > > > > > > - Improve the error messages: > > > > > > > > > > > > > > > > > > *2. Dictionary Support* > > > > > > > > > > > > > > > > > > To support zstd's dictionary feature in the future (if > > needed), > > > > we > > > > > > need > > > > > > > > to > > > > > > > > > sketch how it should be and leave some room for it. As of > > now, > > > > > there > > > > > > > has > > > > > > > > > been no discussion on this topic yet. > > > > > > > > > > > > > > > > > > *3. License* > > > > > > > > > > > > > > > > > > To use this feature, we need to add license of zstd and > > > zstd-jni > > > > to > > > > > > the > > > > > > > > > project. (Thanks to Viktor Somogyi for raising this issue!) > > It > > > > > seems > > > > > > > like > > > > > > > > > what Apache Spark did would be a good example but there has > > > been > > > > no > > > > > > > > > discussion yet. > > > > > > > > > > > > > > > > > > You can find the details of the above issues in the KIP > > > document. > > > > > > > Please > > > > > > > > > have a look when you are free, and give me feedback. All > > kinds > > > of > > > > > > > > > participating are welcome. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > [^1]: At the time of writing, commit 6b4fb8152. > > > > > > > > > > > > > > > > > > On Sat, Jul 14, 2018 at 10:45 PM Dongjin Lee < > > > dong...@apache.org > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Sorry for the late reply. > > > > > > > > > > > > > > > > > > > > In short, I could not submit the updated KIP by the > feature > > > > > freeze > > > > > > > > > > deadline of 2.0.0. For this reason, it will not be > included > > > in > > > > > the > > > > > > > > 2.0.0 > > > > > > > > > > release and all discussion for this issue were postponed > > > after > > > > > the > > > > > > > > release > > > > > > > > > > of 2.0.0. > > > > > > > > > > > > > > > > > > > > I have been updating the PR following recent updates. > Just > > > > now, I > > > > > > > > rebased > > > > > > > > > > it against the latest trunk and updated the zstd version > > into > > > > > > 1.3.5. > > > > > > > > If you > > > > > > > > > > need some request, don't hesitate to notify me. (But not > > this > > > > > > thread > > > > > > > - > > > > > > > > just > > > > > > > > > > send me the message directly.) > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > On Tue, Jul 10, 2018 at 11:57 PM Bobby Evans < > > > bo...@apache.org > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> I there any update on this. The performance > improvements > > > are > > > > > > quite > > > > > > > > > >> impressive and I really would like to stop forking kafka > > > just > > > > to > > > > > > get > > > > > > > > this > > > > > > > > > >> in. > > > > > > > > > >> > > > > > > > > > >> Thanks, > > > > > > > > > >> > > > > > > > > > >> Bobby > > > > > > > > > >> > > > > > > > > > >> On Wed, Jun 13, 2018 at 8:56 PM Dongjin Lee < > > > > dong...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >> > Ismael, > > > > > > > > > >> > > > > > > > > > > >> > Oh, I forgot all of you are on working frenzy for 2.0! > > No > > > > > > problem, > > > > > > > > take > > > > > > > > > >> > your time. I am also working at another issue now. > Thank > > > you > > > > > for > > > > > > > > > >> letting me > > > > > > > > > >> > know. > > > > > > > > > >> > > > > > > > > > > >> > Best, > > > > > > > > > >> > Dongjin > > > > > > > > > >> > > > > > > > > > > >> > On Wed, Jun 13, 2018, 11:44 PM Ismael Juma < > > > > isma...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > > >> > > Sorry for the delay Dongjin. Everyone is busy > > finalising > > > > > > 2.0.0. > > > > > > > > This > > > > > > > > > >> KIP > > > > > > > > > >> > > seems like a great candidate for 2.1.0 and hopefully > > > there > > > > > > will > > > > > > > be > > > > > > > > > >> more > > > > > > > > > >> > of > > > > > > > > > >> > > a discussion next week. :) > > > > > > > > > >> > > > > > > > > > > > >> > > Ismael > > > > > > > > > >> > > > > > > > > > > > >> > > On Wed, 13 Jun 2018, 05:17 Dongjin Lee, < > > > > dong...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > > > >> > > > Hello. I just updated my draft implementation: > > > > > > > > > >> > > > > > > > > > > > > >> > > > 1. Rebased to latest trunk (commit 5145d6b) > > > > > > > > > >> > > > 2. Apply ZStd 1.3.4 > > > > > > > > > >> > > > > > > > > > > > > >> > > > You can check out the implementation from here > > > > > > > > > >> > > > <https://github.com/apache/kafka/pull/2267>. If > you > > > > > > > experience > > > > > > > > any > > > > > > > > > >> > > problem > > > > > > > > > >> > > > running it, don't hesitate to give me a mention. > > > > > > > > > >> > > > > > > > > > > > > >> > > > Best, > > > > > > > > > >> > > > Dongjin > > > > > > > > > >> > > > > > > > > > > > > >> > > > On Tue, Jun 12, 2018 at 6:50 PM Dongjin Lee < > > > > > > > dong...@apache.org > > > > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > > >> > > > > > > > > > > > > >> > > > > Here is the short conclusion about the license > > > > problem: > > > > > > *We > > > > > > > > can > > > > > > > > > >> use > > > > > > > > > >> > > zstd > > > > > > > > > >> > > > > and zstd-jni without any problem, but we need to > > > > include > > > > > > > their > > > > > > > > > >> > license, > > > > > > > > > >> > > > > e.g., BSD license.* > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > Both of BSD 2 Clause License & 3 Clause License > > > > requires > > > > > > to > > > > > > > > > >> include > > > > > > > > > >> > the > > > > > > > > > >> > > > > license used, and BSD 3 Clause License requires > > that > > > > the > > > > > > > name > > > > > > > > of > > > > > > > > > >> the > > > > > > > > > >> > > > > contributor can't be used to endorse or promote > > the > > > > > > product. > > > > > > > > > >> That's > > > > > > > > > >> > it > > > > > > > > > >> > > > > < > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> http://www.mikestratton.net/2011/12/is-bsd-license- > > > > > > > > compatible-with-apache-2-0-license/ > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > - They are not listed in the list of prohibited > > > > licenses > > > > > > > > > >> > > > > <https://www.apache.org/legal/ > > > > resolved.html#category-x> > > > > > > > also. > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > Here is how Spark did for it > > > > > > > > > >> > > > > <https://issues.apache.org/ > > jira/browse/SPARK-19112 > > > >: > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > - They made a directory dedicated to the > > dependency > > > > > > license > > > > > > > > files > > > > > > > > > >> > > > > < > > > https://github.com/apache/spark/tree/master/licenses > > > > > > > > > > > and > > > > > > > > added > > > > > > > > > >> > > > licenses > > > > > > > > > >> > > > > for Zstd > > > > > > > > > >> > > > > < > > > > > > > > > >> > https://github.com/apache/spark/blob/master/licenses/ > > > > > > > > LICENSE-zstd.txt > > > > > > > > > >> > > > > > > > > > > > > >> > > > & > > > > > > > > > >> > > > > Zstd-jni > > > > > > > > > >> > > > > < > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> https://github.com/apache/spark/blob/master/licenses/ > > > > > > > > LICENSE-zstd-jni.txt > > > > > > > > > >> > > > > > > > > > > >> > > > > . > > > > > > > > > >> > > > > - Added a link to the original license files in > > > > LICENSE. > > > > > > > > > >> > > > > < > https://github.com/apache/spark/pull/18805/files > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > If needed, I can make a similar update. > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > Thanks for pointing out this problem, Viktor! > Nice > > > > > catch! > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > Best, > > > > > > > > > >> > > > > Dongjin > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > On Mon, Jun 11, 2018 at 11:50 PM Dongjin Lee < > > > > > > > > dong...@apache.org> > > > > > > > > > >> > > wrote: > > > > > > > > > >> > > > > > > > > > > > > > >> > > > >> I greatly appreciate your comprehensive > > reasoning. > > > > so: > > > > > +1 > > > > > > > > for b > > > > > > > > > >> > until > > > > > > > > > >> > > > now. > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> For the license issues, I will have a check on > > how > > > > the > > > > > > over > > > > > > > > > >> projects > > > > > > > > > >> > > are > > > > > > > > > >> > > > >> doing and share the results. > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> Best, > > > > > > > > > >> > > > >> Dongjin > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> On Mon, Jun 11, 2018 at 10:08 PM Viktor > Somogyi < > > > > > > > > > >> > > > viktorsomo...@gmail.com> > > > > > > > > > >> > > > >> wrote: > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >>> Hi Dongjin, > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>> A couple of comments: > > > > > > > > > >> > > > >>> I would vote for option b. in the "backward > > > > > > compatibility" > > > > > > > > > >> section. > > > > > > > > > >> > > My > > > > > > > > > >> > > > >>> reasoning for this is that users upgrading to > a > > > zstd > > > > > > > > compatible > > > > > > > > > >> > > version > > > > > > > > > >> > > > >>> won't start to use it automatically, so manual > > > > > > > > reconfiguration > > > > > > > > > >> is > > > > > > > > > >> > > > >>> required. > > > > > > > > > >> > > > >>> Therefore an upgrade won't mess up the > cluster. > > If > > > > not > > > > > > all > > > > > > > > the > > > > > > > > > >> > > clients > > > > > > > > > >> > > > >>> are > > > > > > > > > >> > > > >>> upgraded but just some of them and they'd > start > > to > > > > use > > > > > > > zstd > > > > > > > > > >> then it > > > > > > > > > >> > > > would > > > > > > > > > >> > > > >>> cause errors in the cluster. I'd like to > presume > > > > > though > > > > > > > that > > > > > > > > > >> this > > > > > > > > > >> > is > > > > > > > > > >> > > a > > > > > > > > > >> > > > >>> very > > > > > > > > > >> > > > >>> obvious failure case and nobody should be > > > surprised > > > > if > > > > > > it > > > > > > > > didn't > > > > > > > > > >> > > work. > > > > > > > > > >> > > > >>> I wouldn't choose a. as I think we should bump > > the > > > > > fetch > > > > > > > and > > > > > > > > > >> > produce > > > > > > > > > >> > > > >>> requests if it's a change in the message > format. > > > > > > Moreover > > > > > > > if > > > > > > > > > >> some > > > > > > > > > >> > of > > > > > > > > > >> > > > the > > > > > > > > > >> > > > >>> producers and the brokers are upgraded but > some > > of > > > > the > > > > > > > > consumers > > > > > > > > > >> > are > > > > > > > > > >> > > > not, > > > > > > > > > >> > > > >>> then we wouldn't prevent the error when the > old > > > > > consumer > > > > > > > > tries > > > > > > > > > >> to > > > > > > > > > >> > > > consume > > > > > > > > > >> > > > >>> the zstd compressed messages. > > > > > > > > > >> > > > >>> I wouldn't choose c. either as I think binding > > the > > > > > > > > compression > > > > > > > > > >> type > > > > > > > > > >> > > to > > > > > > > > > >> > > > an > > > > > > > > > >> > > > >>> API is not so obvious from the developer's > > > > > perspective. > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>> I would also prefer to use the existing > binding, > > > > > however > > > > > > > we > > > > > > > > must > > > > > > > > > >> > > > respect > > > > > > > > > >> > > > >>> the licenses: > > > > > > > > > >> > > > >>> "The code for these JNI bindings is licenced > > under > > > > > > > 2-clause > > > > > > > > BSD > > > > > > > > > >> > > > license. > > > > > > > > > >> > > > >>> The native Zstd library is licensed under > > 3-clause > > > > BSD > > > > > > > > license > > > > > > > > > >> and > > > > > > > > > >> > > > GPL2" > > > > > > > > > >> > > > >>> Based on the FAQ page > > > > > > > > > >> > > > >>> https://www.apache.org/legal/ > > > > resolved.html#category-a > > > > > > > > > >> > > > >>> we may use 2- and 3-clause BSD licenses but > the > > > > Apache > > > > > > > > license > > > > > > > > > >> is > > > > > > > > > >> > not > > > > > > > > > >> > > > >>> compatible with GPL2. I'm hoping that the > > > "3-clause > > > > > BSD > > > > > > > > license > > > > > > > > > >> and > > > > > > > > > >> > > > GPL2" > > > > > > > > > >> > > > >>> is really not an AND but an OR in this case, > but > > > I'm > > > > > no > > > > > > > > lawyer, > > > > > > > > > >> > just > > > > > > > > > >> > > > >>> wanted > > > > > > > > > >> > > > >>> to make the point that we should watch out for > > > > > licenses. > > > > > > > :) > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>> Regards, > > > > > > > > > >> > > > >>> Viktor > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>> On Sun, Jun 10, 2018 at 3:02 AM Ivan Babrou < > > > > > > > > ibob...@gmail.com> > > > > > > > > > >> > > wrote: > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>> > Hello, > > > > > > > > > >> > > > >>> > > > > > > > > > > >> > > > >>> > This is Ivan and I still very much support > the > > > > fact > > > > > > that > > > > > > > > zstd > > > > > > > > > >> > > > >>> compression > > > > > > > > > >> > > > >>> > should be included out of the box. > > > > > > > > > >> > > > >>> > > > > > > > > > > >> > > > >>> > Please think about the environment, you can > > save > > > > > > quite a > > > > > > > > lot > > > > > > > > > >> of > > > > > > > > > >> > > > >>> hardware > > > > > > > > > >> > > > >>> > with it. > > > > > > > > > >> > > > >>> > > > > > > > > > > >> > > > >>> > Thank you. > > > > > > > > > >> > > > >>> > > > > > > > > > > >> > > > >>> > On Sat, Jun 9, 2018 at 14:14 Dongjin Lee < > > > > > > > > dong...@apache.org> > > > > > > > > > >> > > wrote: > > > > > > > > > >> > > > >>> > > > > > > > > > > >> > > > >>> > > Since there are no responses for a week, I > > > > decided > > > > > > to > > > > > > > > > >> > reinitiate > > > > > > > > > >> > > > the > > > > > > > > > >> > > > >>> > > discussion thread. > > > > > > > > > >> > > > >>> > > > > > > > > > > > >> > > > >>> > > > > > > > > > > > >> > > > >>> > > > > > > > > > > > >> > > > >>> > > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > > > > 110%3A+Add+Codec+for+ZStandard+Compression > > > > > > > > > >> > > > >>> > > > > > > > > > > > >> > > > >>> > > This KIP is about to introduce ZStandard > > > > > Compression > > > > > > > > into > > > > > > > > > >> > Apache > > > > > > > > > >> > > > >>> Kafka. > > > > > > > > > >> > > > >>> > > The reason why it is posted again has a > > story: > > > > It > > > > > > was > > > > > > > > > >> > originally > > > > > > > > > >> > > > >>> posted > > > > > > > > > >> > > > >>> > to > > > > > > > > > >> > > > >>> > > the dev mailing list more than one year > ago > > > but > > > > > > since > > > > > > > it > > > > > > > > > >> has no > > > > > > > > > >> > > > >>> > performance > > > > > > > > > >> > > > >>> > > report included, it was postponed later. > But > > > > Some > > > > > > > people > > > > > > > > > >> > > (including > > > > > > > > > >> > > > >>> Ivan) > > > > > > > > > >> > > > >>> > > reported excellent performance report with > > the > > > > > draft > > > > > > > PR, > > > > > > > > > >> this > > > > > > > > > >> > > work > > > > > > > > > >> > > > >>> is now > > > > > > > > > >> > > > >>> > > reactivated. > > > > > > > > > >> > > > >>> > > > > > > > > > > > >> > > > >>> > > The updated KIP document includes some > > > expected > > > > > > > > problems and > > > > > > > > > >> > > their > > > > > > > > > >> > > > >>> > > candidate alternatives. Please have a look > > > when > > > > > you > > > > > > > are > > > > > > > > > >> free, > > > > > > > > > >> > and > > > > > > > > > >> > > > >>> give > > > > > > > > > >> > > > >>> > me a > > > > > > > > > >> > > > >>> > > feedback. All kinds of participating are > > > > welcome. > > > > > > > > > >> > > > >>> > > > > > > > > > > > >> > > > >>> > > Best, > > > > > > > > > >> > > > >>> > > Dongjin > > > > > > > > > >> > > > >>> > > > > > > > > > > > >> > > > >>> > > -- > > > > > > > > > >> > > > >>> > > *Dongjin Lee* > > > > > > > > > >> > > > >>> > > > > > > > > > > > >> > > > >>> > > *A hitchhiker in the mathematical world.* > > > > > > > > > >> > > > >>> > > > > > > > > > > > >> > > > >>> > > *github: <http://goog_969573159/>github > > > > > > > > .com/dongjinleekr > > > > > > > > > >> > > > >>> > > <http://github.com/dongjinleekr>linkedin: > > > > > > > > > >> > > > >>> > kr.linkedin.com/in/dongjinleekr > > > > > > > > > >> > > > >>> > > <http://kr.linkedin.com/in/dongjinleekr > > > > > >slideshare: > > > > > > > > > >> > > > >>> > www.slideshare.net/dongjinleekr > > > > > > > > > >> > > > >>> > > <http://www.slideshare.net/dongjinleekr>* > > > > > > > > > >> > > > >>> > > > > > > > > > > > >> > > > >>> > > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >> -- > > > > > > > > > >> > > > >> *Dongjin Lee* > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> *A hitchhiker in the mathematical world.* > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> *github: <http://goog_969573159/>github > > > > > > .com/dongjinleekr > > > > > > > > > >> > > > >> <http://github.com/dongjinleekr>linkedin: > > > > > > > > > >> > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > >> > > > >> <http://kr.linkedin.com/in/dongjinleekr > > > >slideshare: > > > > > > > > > >> > > > www.slideshare.net/dongjinleekr > > > > > > > > > >> > > > >> <http://www.slideshare.net/dongjinleekr>* > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > -- > > > > > > > > > >> > > > > *Dongjin Lee* > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > *github: <http://goog_969573159/> > > > > > github.com/dongjinleekr > > > > > > > > > >> > > > > <http://github.com/dongjinleekr>linkedin: > > > > > > > > > >> > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > >> > > > > <http://kr.linkedin.com/in/dongjinleekr > > >slideshare: > > > > > > > > > >> > > > www.slideshare.net/dongjinleekr > > > > > > > > > >> > > > > <http://www.slideshare.net/dongjinleekr>* > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > -- > > > > > > > > > >> > > > *Dongjin Lee* > > > > > > > > > >> > > > > > > > > > > > > >> > > > *A hitchhiker in the mathematical world.* > > > > > > > > > >> > > > > > > > > > > > > >> > > > *github: <http://goog_969573159/>github > > > > .com/dongjinleekr > > > > > > > > > >> > > > <http://github.com/dongjinleekr>linkedin: > > > > > > > > > >> > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > >> > > > <http://kr.linkedin.com/in/dongjinleekr > >slideshare: > > > > > > > > > >> > > > www.slideshare.net/dongjinleekr > > > > > > > > > >> > > > <http://www.slideshare.net/dongjinleekr>* > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> -- > > > > > > > > > >> *Dongjin Lee* > > > > > > > > > >> > > > > > > > > > >> *A hitchhiker in the mathematical world.* > > > > > > > > > >> > > > > > > > > > >> *github: <http://goog_969573159/> > github.com/dongjinleekr > > > > > > > > > >> <http://github.com/dongjinleekr>linkedin: > > > kr.linkedin.com/in/ > > > > > > > > dongjinleekr > > > > > > > > > >> <http://kr.linkedin.com/in/dongjinleekr>slideshare: > > > > > > > > www.slideshare.net/dongjinleekr > > > > > > > > > >> <http://www.slideshare.net/dongjinleekr>* > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > > > > > > > <http://github.com/dongjinleekr>linkedin: > > kr.linkedin.com/in/ > > > > > > > > dongjinleekr > > > > > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > > > > > > > > > www.slideshare.net/dongjinleekr > > > > > > > > > <http://www.slideshare.net/dongjinleekr>* > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > > > > > <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/ > > > > > > dongjinleekr > > > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > > > > > www.slideshare.net/ > > > > > > dongjinleekr > > > > > > > <http://www.slideshare.net/dongjinleekr>* > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > *Dongjin Lee* > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > > > <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/ > > > > dongjinleekr > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > > > www.slideshare.net/ > > > > dongjinleekr > > > > > <http://www.slideshare.net/dongjinleekr>* > > > > > > > > > > > > > > >
