Thanks; Matthias, Very useful.
On Mon, Apr 6, 2020 at 8:05 PM Matthias J. Sax <mj...@apache.org> wrote: > I guess one important point to mention is why Kafka Streams needs the > internal config though: it's about a save upgrade path. > > Even if the user tells us that they are on old brokers, we call the new > `sendOffsetsToTransaction()` API blindly and let the producer downgrade > the request. If the user upgrades their brokers, the producer can > automatically switch to the new request type. During this phase, the old > `transaction.id` fencing _and_ the new fetch-offset-request fencing are > both active at the same time. > > Because both fencing mechanism are active, a user can now switch the > config to enable eos-beta and do a single round of rolling bounces. > > It is not save to have only `transaction.id` fencing (but not > fetch-offset-request fencing) and switch to fetch-offset-request fencing > in a single rolling bounce. If an error occurs during the upgrade, EOS > might be violated. > > Thus, in the end the internal config is a way to simplify the upgrade > mechanism. We could also have chosen to add a new public config to Kafka > Streams instead, but we wanted to improve the user experience. > > Thus, for your use case Gary, you would need to have one more config > value for the upgrade mode in which you still use the old model but > enable fetch-offset-request fencing in parallel. An upgrade would > require two rolling bounce, first to enable upgrade mode, second to > enable the new feature. > > > -Matthias > > On 4/6/20 12:10 PM, Gary Russell wrote: > > Thanks, all, > > > >> Just to clarify, even for Streams client it cannot detect automatically > the > > broker's version and hence as KIP-447 proposed, the customer needs to > set a > > config value indicating that she is assured the broker version is newer > and > > hence the new API can be used. > > > > Yes, I noticed that; Ok, I'll go the same way, then and make it a > > configuration option. > > > > Yes, KIP-584 looks interesting to me. > > > > Thanks again. > > > > > > On Sun, Apr 5, 2020 at 5:30 PM Guozhang Wang <wangg...@gmail.com> wrote: > > > >> Hello Gary, > >> > >> Just to clarify, even for Streams client it cannot detect automatically > the > >> broker's version and hence as KIP-447 proposed, the customer needs to > set a > >> config value indicating that she is assured the broker version is newer > and > >> hence the new API can be used. On the other hand, if the config is not > set, > >> even if broker is on newer version we would still use the old behavior > (one > >> producer per task) and the although overloaded sendOffsetsToTransaction > is > >> used, it would not be much helpful in the scope of KIP-447. > >> > >> As for your case, currently there's no good ways to determine if the new > >> overloaded function can be safely used except the users need to indicate > >> (via configs, for example) that she's assured the broker version is > newer. > >> I think the ultimate solution for you would be KIP-584, in which client > can > >> dynamically ask the broker cluster which version(s) they would support > and > >> decides which function to trigger accordingly. > >> > >> > >> Guozhang > >> > >> > >> On Sat, Apr 4, 2020 at 2:46 PM Boyang Chen <reluctanthero...@gmail.com> > >> wrote: > >> > >>> That’s a fair point Ismael. After a second thought, I feel that if Gary > >> is > >>> building frameworks for general purpose usage, relying on private flag > >>> seems not a good idea. > >>> > >>> On Sat, Apr 4, 2020 at 10:01 AM Ismael Juma <ism...@juma.me.uk> wrote: > >>> > >>>> The internal config was meant to be internal, right? That is, no > >>>> compatibility guarantees are offered? The current discussion implies > we > >>> are > >>>> considering it a public config. > >>>> > >>>> Ismael > >>>> > >>>> On Sat, Apr 4, 2020 at 9:31 AM Boyang Chen <boy...@confluent.io> > >> wrote: > >>>> > >>>>> For Gary's case, I think the internal config should be a sort of > >> help, > >>>> and > >>>>> not violating our public agreement. > >>>>> > >>>>> On Fri, Apr 3, 2020 at 7:44 PM Matthias J. Sax <mj...@apache.org> > >>> wrote: > >>>>> > >>>>>> I guess you would need to catch the exception and retry? > >>>>>> > >>>>>> It's a little unfortunate. Not sure if we could back-port the > >>> internal > >>>>>> producer config that we add in 2.6 for auto-downgrade to a 2.5 bug > >>> fix > >>>>>> release? > >>>>>> > >>>>>> > >>>>>> -Matthias > >>>>>> > >>>>>> > >>>>>> On 4/2/20 7:25 PM, Gary Russell wrote: > >>>>>>> Thanks Mattias > >>>>>>> > >>>>>>>> Hence, why do you want/need to switch to the newer overload that > >>>> only > >>>>>>> works for 2.5+ brokers? > >>>>>>> > >>>>>>> So I can choose to use the producer per consumer thread Vs. the > >>>>> producer > >>>>>>> per group/topic/partition threading model for zombie fencing, > >> based > >>>> on > >>>>>> the > >>>>>>> broker version. > >>>>>>> > >>>>>>> I don't have the same luxury as kafka streams (i.e. don't use > >>> streams > >>>>> 2.6 > >>>>>>> unless you have 2.5+ brokers). > >>>>>>> > >>>>>>> I add new features with each minor release (and try to use the > >>> latest > >>>>>>> kafka-clients as they become available). > >>>>>>> > >>>>>>> Users may want other new features, not related to EOS, and they > >>> might > >>>>>> stay > >>>>>>> on old brokers. > >>>>>>> > >>>>>>> Other users might want to take advantage of the improved > >>> performance > >>>> of > >>>>>> the > >>>>>>> new EOS so I need to support both APIs. > >>>>>>> > >>>>>>> Many enterprises take forever to upgrade their brokers. I > >> recently > >>>> had > >>>>> a > >>>>>>> question of why my latest version won't work with an 0.9.x.x > >> broker > >>>>>> (sigh). > >>>>>>> > >>>>>>> Spring versioning rules don't allow me to bump kafka-clients > >>> versions > >>>>> in > >>>>>> a > >>>>>>> patch release so I am already supporting 4 active branches and I > >> am > >>>>>> trying > >>>>>>> to avoid supporting a fifth. > >>>>>>> > >>>>>>> Thanks again. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Thu, Apr 2, 2020 at 8:23 PM Matthias J. Sax <mj...@apache.org > >>> > >>>>> wrote: > >>>>>>> > >>>>>>>> Gary, > >>>>>>>> > >>>>>>>> thanks for the question. We recently had a discussion about the > >>>> exact > >>>>>>>> some topic: > >>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E > >>>>>>>> > >>>>>>>> Note that the "old" `sendOffsetsToTransaction(..., String > >>> groupId)` > >>>> is > >>>>>>>> not deprecated. Hence, why do you want/need to switch to the > >> newer > >>>>>>>> overload that only works for 2.5+ brokers? For many/most cases, > >>> the > >>>>>>>> "old" API that is compatible with older broker still does what > >> you > >>>>> need > >>>>>>>> and there in not need to switch to the newer API. > >>>>>>>> > >>>>>>>> > >>>>>>>> -Matthias > >>>>>>>> > >>>>>>>> > >>>>>>>> On 4/2/20 1:54 PM, Gary Russell wrote: > >>>>>>>>> Thanks, Boyang, > >>>>>>>>> > >>>>>>>>> I maintain a framework (Spring for Apache Kafka) that sits on > >> top > >>>> of > >>>>>> the > >>>>>>>>> clients, and I would like to be able to support all broker > >>>> versions. > >>>>> I > >>>>>>>>> don't have control over what brokers my users are using. > >>>>>>>>> > >>>>>>>>> You guys have done a great job since 0.10.2.0 (I think) of > >>>> supporting > >>>>>>>> older > >>>>>>>>> brokers from newer clients but this one's a blocker for me. > >>>>>>>>> > >>>>>>>>> My framework will enforce the proper semantics for EOS, > >> depending > >>>> on > >>>>>> the > >>>>>>>>> broker version, but I need to know which model to use at > >> runtime. > >>>>>>>>> > >>>>>>>>> As I said, I can have a property that the user can set to tell > >>> the > >>>>>>>>> framework that the broker is >= 2.5 but it would be cleaner if > >> I > >>>>> could > >>>>>>>> stay > >>>>>>>>> away from that. > >>>>>>>>> > >>>>>>>>> Something like KafkaAdminClient.brokerApi() (or add the lowest > >>>>>> API/broker > >>>>>>>>> version to describeCluster()), would be helpful. > >>>>>>>>> > >>>>>>>>> Worst case, I'll add a configuration option. > >>>>>>>>> > >>>>>>>>> Thanks. > >>>>>>>>> > >>>>>>>>> On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen < > >>>>> reluctanthero...@gmail.com > >>>>>>> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Thanks for the question Gary. The reasoning for crash the new > >>>>>>>>>> sendTxnOffsets API is because we don't want users to > >>> unconsciously > >>>>>>>> violate > >>>>>>>>>> the EOS guarantee. In your case, using this API with 2.4.1 is > >>> not > >>>>>>>> supported > >>>>>>>>>> anyway, so the upgrade path has to start from broker first to > >>> 2.5, > >>>>> and > >>>>>>>> then > >>>>>>>>>> client binaries. Is there any further concern that blocks you > >>> from > >>>>>>>> getting > >>>>>>>>>> the broker side upgrade first before using the new API? > >>>>>>>>>> > >>>>>>>>>> Boyang > >>>>>>>>>> > >>>>>>>>>> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell < > >>> gruss...@pivotal.io> > >>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Is there any way to determine the broker version in the > >>>>>> kafka-clients? > >>>>>>>>>>> > >>>>>>>>>>> I need to determine whether I can use the new > >>>>>> sendOffsetsToTransaction > >>>>>>>>>>> with ConsumerGroupMetadata or use the old one. > >>>>>>>>>>> > >>>>>>>>>>> If I use the new API with a 2.4.1 broker, I get > >>>>>>>>>>> > >>>>>>>>>>> UpsupportedVersionException: Attempted to write a non-default > >>>>>>>>>> generationId > >>>>>>>>>>> at version 2 > >>>>>>>>>>> > >>>>>>>>>>> Alternatively, couldn't the client simply extract the groupId > >>>> from > >>>>>> the > >>>>>>>>>>> ConsumerGroupMetadata and use the old struct if the broker is > >>> too > >>>>>> old? > >>>>>>>>>>> > >>>>>>>>>>> I'd rather not have a user property in my framework to tell > >> us > >>>>> which > >>>>>>>> API > >>>>>>>>>> to > >>>>>>>>>>> use. > >>>>>>>>>>> > >>>>>>>>>>> Thanks in advance. > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > >> > >> -- > >> -- Guozhang > >> > > > >
