I guess one important point to mention is why Kafka Streams needs the internal config though: it's about a save upgrade path.
Even if the user tells us that they are on old brokers, we call the new `sendOffsetsToTransaction()` API blindly and let the producer downgrade the request. If the user upgrades their brokers, the producer can automatically switch to the new request type. During this phase, the old `transaction.id` fencing _and_ the new fetch-offset-request fencing are both active at the same time. Because both fencing mechanism are active, a user can now switch the config to enable eos-beta and do a single round of rolling bounces. It is not save to have only `transaction.id` fencing (but not fetch-offset-request fencing) and switch to fetch-offset-request fencing in a single rolling bounce. If an error occurs during the upgrade, EOS might be violated. Thus, in the end the internal config is a way to simplify the upgrade mechanism. We could also have chosen to add a new public config to Kafka Streams instead, but we wanted to improve the user experience. Thus, for your use case Gary, you would need to have one more config value for the upgrade mode in which you still use the old model but enable fetch-offset-request fencing in parallel. An upgrade would require two rolling bounce, first to enable upgrade mode, second to enable the new feature. -Matthias On 4/6/20 12:10 PM, Gary Russell wrote: > Thanks, all, > >> Just to clarify, even for Streams client it cannot detect automatically the > broker's version and hence as KIP-447 proposed, the customer needs to set a > config value indicating that she is assured the broker version is newer and > hence the new API can be used. > > Yes, I noticed that; Ok, I'll go the same way, then and make it a > configuration option. > > Yes, KIP-584 looks interesting to me. > > Thanks again. > > > On Sun, Apr 5, 2020 at 5:30 PM Guozhang Wang <wangg...@gmail.com> wrote: > >> Hello Gary, >> >> Just to clarify, even for Streams client it cannot detect automatically the >> broker's version and hence as KIP-447 proposed, the customer needs to set a >> config value indicating that she is assured the broker version is newer and >> hence the new API can be used. On the other hand, if the config is not set, >> even if broker is on newer version we would still use the old behavior (one >> producer per task) and the although overloaded sendOffsetsToTransaction is >> used, it would not be much helpful in the scope of KIP-447. >> >> As for your case, currently there's no good ways to determine if the new >> overloaded function can be safely used except the users need to indicate >> (via configs, for example) that she's assured the broker version is newer. >> I think the ultimate solution for you would be KIP-584, in which client can >> dynamically ask the broker cluster which version(s) they would support and >> decides which function to trigger accordingly. >> >> >> Guozhang >> >> >> On Sat, Apr 4, 2020 at 2:46 PM Boyang Chen <reluctanthero...@gmail.com> >> wrote: >> >>> That’s a fair point Ismael. After a second thought, I feel that if Gary >> is >>> building frameworks for general purpose usage, relying on private flag >>> seems not a good idea. >>> >>> On Sat, Apr 4, 2020 at 10:01 AM Ismael Juma <ism...@juma.me.uk> wrote: >>> >>>> The internal config was meant to be internal, right? That is, no >>>> compatibility guarantees are offered? The current discussion implies we >>> are >>>> considering it a public config. >>>> >>>> Ismael >>>> >>>> On Sat, Apr 4, 2020 at 9:31 AM Boyang Chen <boy...@confluent.io> >> wrote: >>>> >>>>> For Gary's case, I think the internal config should be a sort of >> help, >>>> and >>>>> not violating our public agreement. >>>>> >>>>> On Fri, Apr 3, 2020 at 7:44 PM Matthias J. Sax <mj...@apache.org> >>> wrote: >>>>> >>>>>> I guess you would need to catch the exception and retry? >>>>>> >>>>>> It's a little unfortunate. Not sure if we could back-port the >>> internal >>>>>> producer config that we add in 2.6 for auto-downgrade to a 2.5 bug >>> fix >>>>>> release? >>>>>> >>>>>> >>>>>> -Matthias >>>>>> >>>>>> >>>>>> On 4/2/20 7:25 PM, Gary Russell wrote: >>>>>>> Thanks Mattias >>>>>>> >>>>>>>> Hence, why do you want/need to switch to the newer overload that >>>> only >>>>>>> works for 2.5+ brokers? >>>>>>> >>>>>>> So I can choose to use the producer per consumer thread Vs. the >>>>> producer >>>>>>> per group/topic/partition threading model for zombie fencing, >> based >>>> on >>>>>> the >>>>>>> broker version. >>>>>>> >>>>>>> I don't have the same luxury as kafka streams (i.e. don't use >>> streams >>>>> 2.6 >>>>>>> unless you have 2.5+ brokers). >>>>>>> >>>>>>> I add new features with each minor release (and try to use the >>> latest >>>>>>> kafka-clients as they become available). >>>>>>> >>>>>>> Users may want other new features, not related to EOS, and they >>> might >>>>>> stay >>>>>>> on old brokers. >>>>>>> >>>>>>> Other users might want to take advantage of the improved >>> performance >>>> of >>>>>> the >>>>>>> new EOS so I need to support both APIs. >>>>>>> >>>>>>> Many enterprises take forever to upgrade their brokers. I >> recently >>>> had >>>>> a >>>>>>> question of why my latest version won't work with an 0.9.x.x >> broker >>>>>> (sigh). >>>>>>> >>>>>>> Spring versioning rules don't allow me to bump kafka-clients >>> versions >>>>> in >>>>>> a >>>>>>> patch release so I am already supporting 4 active branches and I >> am >>>>>> trying >>>>>>> to avoid supporting a fifth. >>>>>>> >>>>>>> Thanks again. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Apr 2, 2020 at 8:23 PM Matthias J. Sax <mj...@apache.org >>> >>>>> wrote: >>>>>>> >>>>>>>> Gary, >>>>>>>> >>>>>>>> thanks for the question. We recently had a discussion about the >>>> exact >>>>>>>> some topic: >>>>>>>> >>>>>>>> >>>>>> >>>>> >>>> >>> >> http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E >>>>>>>> >>>>>>>> Note that the "old" `sendOffsetsToTransaction(..., String >>> groupId)` >>>> is >>>>>>>> not deprecated. Hence, why do you want/need to switch to the >> newer >>>>>>>> overload that only works for 2.5+ brokers? For many/most cases, >>> the >>>>>>>> "old" API that is compatible with older broker still does what >> you >>>>> need >>>>>>>> and there in not need to switch to the newer API. >>>>>>>> >>>>>>>> >>>>>>>> -Matthias >>>>>>>> >>>>>>>> >>>>>>>> On 4/2/20 1:54 PM, Gary Russell wrote: >>>>>>>>> Thanks, Boyang, >>>>>>>>> >>>>>>>>> I maintain a framework (Spring for Apache Kafka) that sits on >> top >>>> of >>>>>> the >>>>>>>>> clients, and I would like to be able to support all broker >>>> versions. >>>>> I >>>>>>>>> don't have control over what brokers my users are using. >>>>>>>>> >>>>>>>>> You guys have done a great job since 0.10.2.0 (I think) of >>>> supporting >>>>>>>> older >>>>>>>>> brokers from newer clients but this one's a blocker for me. >>>>>>>>> >>>>>>>>> My framework will enforce the proper semantics for EOS, >> depending >>>> on >>>>>> the >>>>>>>>> broker version, but I need to know which model to use at >> runtime. >>>>>>>>> >>>>>>>>> As I said, I can have a property that the user can set to tell >>> the >>>>>>>>> framework that the broker is >= 2.5 but it would be cleaner if >> I >>>>> could >>>>>>>> stay >>>>>>>>> away from that. >>>>>>>>> >>>>>>>>> Something like KafkaAdminClient.brokerApi() (or add the lowest >>>>>> API/broker >>>>>>>>> version to describeCluster()), would be helpful. >>>>>>>>> >>>>>>>>> Worst case, I'll add a configuration option. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen < >>>>> reluctanthero...@gmail.com >>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks for the question Gary. The reasoning for crash the new >>>>>>>>>> sendTxnOffsets API is because we don't want users to >>> unconsciously >>>>>>>> violate >>>>>>>>>> the EOS guarantee. In your case, using this API with 2.4.1 is >>> not >>>>>>>> supported >>>>>>>>>> anyway, so the upgrade path has to start from broker first to >>> 2.5, >>>>> and >>>>>>>> then >>>>>>>>>> client binaries. Is there any further concern that blocks you >>> from >>>>>>>> getting >>>>>>>>>> the broker side upgrade first before using the new API? >>>>>>>>>> >>>>>>>>>> Boyang >>>>>>>>>> >>>>>>>>>> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell < >>> gruss...@pivotal.io> >>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Is there any way to determine the broker version in the >>>>>> kafka-clients? >>>>>>>>>>> >>>>>>>>>>> I need to determine whether I can use the new >>>>>> sendOffsetsToTransaction >>>>>>>>>>> with ConsumerGroupMetadata or use the old one. >>>>>>>>>>> >>>>>>>>>>> If I use the new API with a 2.4.1 broker, I get >>>>>>>>>>> >>>>>>>>>>> UpsupportedVersionException: Attempted to write a non-default >>>>>>>>>> generationId >>>>>>>>>>> at version 2 >>>>>>>>>>> >>>>>>>>>>> Alternatively, couldn't the client simply extract the groupId >>>> from >>>>>> the >>>>>>>>>>> ConsumerGroupMetadata and use the old struct if the broker is >>> too >>>>>> old? >>>>>>>>>>> >>>>>>>>>>> I'd rather not have a user property in my framework to tell >> us >>>>> which >>>>>>>> API >>>>>>>>>> to >>>>>>>>>>> use. >>>>>>>>>>> >>>>>>>>>>> Thanks in advance. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >> >> -- >> -- Guozhang >> >
signature.asc
Description: OpenPGP digital signature
