That’s a fair point Ismael. After a second thought, I feel that if Gary is building frameworks for general purpose usage, relying on private flag seems not a good idea.
On Sat, Apr 4, 2020 at 10:01 AM Ismael Juma <ism...@juma.me.uk> wrote: > The internal config was meant to be internal, right? That is, no > compatibility guarantees are offered? The current discussion implies we are > considering it a public config. > > Ismael > > On Sat, Apr 4, 2020 at 9:31 AM Boyang Chen <boy...@confluent.io> wrote: > > > For Gary's case, I think the internal config should be a sort of help, > and > > not violating our public agreement. > > > > On Fri, Apr 3, 2020 at 7:44 PM Matthias J. Sax <mj...@apache.org> wrote: > > > > > I guess you would need to catch the exception and retry? > > > > > > It's a little unfortunate. Not sure if we could back-port the internal > > > producer config that we add in 2.6 for auto-downgrade to a 2.5 bug fix > > > release? > > > > > > > > > -Matthias > > > > > > > > > On 4/2/20 7:25 PM, Gary Russell wrote: > > > > Thanks Mattias > > > > > > > >> Hence, why do you want/need to switch to the newer overload that > only > > > > works for 2.5+ brokers? > > > > > > > > So I can choose to use the producer per consumer thread Vs. the > > producer > > > > per group/topic/partition threading model for zombie fencing, based > on > > > the > > > > broker version. > > > > > > > > I don't have the same luxury as kafka streams (i.e. don't use streams > > 2.6 > > > > unless you have 2.5+ brokers). > > > > > > > > I add new features with each minor release (and try to use the latest > > > > kafka-clients as they become available). > > > > > > > > Users may want other new features, not related to EOS, and they might > > > stay > > > > on old brokers. > > > > > > > > Other users might want to take advantage of the improved performance > of > > > the > > > > new EOS so I need to support both APIs. > > > > > > > > Many enterprises take forever to upgrade their brokers. I recently > had > > a > > > > question of why my latest version won't work with an 0.9.x.x broker > > > (sigh). > > > > > > > > Spring versioning rules don't allow me to bump kafka-clients versions > > in > > > a > > > > patch release so I am already supporting 4 active branches and I am > > > trying > > > > to avoid supporting a fifth. > > > > > > > > Thanks again. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 2, 2020 at 8:23 PM Matthias J. Sax <mj...@apache.org> > > wrote: > > > > > > > >> Gary, > > > >> > > > >> thanks for the question. We recently had a discussion about the > exact > > > >> some topic: > > > >> > > > >> > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E > > > >> > > > >> Note that the "old" `sendOffsetsToTransaction(..., String groupId)` > is > > > >> not deprecated. Hence, why do you want/need to switch to the newer > > > >> overload that only works for 2.5+ brokers? For many/most cases, the > > > >> "old" API that is compatible with older broker still does what you > > need > > > >> and there in not need to switch to the newer API. > > > >> > > > >> > > > >> -Matthias > > > >> > > > >> > > > >> On 4/2/20 1:54 PM, Gary Russell wrote: > > > >>> Thanks, Boyang, > > > >>> > > > >>> I maintain a framework (Spring for Apache Kafka) that sits on top > of > > > the > > > >>> clients, and I would like to be able to support all broker > versions. > > I > > > >>> don't have control over what brokers my users are using. > > > >>> > > > >>> You guys have done a great job since 0.10.2.0 (I think) of > supporting > > > >> older > > > >>> brokers from newer clients but this one's a blocker for me. > > > >>> > > > >>> My framework will enforce the proper semantics for EOS, depending > on > > > the > > > >>> broker version, but I need to know which model to use at runtime. > > > >>> > > > >>> As I said, I can have a property that the user can set to tell the > > > >>> framework that the broker is >= 2.5 but it would be cleaner if I > > could > > > >> stay > > > >>> away from that. > > > >>> > > > >>> Something like KafkaAdminClient.brokerApi() (or add the lowest > > > API/broker > > > >>> version to describeCluster()), would be helpful. > > > >>> > > > >>> Worst case, I'll add a configuration option. > > > >>> > > > >>> Thanks. > > > >>> > > > >>> On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen < > > reluctanthero...@gmail.com > > > > > > > >>> wrote: > > > >>> > > > >>>> Thanks for the question Gary. The reasoning for crash the new > > > >>>> sendTxnOffsets API is because we don't want users to unconsciously > > > >> violate > > > >>>> the EOS guarantee. In your case, using this API with 2.4.1 is not > > > >> supported > > > >>>> anyway, so the upgrade path has to start from broker first to 2.5, > > and > > > >> then > > > >>>> client binaries. Is there any further concern that blocks you from > > > >> getting > > > >>>> the broker side upgrade first before using the new API? > > > >>>> > > > >>>> Boyang > > > >>>> > > > >>>> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell <gruss...@pivotal.io> > > > >> wrote: > > > >>>> > > > >>>>> Is there any way to determine the broker version in the > > > kafka-clients? > > > >>>>> > > > >>>>> I need to determine whether I can use the new > > > sendOffsetsToTransaction > > > >>>>> with ConsumerGroupMetadata or use the old one. > > > >>>>> > > > >>>>> If I use the new API with a 2.4.1 broker, I get > > > >>>>> > > > >>>>> UpsupportedVersionException: Attempted to write a non-default > > > >>>> generationId > > > >>>>> at version 2 > > > >>>>> > > > >>>>> Alternatively, couldn't the client simply extract the groupId > from > > > the > > > >>>>> ConsumerGroupMetadata and use the old struct if the broker is too > > > old? > > > >>>>> > > > >>>>> I'd rather not have a user property in my framework to tell us > > which > > > >> API > > > >>>> to > > > >>>>> use. > > > >>>>> > > > >>>>> Thanks in advance. > > > >>>>> > > > >>>> > > > >>> > > > >> > > > >> > > > > > > > > > > > > >
