The internal config was meant to be internal, right? That is, no compatibility guarantees are offered? The current discussion implies we are considering it a public config.
Ismael On Sat, Apr 4, 2020 at 9:31 AM Boyang Chen <boy...@confluent.io> wrote: > For Gary's case, I think the internal config should be a sort of help, and > not violating our public agreement. > > On Fri, Apr 3, 2020 at 7:44 PM Matthias J. Sax <mj...@apache.org> wrote: > > > I guess you would need to catch the exception and retry? > > > > It's a little unfortunate. Not sure if we could back-port the internal > > producer config that we add in 2.6 for auto-downgrade to a 2.5 bug fix > > release? > > > > > > -Matthias > > > > > > On 4/2/20 7:25 PM, Gary Russell wrote: > > > Thanks Mattias > > > > > >> Hence, why do you want/need to switch to the newer overload that only > > > works for 2.5+ brokers? > > > > > > So I can choose to use the producer per consumer thread Vs. the > producer > > > per group/topic/partition threading model for zombie fencing, based on > > the > > > broker version. > > > > > > I don't have the same luxury as kafka streams (i.e. don't use streams > 2.6 > > > unless you have 2.5+ brokers). > > > > > > I add new features with each minor release (and try to use the latest > > > kafka-clients as they become available). > > > > > > Users may want other new features, not related to EOS, and they might > > stay > > > on old brokers. > > > > > > Other users might want to take advantage of the improved performance of > > the > > > new EOS so I need to support both APIs. > > > > > > Many enterprises take forever to upgrade their brokers. I recently had > a > > > question of why my latest version won't work with an 0.9.x.x broker > > (sigh). > > > > > > Spring versioning rules don't allow me to bump kafka-clients versions > in > > a > > > patch release so I am already supporting 4 active branches and I am > > trying > > > to avoid supporting a fifth. > > > > > > Thanks again. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 2, 2020 at 8:23 PM Matthias J. Sax <mj...@apache.org> > wrote: > > > > > >> Gary, > > >> > > >> thanks for the question. We recently had a discussion about the exact > > >> some topic: > > >> > > >> > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E > > >> > > >> Note that the "old" `sendOffsetsToTransaction(..., String groupId)` is > > >> not deprecated. Hence, why do you want/need to switch to the newer > > >> overload that only works for 2.5+ brokers? For many/most cases, the > > >> "old" API that is compatible with older broker still does what you > need > > >> and there in not need to switch to the newer API. > > >> > > >> > > >> -Matthias > > >> > > >> > > >> On 4/2/20 1:54 PM, Gary Russell wrote: > > >>> Thanks, Boyang, > > >>> > > >>> I maintain a framework (Spring for Apache Kafka) that sits on top of > > the > > >>> clients, and I would like to be able to support all broker versions. > I > > >>> don't have control over what brokers my users are using. > > >>> > > >>> You guys have done a great job since 0.10.2.0 (I think) of supporting > > >> older > > >>> brokers from newer clients but this one's a blocker for me. > > >>> > > >>> My framework will enforce the proper semantics for EOS, depending on > > the > > >>> broker version, but I need to know which model to use at runtime. > > >>> > > >>> As I said, I can have a property that the user can set to tell the > > >>> framework that the broker is >= 2.5 but it would be cleaner if I > could > > >> stay > > >>> away from that. > > >>> > > >>> Something like KafkaAdminClient.brokerApi() (or add the lowest > > API/broker > > >>> version to describeCluster()), would be helpful. > > >>> > > >>> Worst case, I'll add a configuration option. > > >>> > > >>> Thanks. > > >>> > > >>> On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen < > reluctanthero...@gmail.com > > > > > >>> wrote: > > >>> > > >>>> Thanks for the question Gary. The reasoning for crash the new > > >>>> sendTxnOffsets API is because we don't want users to unconsciously > > >> violate > > >>>> the EOS guarantee. In your case, using this API with 2.4.1 is not > > >> supported > > >>>> anyway, so the upgrade path has to start from broker first to 2.5, > and > > >> then > > >>>> client binaries. Is there any further concern that blocks you from > > >> getting > > >>>> the broker side upgrade first before using the new API? > > >>>> > > >>>> Boyang > > >>>> > > >>>> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell <gruss...@pivotal.io> > > >> wrote: > > >>>> > > >>>>> Is there any way to determine the broker version in the > > kafka-clients? > > >>>>> > > >>>>> I need to determine whether I can use the new > > sendOffsetsToTransaction > > >>>>> with ConsumerGroupMetadata or use the old one. > > >>>>> > > >>>>> If I use the new API with a 2.4.1 broker, I get > > >>>>> > > >>>>> UpsupportedVersionException: Attempted to write a non-default > > >>>> generationId > > >>>>> at version 2 > > >>>>> > > >>>>> Alternatively, couldn't the client simply extract the groupId from > > the > > >>>>> ConsumerGroupMetadata and use the old struct if the broker is too > > old? > > >>>>> > > >>>>> I'd rather not have a user property in my framework to tell us > which > > >> API > > >>>> to > > >>>>> use. > > >>>>> > > >>>>> Thanks in advance. > > >>>>> > > >>>> > > >>> > > >> > > >> > > > > > > > >
