Hello everyone, Thanks Chia-Ping for the advice. I’ve created a table to cover all upgrade path scenarios, hoping it provides more clarity. Please let me know if I’ve misunderstood anything. I appreciate any corrections!
Additionally, as I recently updated the KIP title, here’s the new link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1124%3A+Providing+a+clear+Kafka+Client+upgrade+path+for+4.x Regarding Kafka Connect, I’m still investigating and will update the KIP soon. I’ll share any new findings with you as soon as possible. Thank you! Best, Kuan-Po On 2025/02/23 19:12:41 Chia-Ping Tsai wrote: > hi Kuan-Po > > > Apologies for my mistake... Indeed, 2.1 should be the starting point for > > the bridge version. > > Have you updated the KIP? it seems the bridge version of client still starts > from 2.0 > > Additionally, if I were a user hesitant to adopt the bridge version, it would > be helpful to list common reasons to aid in choosing the "best" bridge > version. For example: > > Client Upgrade Paths > > // here is the table > > Best Bridge Version to you > > // add some explanation > > 1. minimize code refactoring - // Kafka Client: 3.3.x - 3.9.x + Kafka > Streams: 3.6.x - 3.9.x > 2. starts with quorum APIs - // 3.3.x - 3.9.x > 3. xxx > 4. aaa > n. last stable/active version: 3.9.x // we can emphasize the 3.9 is > recommended by community > > Best, > Chia-Ping > > > > On 2025/02/23 16:10:03 Kuan Po Tseng wrote: > > Thanks, Jun and Juma, > > > > Apologies for my mistake... Indeed, 2.1 should be the starting point for > > the bridge version. > > I will revise my statement as follows: > > - For Kafka Clients below 2.1, users need to upgrade to 3.9 first, then to > > 4.x. > > - For Kafka Clients from 2.1 or above, users can directly upgrade to 4.x. > > > > As for Kafka Connect, I initially didn’t consider it because I saw it as > > another form of a server. > > However, I’m not very familiar with this area and will need some time to > > look into it. > > > > --- > > > > Thanks, Matthias and Sophie, for the detailed explanation. > > > > The Streams upgrade is indeed complex... I took some time to digest these > > details. > > It sounds like the key concerns for the Streams upgrade are: > > 1. Users still using EOSv1 > > 2. Users relying on Eager rebalancing (i.e., versions before 2.4) > > > > With this in mind, should we adjust the recommended upgrade path to the > > following? > > - For Kafka Streams below 2.4, you’ll need to first upgrade to 2.8, then to > > 3.9, and finally to 4.x. > > - For Kafka Streams using EOSv1 on versions 2.4 or above, you’ll need to > > upgrade to 3.9 first, then to 4.x. > > - Others can directly upgrade to 4.x. > > *Note*: If upgrading Streams from 3.4 or below, you’ll need to perform two > > rolling bounces. > > For more details, please refer to: [Kafka Streams Upgrade > > Guide](https://kafka.apache.org/39/documentation/streams/upgrade-guide). > > Also, Kafka Streams requires broker compatibility considerations, see: > > [Streams API Broker > > Compatibility](https://kafka.apache.org/39/documentation/streams/upgrade-guide#streams_api_broker_compat) > > > > The above approach simplifies the definition of the bridge version. > > Instead of providing a range (e.g., [3.5-3.9]), would it be better to > > specify 3.9 directly to > > reduce users' decision anxiety? This aligns with Juma’s recommendation for > > bridge versions > > in the Kafka Clients upgrade. 3.9 is the last version before 4.0, > > containing the most bug fixes and offering greater stability. > > For detailed upgrade steps, such as two-rolling bounce upgrades, > > like Matthias recommended we should direct users to the Streams upgrade > > documentation. > > And I feel the examples in my KIP are too simplistic, > > but I don’t think I should make them as detailed as the Streams upgrade > > guide. > > Otherwise, I’d just be duplicating content. The goal of this KIP should be > > to provide > > users with a clear high-level upgrade path, while the detailed steps should > > be > > referenced from the Streams upgrade documentation. > > > > Regarding ALOS, it looks like we support versions from 0.11.x to 4.x, > > so there’s no need to specify additional details, right? Or am I missing > > something? > > > > Lastly, based on Matthias' suggestion, I have revised the motivation > > section to emphasize: > > - Simplifying testing > > - Reducing compatibility challenges across too many versions > > - Clearly defining the recommended upgrade paths > > > > Best, > > Kuan-Po Tseng > > > > On 2025/02/21 23:55:51 Sophie Blee-Goldman wrote: > > > whew, long response from Matthias :P Lot to digest but I want to add > > > on/respond to a few points: > > > > > > If they want to be "advantageous", they could make it a two step upgrade > > > > I guess, and go from 2.5 (or older) directly to 3.x and apply all > > > > required code changes in a single upgrade step, and repeat the same to > > > > upgrade to 4.0. But I would not necessarily recommend to do an non-API > > > > compatible upgrade directly, and for sure officially discourage it for > > > > two major releases. > > > > > > Are we still talking about only API compatibility here? Because I'm not so > > > sure why we would > > > officially discourage upgrading across 2 major releases as long as their > > > code is compatible. Of > > > course if you're referring to possible gotchas from upgrading over such a > > > long period, that's worth > > > discussing, but it's independent of API compatibility. Imo API > > > compatibility is a binary thing: either > > > it's possible to do a direct upgrade or it's not. Why do we have to > > > officially recommend anything? > > > > > > Or we can distinguish > > > > between ALOS and EOS and have an "bridge release version range" for both > > > > cases. > > > > > > I like this idea. EOS and ALOS are very distinct in Streams and may only > > > become moreso divided > > > over time. It's worth calling them out as separate cases > > > > > > Now regarding the eager/cooperative rebalancing protocol thing in Streams: > > > > > > As Matthias said we hope to officially drop support for eager rebalancing > > > in 4.0, and I've prepared > > > a PR for this already: https://github.com/apache/kafka/pull/18988 > > > > > > This does have the effect of forcing a bridge release for users hoping to > > > upgrade directly from 2.3 > > > or below to 4.0+, and users will have to follow a specific upgrade path to > > > do so as outlined in the > > > PR description. Assuming we fit that into 4.0, it should definitely be > > > called out in this KIP. (Basically > > > users need to use the `upgrade.from` config to first upgrade to the bridge > > > release, then go on to 4.0) > > > > > > There are also other runtime incompatibilities that have been introduced > > > into Streams over the years > > > that restrict direct live upgrades across certain versions. It would also > > > be good to call this out in > > > the KIP and point to the `upgrade.from` config, though we can point to the > > > Streams upgrade guide > > > for details rather than try to reiterate everything here. > > > > > > > > > On Thu, Feb 20, 2025 at 5:21 PM Matthias J. Sax <mj...@apache.org> wrote: > > > > > > > Hello, > > > > > > > > took me some time, and sorry for the long email, but it's complicated... > > > > > > > > First, I just re-read the latest version of the KIP. Thanks for all the > > > > updates. > > > > > > > > > > > > One thing that I an missing in the motivation is, that we really want to > > > > stop support direct upgrades from older versions, to cut down our > > > > upgrade matrix for testing. The motivation does somewhat touch on it, > > > > but I think it would be good to be more explicit. Even if it isn't > > > > something users care about, it's a second main motivation for us, in > > > > addition to the complexity to actually keep the versions compatible. > > > > > > > > > > > > I also want to further clarify my understanding of the KIP. The goal is > > > > not to define what upgrades are possible, right? What is possible is > > > > much more nuanced. -- But we rather want to define what we recommend? Is > > > > this understanding correct? If yes, it might also be worth to add to the > > > > motivation section. > > > > > > > > > > > > I also think, we actually need to more explicitly distinguish three > > > > categories of compatibility, but did so far only discuss two of them. > > > > Even if the KIP does mention all three. Ideally, we should have a > > > > section in the motivation, explaining the three different types of > > > > compatibility, and explicitly state which one this KIP is concerned > > > > with, and which ones it's not concerned with. > > > > > > > > > > > > (1) protocol compatibility: ie, what client-broker versions are > > > > compatible > > > > > > > > This one is not in the focus of the KIP, but it might still be good to > > > > be explicit about it. Could be explained in the motivation for > > > > completeness, and maybe refer to KIP-896 for 4.x related changes. > > > > > > > > Btw: there is also some additional limitations for KS-broker > > > > compatibility: > > > > > > > > https://kafka.apache.org/39/documentation/streams/upgrade-guide#streams_api_broker_compat > > > > > > > > Many of you know this, but wanted to mention it for completeness. Not > > > > sure if we need to mention it on the KIP. > > > > > > > > > > > > > > > > (2) API compatibility (ie Java/Scala API). > > > > > > > > This is only mentioned briefly in the KIP, and again, it's not the core > > > > of the KIP, but I think it is still important to include it more > > > > explicitly, because we talk about "bridge version". > > > > > > > > Given the rule that we are allowed to break API compatibility in major > > > > release, but still guarantee API compatibility for the last three minor > > > > releases, it can be confusing and it would be great to explain it > > > > better. > > > > > > > > In the end, directly upgrading from 2.5 or older to 4.x is practically > > > > impossible as we went through two major releases which did remove > > > > deprecated APIs, and I would not recommend to do such a direct upgrade. > > > > > > > > From an API POV, if one is on 2.5 or older, they should first upgrade > > > > to 2.6/2.7/2.8, and than lazily migrate off any older stuff what is > > > > removed with 3.0. Afterwards, they can upgrade to 3.7/3.8/3.9 following > > > > the same pattern, and only upgrade to 4.0 in a third step. > > > > > > > > If they want to be "advantageous", they could make it a two step upgrade > > > > I guess, and go from 2.5 (or older) directly to 3.x and apply all > > > > required code changes in a single upgrade step, and repeat the same to > > > > upgrade to 4.0. But I would not necessarily recommend to do an non-API > > > > compatible upgrade directly, and for sure officially discourage it for > > > > two major releases. > > > > > > > > Thus, the information in the KIP about "bridge version" to 4.x begin > > > > "2.4.x - 3.9.x" seems to fall short, and mentioning > > > > > > > > > To minimize code refactoring, we recommend the following bridge > > > > > versions > > > > that maintain API compatibility with Kafka 4.x: > > > > > > > > > > Kafka Client: 3.3.x - 3.9.x > > > > > Kafka Streams: 3.6.x - 3.9.x > > > > > > > > seems not to be sufficient to me. > > > > > > > > > > > > Hence, the provided "Upgrade Examples" might be oversimplified, and we > > > > might want to refine them. > > > > > > > > > > > > > > > > (3) Runtime compatibility. This one is specific to Kafka Streams, but > > > > not to clients from my understanding. Client are stateless and thus they > > > > don't face any issue, but Kafka Streams is stateful, and thus need to > > > > take care of it. Please correct me if I am wrong. > > > > > > > > The KIP so far, seems to only consider this one, and what is proposed > > > > make sense to me on a high level. However, I am confused why Kafka > > > > Clients are mentioned here, too, as this type of compatibility should > > > > not really be relevant for them? Even if clients might also have some > > > > semantic changes, these should always align with API changes (ie, old > > > > deprecate API might have slightly different semantics than new API). > > > > > > > > Now about the currently proposes ranges from Kafka Streams: > > > > > > > > > Kafka Streams > > > > > Current Version 0.11.x - 2.3.x > > > > > Bridge Release 2.4.x - 3.9.x > > > > > Target Version 4.x > > > > > > > > This could make sense for "eager vs cooperative" rebalancing, however, > > > > at the current point, we did not remove "eager" in 4.0 yet. I was > > > > actually just syncing up with Sophie about it, and it was a slip, and we > > > > want to propose to remove "eager" in 4.0 (Sophie will prepare a PR), so > > > > we can avoid keeping "eager" until 5.0. > > > > > > > > We did officially deprecate "eager" in 3.1 release, so we are covered to > > > > actually remove it with 4.0. > > > > > > > > If we would not drop "eager", using `2.4.x to 3.9.x` would not make > > > > sense though. If we keep "eager" in 4.0, user can still upgrade from > > > > 2.0.x to 4.0.x w/o issues from a runtime perspective. > > > > > > > > If we drop "eager" we also need to drop the corresponding system tests > > > > that upgrade to 4.0, and also stop testing upgrading from "eager to > > > > cooperative" with 3.9 being the highest target version in this system > > > > test. And if we don't test it, it's not officially supported any > > > > longer... (even if people could still upgrade via an offline upgrade -- > > > > what really breaks if we remove "eager" is "only" the online > > > > [two-]rolling bounce upgrade...) > > > > > > > > However, there is another change we want to consider: we did remove > > > > EOSv1 in 4.0 release, which was replace with EOSv2 in Kafka Streams 2.6 > > > > via KIP-447. > > > > > > > > Thus, for EOSv1 users, they cannot directly upgrade to 4.0 either, but > > > > only EOSv2 users can. Thus, it might make sense to actually use "bridge > > > > releases 2.6.x - 3.9.x" just to keep it simple... Or we can distinguish > > > > between ALOS and EOS and have an "bridge release version range" for both > > > > cases. > > > > > > > > Btw: using EOSv2 required broker version 2.5+, that we might also want > > > > to call out. > > > > > > > > > > > > > > > > Last but not least, while we are very explicit in the KS upgrade docs, > > > > it might be worth to call out that some upgrades require a two-rolling > > > > bounce approach, and users should always consult the upgrade docs... We > > > > use two-rolling bounce upgrade to bridge runtime backward incompatible > > > > changes (similar to what we do broker side, when IBP version is bumped). > > > > > > > > > > > > > > > > > > > > So overall, it seems that we need to really have two guidelines, not > > > > just one? For for API compatibility, which is much stricter, and one for > > > > runtime compatibility? > > > > > > > > If we really want to make a recommendation that is most easy to > > > > understand, we might want to only go with API compatibility. Not sure if > > > > this might be "too restrictive" though? > > > > > > > > > > > > Curious to get you though on all this. > > > > > > > > > > > > -Matthias > > > > > > > > > > > > > > > > > > > > On 2/19/25 5:51 PM, Kuan Po Tseng wrote: > > > > > Hi Lianet, > > > > > > > > > > Thank you for your feedback! > > > > > > > > > > Yes, the current KIP focuses solely on the client upgrade for 4.x. I > > > > have updated the title accordingly and also included the KS upgrade > > > > link in > > > > the KIP. > > > > > > > > > > Thanks again! > > > > > > > > > > Best regards, > > > > > Kuan-Po > > > > > > > > > > On 2025/02/19 16:59:25 "Lianet M." wrote: > > > > >> Hello all, sorry a bit late, just minor comments on this one: > > > > >> > > > > >> - Should we clarify in the title or at the beginning of the KIP that > > > > >> it > > > > is > > > > >> proposing a client upgrade path for 4.x? The broader considerations > > > > >> for > > > > >> upgrades discussed in this thread will be tackled separately (seems > > > > >> we > > > > all > > > > >> agree). > > > > >> > > > > >> - The KS upgrade path seems to be the tricky one, and all that the > > > > >> user > > > > >> needs to consider to successfully follow the provided path for KS is > > > > >> not > > > > >> clear in the KIP, but it's all well explained on the KS upgrade notes > > > > for > > > > >> 3.9, should we add a ref to that? > > > > >> https://kafka.apache.org/39/documentation/streams/upgrade-guide > > > > >> > > > > >> Thanks Kuan Po! > > > > >> > > > > >> Lianet > > > > >> > > > > >> On Tue, Feb 11, 2025 at 11:22 AM Kuan Po Tseng <brandb...@gmail.com> > > > > wrote: > > > > >> > > > > >>> Hello everyone, > > > > >>> > > > > >>> If there are no other opinions, I would like to start a vote > > > > >>> tomorrow, > > > > >>> thank you! > > > > >>> > > > > >>> Best, > > > > >>> Kuan Po > > > > >>> > > > > >>> On Sat, Feb 8, 2025 at 1:51 AM Kuan Po Tseng <brandb...@apache.org> > > > > wrote: > > > > >>> > > > > >>>> Hi all, > > > > >>>> > > > > >>>> Based on our discussion, I added a section on choosing the > > > > >>>> appropriate > > > > >>>> bridge version from an API compatibility perspective for upgrading > > > > >>>> to > > > > >>> Kafka > > > > >>>> 4.0. Let me know if you have any thoughts. Thank you! > > > > >>>> > > > > >>>> Best, > > > > >>>> Kuan-Po > > > > >>>> > > > > >>>> On 2025/02/07 03:34:46 Kuan Po Tseng wrote: > > > > >>>>> Hi Chia-Ping, > > > > >>>>> > > > > >>>>> Sorry for the delayed response. I’ve checked all relevant JIRAs > > > > >>>>> using > > > > >>>> the following Jira Query Language: > > > > >>>>> > > > > >>>>> project = KAFKA AND status in (Resolved, Closed) AND fixVersion = > > > > 4.0.0 > > > > >>>> AND text ~ "Remove" order by updated DESC > > > > >>>>> > > > > >>>>> Based on this, I checked the JIRAs related to removing deprecated > > > > >>>> methods in client modules. The minimum backward-compatible client > > > > >>> versions > > > > >>>> for client 4.0 are as follows: > > > > >>>>> - Producer: 3.3.0 > > > > >>>>> Reason: Partitioner#onNewBatch was deprecated in 3.3.0, and was > > > > >>>> removed by https://issues.apache.org/jira/browse/KAFKA-18295 > > > > >>>>> - Consumer: 2.4.0 > > > > >>>>> Reason: Consumer#committed was deprecated in 2.4.0, and was > > > > removed > > > > >>> by > > > > >>>> https://issues.apache.org/jira/browse/KAFKA-17451 > > > > >>>>> - Admin: 3.3.0 > > > > >>>>> Reason: ListConsumerGroupOffsetsOptions was deprecated in 3.3.0 > > > > and > > > > >>>> was removed by https://issues.apache.org/jira/browse/KAFKA-18291 > > > > >>>>> > > > > >>>>> You can find a list of all related JIRAs and pull requests in this > > > > >>>> Google Sheet: > > > > >>>>> > > > > >>>> > > > > >>> > > > > https://docs.google.com/spreadsheets/d/1ZWNRk1rjWptjpGM2UtT0Q3lDULhrqkP_UfHr9roQW3M/edit?usp=sharing > > > > >>>>> > > > > >>>>> There are also some public methods removed in 4.0, such as: > > > > >>>>> - KafkaFuture#Function, KafkaFuture#thenApply > > > > >>>> https://issues.apache.org/jira/browse/KAFKA-17903 > > > > >>>>> - JmxReported(String) > > > > >>> https://issues.apache.org/jira/browse/KAFKA-18077 > > > > >>>>> , but I'm uncertain about how we should handle these. > > > > >>>>> > > > > >>>>> Best, > > > > >>>>> Kuan-Po > > > > >>>>> > > > > >>>>> On 2025/02/06 19:08:49 Chia-Ping Tsai wrote: > > > > >>>>>> hi Kuan-Po > > > > >>>>>> > > > > >>>>>> any update? Now that an upgrade path for bridge versions exists, > > > > >>>>>> we > > > > >>>> can introduce additional "conditions" to assist users in selecting > > > > >>>> the > > > > >>>> "best" bridge version. For example, we can provide guidance on > > > > >>>> which > > > > >>> bridge > > > > >>>> versions offer backward compatibility with Kafka 4.0 client or are > > > > >>>> compatible with Kafka 4.0 server. > > > > >>>>>> > > > > >>>>>> Best, > > > > >>>>>> Chia-Ping > > > > >>>>>> > > > > >>>>>> On 2025/01/22 04:48:36 Chia-Ping Tsai wrote: > > > > >>>>>>>> - If we support 2.0+ to 4.0 client/KS upgrade it's simpler, but > > > > >>> of > > > > >>>> course brokers cannot be 4.0 yet -- but I guess this would be > > > > something > > > > >>>> natural? Given that the clients would be on 2.0, brokers cannot be > > > > >>>> 4.0 > > > > >>> yet, > > > > >>>> or clients would have crashed already... Thus, I think I slightly > > > > prefer > > > > >>>> this one. > > > > >>>>>>> > > > > >>>>>>> Using a major version as a bridge is a viable approach. We can > > > > >>>> emphasize the limitations of this method to guide users in > > > > >>>> selecting > > > > the > > > > >>>> most suitable bridge version. > > > > >>>>>>> > > > > >>>>>>>> For KS, from an API compatibility POV, upgrading from anything > > > > >>>> older than 3.6 might not work any longer (for DSL users; of course, > > > > >>>> depending on what APIs they are using). And for PAPI, the old API > > > > >>>> was > > > > >>>> removed too, so only if the new one is use (introduced in 2.7) a > > > > seamless > > > > >>>> upgrade would work smoothly. > > > > >>>>>>> > > > > >>>>>>> You make a valid point. The previous discussion overlooked the > > > > >>>>>>> APIs > > > > >>>> that were removed in version 4.0. > > > > >>>>>>> > > > > >>>>>>> We could also emphasize the BC advantages. As an example, users > > > > >>> have > > > > >>>> the option of using version 2.7 as a bridge and subsequently > > > > >>>> upgrade > > > > >>>> without code alterations or recompilation. Of course, we need to > > > > >>>> check > > > > >>> the > > > > >>>> version of other PAPI removal. > > > > >>>>>>> > > > > >>>>>>> Best, > > > > >>>>>>> Chia-Ping > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>>> Matthias J. Sax <mj...@apache.org> 於 2025年1月22日 凌晨2:55 寫道： > > > > >>>>>>>> For KS, from an API compatibility POV, upgrading from anything > > > > >>>> older than 3.6 might not work any longer (for DSL users; of course, > > > > >>>> depending on what APIs they are using). And for PAPI, the old API > > > > >>>> was > > > > >>>> removed too, so only if the new one is use (introduced in 2.7) a > > > > seamless > > > > >>>> upgrade would work smoothly. > > > > >>>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > >
