hi Kuan-Po > Apologies for my mistake... Indeed, 2.1 should be the starting point for the > bridge version.
Have you updated the KIP? it seems the bridge version of client still starts from 2.0 Additionally, if I were a user hesitant to adopt the bridge version, it would be helpful to list common reasons to aid in choosing the "best" bridge version. For example: Client Upgrade Paths // here is the table Best Bridge Version to you // add some explanation 1. minimize code refactoring - // Kafka Client: 3.3.x - 3.9.x + Kafka Streams: 3.6.x - 3.9.x 2. starts with quorum APIs - // 3.3.x - 3.9.x 3. xxx 4. aaa n. last stable/active version: 3.9.x // we can emphasize the 3.9 is recommended by community Best, Chia-Ping On 2025/02/23 16:10:03 Kuan Po Tseng wrote: > Thanks, Jun and Juma, > > Apologies for my mistake... Indeed, 2.1 should be the starting point for the > bridge version. > I will revise my statement as follows: > - For Kafka Clients below 2.1, users need to upgrade to 3.9 first, then to > 4.x. > - For Kafka Clients from 2.1 or above, users can directly upgrade to 4.x. > > As for Kafka Connect, I initially didn’t consider it because I saw it as > another form of a server. > However, I’m not very familiar with this area and will need some time to look > into it. > > --- > > Thanks, Matthias and Sophie, for the detailed explanation. > > The Streams upgrade is indeed complex... I took some time to digest these > details. > It sounds like the key concerns for the Streams upgrade are: > 1. Users still using EOSv1 > 2. Users relying on Eager rebalancing (i.e., versions before 2.4) > > With this in mind, should we adjust the recommended upgrade path to the > following? > - For Kafka Streams below 2.4, you’ll need to first upgrade to 2.8, then to > 3.9, and finally to 4.x. > - For Kafka Streams using EOSv1 on versions 2.4 or above, you’ll need to > upgrade to 3.9 first, then to 4.x. > - Others can directly upgrade to 4.x. > *Note*: If upgrading Streams from 3.4 or below, you’ll need to perform two > rolling bounces. > For more details, please refer to: [Kafka Streams Upgrade > Guide](https://kafka.apache.org/39/documentation/streams/upgrade-guide). > Also, Kafka Streams requires broker compatibility considerations, see: > [Streams API Broker > Compatibility](https://kafka.apache.org/39/documentation/streams/upgrade-guide#streams_api_broker_compat) > > The above approach simplifies the definition of the bridge version. > Instead of providing a range (e.g., [3.5-3.9]), would it be better to specify > 3.9 directly to > reduce users' decision anxiety? This aligns with Juma’s recommendation for > bridge versions > in the Kafka Clients upgrade. 3.9 is the last version before 4.0, > containing the most bug fixes and offering greater stability. > For detailed upgrade steps, such as two-rolling bounce upgrades, > like Matthias recommended we should direct users to the Streams upgrade > documentation. > And I feel the examples in my KIP are too simplistic, > but I don’t think I should make them as detailed as the Streams upgrade guide. > Otherwise, I’d just be duplicating content. The goal of this KIP should be to > provide > users with a clear high-level upgrade path, while the detailed steps should be > referenced from the Streams upgrade documentation. > > Regarding ALOS, it looks like we support versions from 0.11.x to 4.x, > so there’s no need to specify additional details, right? Or am I missing > something? > > Lastly, based on Matthias' suggestion, I have revised the motivation section > to emphasize: > - Simplifying testing > - Reducing compatibility challenges across too many versions > - Clearly defining the recommended upgrade paths > > Best, > Kuan-Po Tseng > > On 2025/02/21 23:55:51 Sophie Blee-Goldman wrote: > > whew, long response from Matthias :P Lot to digest but I want to add > > on/respond to a few points: > > > > If they want to be "advantageous", they could make it a two step upgrade > > > I guess, and go from 2.5 (or older) directly to 3.x and apply all > > > required code changes in a single upgrade step, and repeat the same to > > > upgrade to 4.0. But I would not necessarily recommend to do an non-API > > > compatible upgrade directly, and for sure officially discourage it for > > > two major releases. > > > > Are we still talking about only API compatibility here? Because I'm not so > > sure why we would > > officially discourage upgrading across 2 major releases as long as their > > code is compatible. Of > > course if you're referring to possible gotchas from upgrading over such a > > long period, that's worth > > discussing, but it's independent of API compatibility. Imo API > > compatibility is a binary thing: either > > it's possible to do a direct upgrade or it's not. Why do we have to > > officially recommend anything? > > > > Or we can distinguish > > > between ALOS and EOS and have an "bridge release version range" for both > > > cases. > > > > I like this idea. EOS and ALOS are very distinct in Streams and may only > > become moreso divided > > over time. It's worth calling them out as separate cases > > > > Now regarding the eager/cooperative rebalancing protocol thing in Streams: > > > > As Matthias said we hope to officially drop support for eager rebalancing > > in 4.0, and I've prepared > > a PR for this already: https://github.com/apache/kafka/pull/18988 > > > > This does have the effect of forcing a bridge release for users hoping to > > upgrade directly from 2.3 > > or below to 4.0+, and users will have to follow a specific upgrade path to > > do so as outlined in the > > PR description. Assuming we fit that into 4.0, it should definitely be > > called out in this KIP. (Basically > > users need to use the `upgrade.from` config to first upgrade to the bridge > > release, then go on to 4.0) > > > > There are also other runtime incompatibilities that have been introduced > > into Streams over the years > > that restrict direct live upgrades across certain versions. It would also > > be good to call this out in > > the KIP and point to the `upgrade.from` config, though we can point to the > > Streams upgrade guide > > for details rather than try to reiterate everything here. > > > > > > On Thu, Feb 20, 2025 at 5:21 PM Matthias J. Sax <mj...@apache.org> wrote: > > > > > Hello, > > > > > > took me some time, and sorry for the long email, but it's complicated... > > > > > > First, I just re-read the latest version of the KIP. Thanks for all the > > > updates. > > > > > > > > > One thing that I an missing in the motivation is, that we really want to > > > stop support direct upgrades from older versions, to cut down our > > > upgrade matrix for testing. The motivation does somewhat touch on it, > > > but I think it would be good to be more explicit. Even if it isn't > > > something users care about, it's a second main motivation for us, in > > > addition to the complexity to actually keep the versions compatible. > > > > > > > > > I also want to further clarify my understanding of the KIP. The goal is > > > not to define what upgrades are possible, right? What is possible is > > > much more nuanced. -- But we rather want to define what we recommend? Is > > > this understanding correct? If yes, it might also be worth to add to the > > > motivation section. > > > > > > > > > I also think, we actually need to more explicitly distinguish three > > > categories of compatibility, but did so far only discuss two of them. > > > Even if the KIP does mention all three. Ideally, we should have a > > > section in the motivation, explaining the three different types of > > > compatibility, and explicitly state which one this KIP is concerned > > > with, and which ones it's not concerned with. > > > > > > > > > (1) protocol compatibility: ie, what client-broker versions are compatible > > > > > > This one is not in the focus of the KIP, but it might still be good to > > > be explicit about it. Could be explained in the motivation for > > > completeness, and maybe refer to KIP-896 for 4.x related changes. > > > > > > Btw: there is also some additional limitations for KS-broker > > > compatibility: > > > > > > https://kafka.apache.org/39/documentation/streams/upgrade-guide#streams_api_broker_compat > > > > > > Many of you know this, but wanted to mention it for completeness. Not > > > sure if we need to mention it on the KIP. > > > > > > > > > > > > (2) API compatibility (ie Java/Scala API). > > > > > > This is only mentioned briefly in the KIP, and again, it's not the core > > > of the KIP, but I think it is still important to include it more > > > explicitly, because we talk about "bridge version". > > > > > > Given the rule that we are allowed to break API compatibility in major > > > release, but still guarantee API compatibility for the last three minor > > > releases, it can be confusing and it would be great to explain it better. > > > > > > In the end, directly upgrading from 2.5 or older to 4.x is practically > > > impossible as we went through two major releases which did remove > > > deprecated APIs, and I would not recommend to do such a direct upgrade. > > > > > > From an API POV, if one is on 2.5 or older, they should first upgrade > > > to 2.6/2.7/2.8, and than lazily migrate off any older stuff what is > > > removed with 3.0. Afterwards, they can upgrade to 3.7/3.8/3.9 following > > > the same pattern, and only upgrade to 4.0 in a third step. > > > > > > If they want to be "advantageous", they could make it a two step upgrade > > > I guess, and go from 2.5 (or older) directly to 3.x and apply all > > > required code changes in a single upgrade step, and repeat the same to > > > upgrade to 4.0. But I would not necessarily recommend to do an non-API > > > compatible upgrade directly, and for sure officially discourage it for > > > two major releases. > > > > > > Thus, the information in the KIP about "bridge version" to 4.x begin > > > "2.4.x - 3.9.x" seems to fall short, and mentioning > > > > > > > To minimize code refactoring, we recommend the following bridge versions > > > that maintain API compatibility with Kafka 4.x: > > > > > > > > Kafka Client: 3.3.x - 3.9.x > > > > Kafka Streams: 3.6.x - 3.9.x > > > > > > seems not to be sufficient to me. > > > > > > > > > Hence, the provided "Upgrade Examples" might be oversimplified, and we > > > might want to refine them. > > > > > > > > > > > > (3) Runtime compatibility. This one is specific to Kafka Streams, but > > > not to clients from my understanding. Client are stateless and thus they > > > don't face any issue, but Kafka Streams is stateful, and thus need to > > > take care of it. Please correct me if I am wrong. > > > > > > The KIP so far, seems to only consider this one, and what is proposed > > > make sense to me on a high level. However, I am confused why Kafka > > > Clients are mentioned here, too, as this type of compatibility should > > > not really be relevant for them? Even if clients might also have some > > > semantic changes, these should always align with API changes (ie, old > > > deprecate API might have slightly different semantics than new API). > > > > > > Now about the currently proposes ranges from Kafka Streams: > > > > > > > Kafka Streams > > > > Current Version 0.11.x - 2.3.x > > > > Bridge Release 2.4.x - 3.9.x > > > > Target Version 4.x > > > > > > This could make sense for "eager vs cooperative" rebalancing, however, > > > at the current point, we did not remove "eager" in 4.0 yet. I was > > > actually just syncing up with Sophie about it, and it was a slip, and we > > > want to propose to remove "eager" in 4.0 (Sophie will prepare a PR), so > > > we can avoid keeping "eager" until 5.0. > > > > > > We did officially deprecate "eager" in 3.1 release, so we are covered to > > > actually remove it with 4.0. > > > > > > If we would not drop "eager", using `2.4.x to 3.9.x` would not make > > > sense though. If we keep "eager" in 4.0, user can still upgrade from > > > 2.0.x to 4.0.x w/o issues from a runtime perspective. > > > > > > If we drop "eager" we also need to drop the corresponding system tests > > > that upgrade to 4.0, and also stop testing upgrading from "eager to > > > cooperative" with 3.9 being the highest target version in this system > > > test. And if we don't test it, it's not officially supported any > > > longer... (even if people could still upgrade via an offline upgrade -- > > > what really breaks if we remove "eager" is "only" the online > > > [two-]rolling bounce upgrade...) > > > > > > However, there is another change we want to consider: we did remove > > > EOSv1 in 4.0 release, which was replace with EOSv2 in Kafka Streams 2.6 > > > via KIP-447. > > > > > > Thus, for EOSv1 users, they cannot directly upgrade to 4.0 either, but > > > only EOSv2 users can. Thus, it might make sense to actually use "bridge > > > releases 2.6.x - 3.9.x" just to keep it simple... Or we can distinguish > > > between ALOS and EOS and have an "bridge release version range" for both > > > cases. > > > > > > Btw: using EOSv2 required broker version 2.5+, that we might also want > > > to call out. > > > > > > > > > > > > Last but not least, while we are very explicit in the KS upgrade docs, > > > it might be worth to call out that some upgrades require a two-rolling > > > bounce approach, and users should always consult the upgrade docs... We > > > use two-rolling bounce upgrade to bridge runtime backward incompatible > > > changes (similar to what we do broker side, when IBP version is bumped). > > > > > > > > > > > > > > > So overall, it seems that we need to really have two guidelines, not > > > just one? For for API compatibility, which is much stricter, and one for > > > runtime compatibility? > > > > > > If we really want to make a recommendation that is most easy to > > > understand, we might want to only go with API compatibility. Not sure if > > > this might be "too restrictive" though? > > > > > > > > > Curious to get you though on all this. > > > > > > > > > -Matthias > > > > > > > > > > > > > > > On 2/19/25 5:51 PM, Kuan Po Tseng wrote: > > > > Hi Lianet, > > > > > > > > Thank you for your feedback! > > > > > > > > Yes, the current KIP focuses solely on the client upgrade for 4.x. I > > > have updated the title accordingly and also included the KS upgrade link > > > in > > > the KIP. > > > > > > > > Thanks again! > > > > > > > > Best regards, > > > > Kuan-Po > > > > > > > > On 2025/02/19 16:59:25 "Lianet M." wrote: > > > >> Hello all, sorry a bit late, just minor comments on this one: > > > >> > > > >> - Should we clarify in the title or at the beginning of the KIP that it > > > is > > > >> proposing a client upgrade path for 4.x? The broader considerations for > > > >> upgrades discussed in this thread will be tackled separately (seems we > > > all > > > >> agree). > > > >> > > > >> - The KS upgrade path seems to be the tricky one, and all that the user > > > >> needs to consider to successfully follow the provided path for KS is > > > >> not > > > >> clear in the KIP, but it's all well explained on the KS upgrade notes > > > for > > > >> 3.9, should we add a ref to that? > > > >> https://kafka.apache.org/39/documentation/streams/upgrade-guide > > > >> > > > >> Thanks Kuan Po! > > > >> > > > >> Lianet > > > >> > > > >> On Tue, Feb 11, 2025 at 11:22 AM Kuan Po Tseng <brandb...@gmail.com> > > > wrote: > > > >> > > > >>> Hello everyone, > > > >>> > > > >>> If there are no other opinions, I would like to start a vote tomorrow, > > > >>> thank you! > > > >>> > > > >>> Best, > > > >>> Kuan Po > > > >>> > > > >>> On Sat, Feb 8, 2025 at 1:51 AM Kuan Po Tseng <brandb...@apache.org> > > > wrote: > > > >>> > > > >>>> Hi all, > > > >>>> > > > >>>> Based on our discussion, I added a section on choosing the > > > >>>> appropriate > > > >>>> bridge version from an API compatibility perspective for upgrading to > > > >>> Kafka > > > >>>> 4.0. Let me know if you have any thoughts. Thank you! > > > >>>> > > > >>>> Best, > > > >>>> Kuan-Po > > > >>>> > > > >>>> On 2025/02/07 03:34:46 Kuan Po Tseng wrote: > > > >>>>> Hi Chia-Ping, > > > >>>>> > > > >>>>> Sorry for the delayed response. I’ve checked all relevant JIRAs > > > >>>>> using > > > >>>> the following Jira Query Language: > > > >>>>> > > > >>>>> project = KAFKA AND status in (Resolved, Closed) AND fixVersion = > > > 4.0.0 > > > >>>> AND text ~ "Remove" order by updated DESC > > > >>>>> > > > >>>>> Based on this, I checked the JIRAs related to removing deprecated > > > >>>> methods in client modules. The minimum backward-compatible client > > > >>> versions > > > >>>> for client 4.0 are as follows: > > > >>>>> - Producer: 3.3.0 > > > >>>>> Reason: Partitioner#onNewBatch was deprecated in 3.3.0, and was > > > >>>> removed by https://issues.apache.org/jira/browse/KAFKA-18295 > > > >>>>> - Consumer: 2.4.0 > > > >>>>> Reason: Consumer#committed was deprecated in 2.4.0, and was > > > removed > > > >>> by > > > >>>> https://issues.apache.org/jira/browse/KAFKA-17451 > > > >>>>> - Admin: 3.3.0 > > > >>>>> Reason: ListConsumerGroupOffsetsOptions was deprecated in 3.3.0 > > > and > > > >>>> was removed by https://issues.apache.org/jira/browse/KAFKA-18291 > > > >>>>> > > > >>>>> You can find a list of all related JIRAs and pull requests in this > > > >>>> Google Sheet: > > > >>>>> > > > >>>> > > > >>> > > > https://docs.google.com/spreadsheets/d/1ZWNRk1rjWptjpGM2UtT0Q3lDULhrqkP_UfHr9roQW3M/edit?usp=sharing > > > >>>>> > > > >>>>> There are also some public methods removed in 4.0, such as: > > > >>>>> - KafkaFuture#Function, KafkaFuture#thenApply > > > >>>> https://issues.apache.org/jira/browse/KAFKA-17903 > > > >>>>> - JmxReported(String) > > > >>> https://issues.apache.org/jira/browse/KAFKA-18077 > > > >>>>> , but I'm uncertain about how we should handle these. > > > >>>>> > > > >>>>> Best, > > > >>>>> Kuan-Po > > > >>>>> > > > >>>>> On 2025/02/06 19:08:49 Chia-Ping Tsai wrote: > > > >>>>>> hi Kuan-Po > > > >>>>>> > > > >>>>>> any update? Now that an upgrade path for bridge versions exists, we > > > >>>> can introduce additional "conditions" to assist users in selecting > > > >>>> the > > > >>>> "best" bridge version. For example, we can provide guidance on which > > > >>> bridge > > > >>>> versions offer backward compatibility with Kafka 4.0 client or are > > > >>>> compatible with Kafka 4.0 server. > > > >>>>>> > > > >>>>>> Best, > > > >>>>>> Chia-Ping > > > >>>>>> > > > >>>>>> On 2025/01/22 04:48:36 Chia-Ping Tsai wrote: > > > >>>>>>>> - If we support 2.0+ to 4.0 client/KS upgrade it's simpler, but > > > >>> of > > > >>>> course brokers cannot be 4.0 yet -- but I guess this would be > > > something > > > >>>> natural? Given that the clients would be on 2.0, brokers cannot be > > > >>>> 4.0 > > > >>> yet, > > > >>>> or clients would have crashed already... Thus, I think I slightly > > > prefer > > > >>>> this one. > > > >>>>>>> > > > >>>>>>> Using a major version as a bridge is a viable approach. We can > > > >>>> emphasize the limitations of this method to guide users in selecting > > > the > > > >>>> most suitable bridge version. > > > >>>>>>> > > > >>>>>>>> For KS, from an API compatibility POV, upgrading from anything > > > >>>> older than 3.6 might not work any longer (for DSL users; of course, > > > >>>> depending on what APIs they are using). And for PAPI, the old API was > > > >>>> removed too, so only if the new one is use (introduced in 2.7) a > > > seamless > > > >>>> upgrade would work smoothly. > > > >>>>>>> > > > >>>>>>> You make a valid point. The previous discussion overlooked the > > > >>>>>>> APIs > > > >>>> that were removed in version 4.0. > > > >>>>>>> > > > >>>>>>> We could also emphasize the BC advantages. As an example, users > > > >>> have > > > >>>> the option of using version 2.7 as a bridge and subsequently upgrade > > > >>>> without code alterations or recompilation. Of course, we need to > > > >>>> check > > > >>> the > > > >>>> version of other PAPI removal. > > > >>>>>>> > > > >>>>>>> Best, > > > >>>>>>> Chia-Ping > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>> Matthias J. Sax <mj...@apache.org> 於 2025年1月22日 凌晨2:55 寫道： > > > >>>>>>>> For KS, from an API compatibility POV, upgrading from anything > > > >>>> older than 3.6 might not work any longer (for DSL users; of course, > > > >>>> depending on what APIs they are using). And for PAPI, the old API was > > > >>>> removed too, so only if the new one is use (introduced in 2.7) a > > > seamless > > > >>>> upgrade would work smoothly. > > > >>>>>>> > > > >>>>>> > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > > > >
