Thanks Chia-Ping for the summary! I'll take some time to refine the KIP as you suggested.
I do have one little question about this: > 1) Client-Broker Protocol Compatibility: Ensure seamless interoperability > between 5.x clients and brokers, as well as with clients and brokers in the > [4.x, 6.x] range. When we're talking about two major versions, should it be [4.x, 5.x] or [4.x, 6.x]? The latter seems to refer to three major versions, doesn't it?" On 2025/01/19 10:15:11 Chia-Ping Tsai wrote: > Dear all, > > > only support upgrades between 2 major version, and also > limit client-broker compatibility to two major version. > > I wholeheartedly support this concept due to its inherent simplicity. > However, a primary concern arises regarding third-party client libraries > implemented in other languages. These libraries serve as the primary > reference for KIP-896, as I understand it. > > Furthermore, I propose expanding the definition of 'compatibility' to > encompass the following aspects: > > 1) Client-Broker Protocol Compatibility: Ensure seamless interoperability > between 5.x clients and brokers, as well as with clients and brokers in the > [4.x, 6.x] range. > > 2) Upgrade Path: Guarantee a smooth upgrade path from 4.x clients and brokers > to the 5.x version. > > The following points require further consideration: > > 1) Removal of Old Protocols: Should we remove protocols prior to 4.0 in the > 5.0 release? This action could streamline the codebase, but we should > carefully weigh the potential benefits against the need to accommodate users > running Kafka with older releases (even if there is no official guarantee) > > 2) Hybrid Broker-Broker Scenarios: How will the system behave in hybrid > environments where 4.x, 5.x, and 6.x brokers coexist? Specifically, what > happens when a 6.x broker is introduced into a 5.x cluster that includes 4.x > brokers? If we intend to remove older protocols in major releases, such > hybrid configurations might need to be disallowed. > > 3) Downgrade Path: Supporting downgrades from 5.x clients and brokers to 4.x > versions may pose challenges, particularly when considering the introduction > of new records. > > To Kuan-Po > > I recommend reorganizing KIP-1124 to include a dedicated section for > discussing the proposed new compatibility policy. Given that we anticipate a > series of discussions on this topic within this thread, incorporating a > summary of the current discussion within the KIP itself would be highly > beneficial. > > Best, > Chia-Ping > > > On 2025/01/16 11:31:15 Bruno Cadonna wrote: > > Thanks for the KIP! > > > > I have to admit that all these different aspects of compatibility are > > really confusing. When we talk about compatibility, we should try to be > > precise about what kind of compatibility we talk about so that for users > > and developers it is clear. > > > > Basically, we have Java/Scala API compatibility and protocol API (i.e. > > client and inter-broker RPC) compatibility. > > > > Java/Scala APIs: > > > > "Any incompatible changes should be ultimately settled in the KIP design > > process, where the usual strategy is to retain the old APIs, mark them > > as deprecated and potentially remove them in some next major release."[1] > > > > At the moment, as far as I understand, this KIP proposes to document > > what KIP-896 proposes and what was accordingly implemented. I do not > > believe, we need a KIP for that. The intent of this KIP should be to > > agree on a general policy when to remove old client protocol versions > > (as Matthias pointed out). I found the following in [1]: > > > > "Our policy is that the Kafka protocols and data formats should support > > backwards compatibility for as many releases to enable no-downtime > > upgrades (unless there is a very compelling counter-argument). This > > means the server MUST be able to support requests from both old and new > > clients simultaneously. This compatibility needs to be retained for at > > least one major release (e.g. 2.0 must accept requests from 1.0 > > clients). A typical upgrade sequence for binary format changes would be > > (1) upgrade server to handle the new message format, (2) upgrade clients > > to use the new message format." > > > > It says "at least one major release (e.g. 2.0 must accept requests from > > 1.0 clients)". I understand that Matthias wants to change "at least" to > > "exactly". I guess, there might always be the option to write a KIP to > > retain compatibility with older versions of certain protocols. > > > > So the question seems to be whether > > > > 1. we want to make the removal of old protocol versions automatically > > (or at least not guarantee the existence) and write KIPs for retaining > > certain old versions of protocols > > > > OR > > > > 2. we want to write KIPs to remove old protocol version (e.g. KIP-896). > > > > IMO, from a code maintenance perspective the former is preferable, from > > a user perspective maybe the latter. However, I am not sure it is really > > preferable for users of clients to be able to postpone upgrades for too > > long. > > > > So, I am in favor of 1. > > > > Best, > > Bruno > > > > > > [1] https://kafka.apache.org/coding-guide > > > > On 16.01.25 00:31, Matthias J. Sax wrote: > > > Thanks for starting this. It's overdue that we define a public contract > > > how far back we want do guarantee for direct upgrades. > > > > > > In general, a lot of information is in the docs (https:// > > > kafka.apache.org/39/documentation/streams/upgrade-guide) but the > > > information is hard to extract. And it's not documented "how far back" > > > on could actually go. > > > > > > About Sophie's point: while important, it seems to be orthogonal to the > > > question on hand? Also, the requirement of a two-round rolling bounce > > > upgrade is not unique to the rebalance protocol. > > > > > > While it's great to define this for already release versions, to me it's > > > actually more important to look into the future, to set expectations > > > right. (For the past, it's not really possible to just "drop" guarantees > > > suddenly...) > > > > > > Historically, we did support backward compatibility across multiple > > > major versions (as long as no APIs are use which might have been > > > removed). However, it think we should cut this down to two major > > > versions going forward. Ie, we won't guarantee that a 3.x app can be > > > upgraded to 5.x, but only 4.x can be upgrade to 5.x. > > > > > > It's really painful to maintain a longer compatibility window, and > > > testing it is also very expensive. > > > > > > Furthermore, and this is an even broader question, we guarantee client- > > > broker backward/forward compatibility across multiple major version, > > > too. I think we should also limit client-broker compatibility to two > > > major versions only. For 4.0 brokers, we already drop support for 2.0 > > > and older clients (and reverse, 4.0 client require 2.1+ brokers), what > > > is still a larger window. > > > > > > But I think for 5.0 brokers, we should only guarantee support for 4.0+ > > > clients and drop 2.x and 3.x... -- And in revers, if upgrading client to > > > 5.x, they would require broker 4.0, and we don't guarantee that 5.x > > > clients work against 3.x/2.x brokers any longer. > > > > > > Given that we did 3 years of release with 2.x and 3.x each, I would > > > expect that 5.x will also be like 3 years in the future, and 3 years is > > > already a very long compatibility guarantee, and with two major version, > > > 6 year old clients can still connect to the latest broker and vice-versa. > > > > > > Of course, if it would really happen that we bump the major version > > > quicker (like we did for 1.0/1.1 to 2.0), we can always add an exception > > > to the rule and document it accordingly. But IMHO, the general rule > > > should be, to only support upgrades between 2 major version, and also > > > limit client-broker compatibility to two major version. > > > > > > Having a public contract like this, should also avoid that we need to do > > > KIP to drop older RPC from the implementation when a new major release > > > comes along streamlining the development process. > > > > > > > > > Thoughts? > > > > > > > > > -Matthias > > > > > > > > > On 1/15/25 3:00 PM, Sophie Blee-Goldman wrote: > > >> Thanks for the KIP. I'm all for documenting the upgrade path like this. > > >> Just want to chime in on the Streams side of things, since there is (at > > >> least) one nuance which is probably worth accounting for: > > >> > > >> Because of the introduction of cooperative rebalancing in 2.4 which was > > >> enabled by default for Kafka Streams, it is not possible to directly > > >> upgrade a Streams app from pre-2.4 to 2.4 or higher. Instead the user > > >> must > > >> follow a specific two-phase upgrade path, with the first rolling bounce > > >> setting the UPGRADE_FROM config and upgraded the bytecode, and the second > > >> rolling bounce simply removing the UPGRADE_FROM config. > > >> > > >> I mention this here because this means if a user needs to upgrade a > > >> Streams > > >> app from, say, version 1.0 to version 4.0 with 2.3 as a bridge release, > > >> they would have to go through 3 total upgrades instead of just 2. For > > >> this > > >> reason I'm wondering if we should recommend a slightly different upgrade > > >> path for Kafka Streams specifically: *[0.11.x - 2.3.x] → [2.4.x - > > >> 3.9.x] → > > >> 4.x* > > >> > > >> We should also call out the special upgrade path as part of the > > >> documentation here, that is, for all Kafka Streams users: > > >> > > >> 1. Set the UPGRADE_FROM config to the starting version and perform the > > >> *[0.11.x > > >> - 2.3.x] → [2.4.x - 3.9.x]* upgrade > > >> 2. Remove the UPGRADE_FROM config and perform the *[2.4.x - 3.9.x] → > > >> 4.x * > > >> upgrade > > >> > > >> Thoughts? > > >> > > >> On Wed, Jan 15, 2025 at 7:57 AM Chia-Ping Tsai <chia7...@gmail.com> > > >> wrote: > > >> > > >>> hi Kuan-Po > > >>> > > >>> thanks for raising this discussion. Some questions are listed below. > > >>> > > >>> chia00: > > >>> > > >>> in the `Upgrade Example`, could you pleas remind that the bridge version > > >>> must be compatible to server's version? > > >>> > > >>> chia01: > > >>> > > >>> could you please add link of KIP-896 in the "Proposed Changes"? > > >>> > > >>> Thanks, > > >>> Chia-Ping > > >>> > > >>> > > >>> Kuan Po Tseng <brandb...@gmail.com> 於 2025年1月15日 週三 下午9:46寫道： > > >>> > > >>>> Hi all, > > >>>> > > >>>> I'd like to initiate a discussion thread on *KIP-1124,* which > > >>>> proposes a > > >>>> clear upgrade path for Kafka Client, including Kafka Streams. > > >>>> You can find the details here: > > >>>> > > >>>> > > >>> https://cwiki.apache.org/confluence/display/KAFKA/ > > >>> KIP-1124%3A+Providing+a+clear+Kafka+Client+upgrade+path > > >>>> I’d appreciate it if you could take a look and share your thoughts or > > >>>> feedback. > > >>>> > > >>>> Best regards, > > >>>> Kuan-Po Tseng > > >>>> > > >>> > > >> > > > > > > > >
