Thanks for the KIP!
I have to admit that all these different aspects of compatibility are
really confusing. When we talk about compatibility, we should try to be
precise about what kind of compatibility we talk about so that for users
and developers it is clear.
Basically, we have Java/Scala API compatibility and protocol API (i.e.
client and inter-broker RPC) compatibility.
Java/Scala APIs:
"Any incompatible changes should be ultimately settled in the KIP design
process, where the usual strategy is to retain the old APIs, mark them
as deprecated and potentially remove them in some next major release."[1]
At the moment, as far as I understand, this KIP proposes to document
what KIP-896 proposes and what was accordingly implemented. I do not
believe, we need a KIP for that. The intent of this KIP should be to
agree on a general policy when to remove old client protocol versions
(as Matthias pointed out). I found the following in [1]:
"Our policy is that the Kafka protocols and data formats should support
backwards compatibility for as many releases to enable no-downtime
upgrades (unless there is a very compelling counter-argument). This
means the server MUST be able to support requests from both old and new
clients simultaneously. This compatibility needs to be retained for at
least one major release (e.g. 2.0 must accept requests from 1.0
clients). A typical upgrade sequence for binary format changes would be
(1) upgrade server to handle the new message format, (2) upgrade clients
to use the new message format."
It says "at least one major release (e.g. 2.0 must accept requests from
1.0 clients)". I understand that Matthias wants to change "at least" to
"exactly". I guess, there might always be the option to write a KIP to
retain compatibility with older versions of certain protocols.
So the question seems to be whether
1. we want to make the removal of old protocol versions automatically
(or at least not guarantee the existence) and write KIPs for retaining
certain old versions of protocols
OR
2. we want to write KIPs to remove old protocol version (e.g. KIP-896).
IMO, from a code maintenance perspective the former is preferable, from
a user perspective maybe the latter. However, I am not sure it is really
preferable for users of clients to be able to postpone upgrades for too
long.
So, I am in favor of 1.
Best,
Bruno
[1] https://kafka.apache.org/coding-guide
On 16.01.25 00:31, Matthias J. Sax wrote:
Thanks for starting this. It's overdue that we define a public contract
how far back we want do guarantee for direct upgrades.
In general, a lot of information is in the docs (https://
kafka.apache.org/39/documentation/streams/upgrade-guide) but the
information is hard to extract. And it's not documented "how far back"
on could actually go.
About Sophie's point: while important, it seems to be orthogonal to the
question on hand? Also, the requirement of a two-round rolling bounce
upgrade is not unique to the rebalance protocol.
While it's great to define this for already release versions, to me it's
actually more important to look into the future, to set expectations
right. (For the past, it's not really possible to just "drop" guarantees
suddenly...)
Historically, we did support backward compatibility across multiple
major versions (as long as no APIs are use which might have been
removed). However, it think we should cut this down to two major
versions going forward. Ie, we won't guarantee that a 3.x app can be
upgraded to 5.x, but only 4.x can be upgrade to 5.x.
It's really painful to maintain a longer compatibility window, and
testing it is also very expensive.
Furthermore, and this is an even broader question, we guarantee client-
broker backward/forward compatibility across multiple major version,
too. I think we should also limit client-broker compatibility to two
major versions only. For 4.0 brokers, we already drop support for 2.0
and older clients (and reverse, 4.0 client require 2.1+ brokers), what
is still a larger window.
But I think for 5.0 brokers, we should only guarantee support for 4.0+
clients and drop 2.x and 3.x... -- And in revers, if upgrading client to
5.x, they would require broker 4.0, and we don't guarantee that 5.x
clients work against 3.x/2.x brokers any longer.
Given that we did 3 years of release with 2.x and 3.x each, I would
expect that 5.x will also be like 3 years in the future, and 3 years is
already a very long compatibility guarantee, and with two major version,
6 year old clients can still connect to the latest broker and vice-versa.
Of course, if it would really happen that we bump the major version
quicker (like we did for 1.0/1.1 to 2.0), we can always add an exception
to the rule and document it accordingly. But IMHO, the general rule
should be, to only support upgrades between 2 major version, and also
limit client-broker compatibility to two major version.
Having a public contract like this, should also avoid that we need to do
KIP to drop older RPC from the implementation when a new major release
comes along streamlining the development process.
Thoughts?
-Matthias
On 1/15/25 3:00 PM, Sophie Blee-Goldman wrote:
Thanks for the KIP. I'm all for documenting the upgrade path like this.
Just want to chime in on the Streams side of things, since there is (at
least) one nuance which is probably worth accounting for:
Because of the introduction of cooperative rebalancing in 2.4 which was
enabled by default for Kafka Streams, it is not possible to directly
upgrade a Streams app from pre-2.4 to 2.4 or higher. Instead the user
must
follow a specific two-phase upgrade path, with the first rolling bounce
setting the UPGRADE_FROM config and upgraded the bytecode, and the second
rolling bounce simply removing the UPGRADE_FROM config.
I mention this here because this means if a user needs to upgrade a
Streams
app from, say, version 1.0 to version 4.0 with 2.3 as a bridge release,
they would have to go through 3 total upgrades instead of just 2. For
this
reason I'm wondering if we should recommend a slightly different upgrade
path for Kafka Streams specifically: *[0.11.x - 2.3.x] → [2.4.x -
3.9.x] →
4.x*
We should also call out the special upgrade path as part of the
documentation here, that is, for all Kafka Streams users:
1. Set the UPGRADE_FROM config to the starting version and perform the
*[0.11.x
- 2.3.x] → [2.4.x - 3.9.x]* upgrade
2. Remove the UPGRADE_FROM config and perform the *[2.4.x - 3.9.x] →
4.x *
upgrade
Thoughts?
On Wed, Jan 15, 2025 at 7:57 AM Chia-Ping Tsai <chia7...@gmail.com>
wrote:
hi Kuan-Po
thanks for raising this discussion. Some questions are listed below.
chia00:
in the `Upgrade Example`, could you pleas remind that the bridge version
must be compatible to server's version?
chia01:
could you please add link of KIP-896 in the "Proposed Changes"?
Thanks,
Chia-Ping
Kuan Po Tseng <brandb...@gmail.com> 於 2025年1月15日 週三 下午9:46寫道:
Hi all,
I'd like to initiate a discussion thread on *KIP-1124,* which
proposes a
clear upgrade path for Kafka Client, including Kafka Streams.
You can find the details here:
https://cwiki.apache.org/confluence/display/KAFKA/
KIP-1124%3A+Providing+a+clear+Kafka+Client+upgrade+path
I’d appreciate it if you could take a look and share your thoughts or
feedback.
Best regards,
Kuan-Po Tseng