Thanks.
I am not a Connect expert, but the new section makes sense to me.
-Matthias
On 3/4/25 3:10 AM, Kuan Po Tseng wrote:
Hi Chia-ping,
Thank you for your feedback. I've updated the KIP and also added an explanation for why
the client bridge version starts at 3.4 - "In 4.0.0, the most rec
Hi Chia-ping,
Thank you for your feedback. I've updated the KIP and also added an explanation
for why the client bridge version starts at 3.4 - "In 4.0.0, the most recent
removed deprecated APIs were from 3.3.0, so the bridge version starts from 3.4."
Best,
Kuan-Po Tseng
On 2025/03/04 10:53:48
hi Kuan-Po
Q1:
in the "Kafka Connect:" section, the word "clients" should be replaced by
"connect", right?
Q2:
Additionally, could you add a brief description explaining why the bridge
version used by Connect is "3.8.x - 3.9.x"? This conflicts with the "API
compatibility" section.
Best,
Chia-Ping
Hello everyone,
Thank you for the discussion. As we’ve previously mentioned, the deprecation
rule would be better addressed in a separate KIP for greater visibility,
allowing other developers to participate in that discussion. I’d like to keep
KIP-1124 focused solely on the upgrade path without
Dear all,
Thank you for the discussion. Apologies for introducing an unrelated topic.
Here's a summary of that discussion.
1. A new KIP or thread will be created to define a formal deprecation policy.
This policy will apply to releases following 4.0, as 4.0 does not fully conform
to it.
2. We
What Chia-Ping says.
To me, if we remove it in 4.0, we did not really keep it for 1 year if
deprecated in 3.7, but it's subject to debate. At least for KS, we
always kept stuff of the last 3 releases.
I agree, that KIP-1124 should focus on clients/streams, and we want to
keep the code as-is
> So that's 3 releases (3.7, 3.8 and 3.9) and over 1 year, no?
KIP-1124 highlights "we keep deprecated APIs for at least 3 prior
versions," but the Connect API change does not follow this rule. It is
valid if the deprecation happens in 3.6.
Best,
Chia-Ping
Mickael Maison 於 2025年3月4日 週二 上午12:40寫
Hi,
For the Connect REST API change, the deprecation is in 3.7.0 which
released in February 2024. So that's 3 releases (3.7, 3.8 and 3.9) and
over 1 year, no?
Mickael
On Mon, Mar 3, 2025 at 5:31 PM Chia-Ping Tsai wrote:
>
> hi all,
>
> > I am also happy to follow Ismael's proposal and say "at l
hi all,
> I am also happy to follow Ismael's proposal and say "at least 3 releases
_and_ a minimum of 12 months".
+1 to this proposal
> Another example is
https://github.com/apache/kafka/commit/a753172ad3e0927f412fb56e468c95a9a81ba3ad
We deprecated our log4j1 appender in 3.8.0 and it's been remo
Thanks Mickeal,
I guess the question is, if we think we need to revert these removals,
or if it's more reasonable to make an exception from the rule?
I cannot really judge it, as I am not familiar with the details for
Connect. Any suggestions from your side?
-Matthias
On 3/3/25 7:44 AM, M
From
https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
We break compatibility (i.e. remove deprecated public methods after a
reasonable period, and typically wait 1 year after deprecation).
To me, given that we do 3 releases per year, "1 year" as stated above
and 3 r
Hi,
Another example is
https://github.com/apache/kafka/commit/a753172ad3e0927f412fb56e468c95a9a81ba3ad
We deprecated our log4j1 appender in 3.8.0 and it's been removed in
4.0.0. Kafka 3.8.0 released in May 2024, so it's less than 1 year.
Thanks,
Mickael
On Mon, Mar 3, 2025 at 4:40 PM Matthias J.
Hi Chia-Ping and Bruno,
Right. Matthias stated that the 3 releases rule is the source of truth and
I don't recall that being the case. The source of truth is 12 months - I
was one of the people who was part of that discussion when the Scala
consumer was removed. I also disagree that the 3 releases
hi Ismael
The thread[0] contains a brief discussion about the one-year rule. I've
also updated the KIP page[1] to highlight this rule. However, declaring
[3.7-3.9] as API compatible with 4.0 can be unrelated to the one-year rule.
We can do this for consistency, ensuring clients, Streams, and Conne
Hi,
I suspect that the three-release-rule was a derivation from the
1-year-rule since we usually have three releases in one year.
IMO, a three-release rule is easier to reason about, because you don't
need to know when the release took place.
However, I recognize that the 1-year-rule seems
Hmm, where is this 3 release rule documented? I don't recall it being
agreed as a project wide rule before. Do you have a reference?
Ismael
On Sat, Mar 1, 2025, 2:32 PM Matthias J. Sax wrote:
> Don't have a strong opinion about the Connect case.
>
> I guess the question is: why was it removed?
hi Matthias,
I apologize, I used the wrong word in my previous response. The PR
"deliberately" removed the public API, but it was an "oversight" that we
weren't aware of the "3 prior releases" rule.[0]
To simplify the upgrade path, I suggest reverting this change in 4.x and
creating a ticket t
> For this case, what was the reason that the 3-prior-release rule was
ignored?
According to KIP-970 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-970%3A+Deprecate+and+remove+Connect%27s+redundant+task+configurations+endpoint),
it seems the main reason to avoid "a bit of a maintenance bu
deliberately.
For this case, what was the reason that the 3-prior-release rule was
ignored?
-Matthias
On 3/1/25 11:29 PM, Chia-Ping Tsai wrote:
hi
If this Connect case is the only one that deviates from that rule, I would
recommended reverting it. But if there are many more cases, we ma
hi
> If this Connect case is the only one that deviates from that rule, I would
recommended reverting it. But if there are many more cases, we may need to
look into them in more detail.
I briefly checked the commit history, and this is the only one that needs
to be highlighted. Fortunately, it is
Don't have a strong opinion about the Connect case.
I guess the question is: why was it removed? Oversight (ie, accidentally
too early) or deliberately? Also, what is the impact if we keep the API
and revert the removal?
IMHO, the 3-releases-rule can have exceptions, if there are good reason
Hi Chia-Ping,
Is this Connect removal the only one that did not meet the stated
requirement?
We do indeed aim not to remove APIs deprecated within the last 12 months
before the next major release, but I'm not sure how strictly this is
followed (since it's not enforced).
If this Connect case is t
One more thing, 3.7 was released in Feb 2024 and hence it will be more than
12 months by the time 4.0 is released. Technically, it would be ok to
remove APIs deprecated in 3.7.
Ismael
On Sat, Mar 1, 2025, 8:34 AM Ismael Juma wrote:
> Hi Chia-Ping,
>
> Is this Connect removal the only one that d
hi Matthias
thanks for your updates. it looks great and readable.
Regarding API compatibility, we state "we only provide API backward
compatibility for version 3.7, 3.8, and 3.9" However, within Connect APIs, we
have removed a public API that was deprecated by 3.7 [0][1].
Since this KIP will
Thanks for pointing me to test Ismael. Glad to see I just missed it, and
that we indeed have a test for it.
Feel free to edit the KIP directly – I'm totally fine with it. Thanks a lot!
Thanks Kuan-Po!
I did update the wiki page and added more details. I did follow Bruno's
suggestion, to be
Hi Matthias,
Feel free to edit the KIP directly – I'm totally fine with it. Thanks a lot!
Best,
Kuan-Po Tseng
On 2025/02/28 07:35:10 "Matthias J. Sax" wrote:
> Thanks all. I am happy that we are making progress to close this out.
>
> As mentioned in my (long) email last Friday, I also think tha
Hi Matthias,
Two quick comments:
1. Yes, please help edit the KIP directly. It's easy enough to revert if
Kuan disagrees.
2. Regarding client upgrades, the test in
https://github.com/apache/kafka/pull/18193/files is an example of test that
only starts from 2.1 (since we did not merge #18193).
Is
Thanks all. I am happy that we are making progress to close this out.
As mentioned in my (long) email last Friday, I also think that the KIP
makes a lot of implicit assumptions, and it might benefit from begin
more explicit, even explaining things that are "set in stone" already,
and the KIP d
Hi Bruno,
A comment below.
On Thu, Feb 27, 2025, 1:20 AM Bruno Cadonna wrote:
> BC2:
> Another aspect that confuses me in the KIP is the bridge version for
> clients. If my broker version is 3.9.x, I can do a direct upgrade from
> 1.0.x to 4.0.x (let apart the Java API compatibility mentioned a
Hi,
Thanks for all the updates and all the discussions regarding this KIP!
IMO, the KIP does not precisely enough explain the rationale behind the
proposed upgrade path. That makes the KIP hard to understand. I believe
the KIP should be a bit more explicit about the reasons. That does not
mea
The difference I see for EOS as the following:
We have a different cut-off version. Instead of 2.3 (or earlier) like we
have for eager/cooperative, also 2.4 and 2.5 require the bridge release
upgrade to 3.9, to transit the app from EOSv1 to EOSv2.
A 2.5 (or earlier) apps with EOSv1 enabled, c
I personally think we should just recommend 3.9 as the sole bridge release,
and not include 2.8. Major versions are about API compatibility, there
shouldn't be any inherent risk in upgrading more than one major release at
a time. Things are already complicated enough :)
I do agree that we should j
Hi Matthias,
I appreciate your feedback; it really helped me a lot!
Regarding the issue of upgrading streams from a very old version to 4.x,
I understand that streams are much more complicated than Kafka Client.
I think it's reasonable to do two bumps, but I'm not a Kafka Streams expert,
and I wou
Thanks all. Seems we are converging. :)
Again, sorry to the previous very long email, but I thought it's
important to paint a full end-to-end picture.
I agree that we should try to keep it simple to the extend reasonable
possible! If we really want to suggest just 2.8 / 3.9 as bridge
version
Hello everyone,
Thanks Chia-Ping for the advice. I’ve created a table to cover all upgrade path
scenarios, hoping it provides more clarity. Please let me know if I’ve
misunderstood anything. I appreciate any corrections!
Additionally, as I recently updated the KIP title, here’s the new link:
ht
hi Kuan-Po
> Apologies for my mistake... Indeed, 2.1 should be the starting point for the
> bridge version.
Have you updated the KIP? it seems the bridge version of client still starts
from 2.0
Additionally, if I were a user hesitant to adopt the bridge version, it would
be helpful to list co
Thanks, Jun and Juma,
Apologies for my mistake... Indeed, 2.1 should be the starting point for the
bridge version.
I will revise my statement as follows:
- For Kafka Clients below 2.1, users need to upgrade to 3.9 first, then to 4.x.
- For Kafka Clients from 2.1 or above, users can directly upgr
For regular clients (producer, consumer, admin client), I think we should
describe this in the following way:
1. What's the minimum version to directly upgrade to 4.0? I suggest we go
with 2.1 - it's the same baseline set by KIP-896 and it's simpler to have
fewer version ranges to explain and worr
whew, long response from Matthias :P Lot to digest but I want to add
on/respond to a few points:
If they want to be "advantageous", they could make it a two step upgrade
> I guess, and go from 2.5 (or older) directly to 3.x and apply all
> required code changes in a single upgrade step, and repea
Hello,
took me some time, and sorry for the long email, but it's complicated...
First, I just re-read the latest version of the KIP. Thanks for all the
updates.
One thing that I an missing in the motivation is, that we really want to
stop support direct upgrades from older versions, to cut
Hi, Kuan,
Thanks for the KIP. A couple of comments.
jr1. Since connect is another client, it would be useful to include the
compatibility for connect too.
jr2. It seems a bit weird to include 2.0 as the bridge release for Kafka
client since it's not compatible with the target release 4.0.
Jun
Hi Lianet,
Thank you for your feedback!
Yes, the current KIP focuses solely on the client upgrade for 4.x. I have
updated the title accordingly and also included the KS upgrade link in the KIP.
Thanks again!
Best regards,
Kuan-Po
On 2025/02/19 16:59:25 "Lianet M." wrote:
> Hello all, sorry a
Hello all, sorry a bit late, just minor comments on this one:
- Should we clarify in the title or at the beginning of the KIP that it is
proposing a client upgrade path for 4.x? The broader considerations for
upgrades discussed in this thread will be tackled separately (seems we all
agree).
- The
Hello everyone,
If there are no other opinions, I would like to start a vote tomorrow,
thank you!
Best,
Kuan Po
On Sat, Feb 8, 2025 at 1:51 AM Kuan Po Tseng wrote:
> Hi all,
>
> Based on our discussion, I added a section on choosing the appropriate
> bridge version from an API compatibility pe
Hi all,
Based on our discussion, I added a section on choosing the appropriate bridge
version from an API compatibility perspective for upgrading to Kafka 4.0. Let
me know if you have any thoughts. Thank you!
Best,
Kuan-Po
On 2025/02/07 03:34:46 Kuan Po Tseng wrote:
> Hi Chia-Ping,
>
> Sorry
Hi Chia-Ping,
Sorry for the delayed response. I’ve checked all relevant JIRAs using the
following Jira Query Language:
project = KAFKA AND status in (Resolved, Closed) AND fixVersion = 4.0.0 AND
text ~ "Remove" order by updated DESC
Based on this, I checked the JIRAs related to removing deprec
hi Kuan-Po
any update? Now that an upgrade path for bridge versions exists, we can
introduce additional "conditions" to assist users in selecting the "best"
bridge version. For example, we can provide guidance on which bridge versions
offer backward compatibility with Kafka 4.0 client or are co
> - If we support 2.0+ to 4.0 client/KS upgrade it's simpler, but of course
> brokers cannot be 4.0 yet -- but I guess this would be something natural?
> Given that the clients would be on 2.0, brokers cannot be 4.0 yet, or clients
> would have crashed already... Thus, I think I slightly prefer
Thanks Ismael. Sounds like a good idea to split the specific 4.0
question and the long term policy.
For 4.0, it's a little tricky and could be confusing for users no matter
what we do.
- If only support 2.1+ to 4.0 client/KS upgrade, it's a weird cut-off
version
- If we support 2.0+ to 4
Juma made a valid point in the Jira ticket: "we cannot change the client
upgrade guarantees outside of a major release." To avoid blocking the 4.0
release, this thread should focus on the "client upgrade path." We can create a
separate ticket to discuss the advanced compatibility policy.
Additi
To Kuan-Po
> When we're talking about two major versions, should it be [4.x, 5.x] or
[4.x, 6.x]? The latter seems to refer to three major versions, doesn't it?"
Sorry for the unclear description. My point is that 5.x can communicate
with both 4.x and 6.x.
> In the meantime, I think we should ali
Thanks for the KIP - one suggestion I have is to perhaps separate the long
term policy (which will naturally require a lot more debate) and what we
plan to do for 4.0.
Why is the long term policy more complicated? Because it is trying to
predict the future (Matthias made a comment regarding how 2.
Thanks Chia-Ping for the summary! I'll take some time to refine the KIP as you
suggested.
I do have one little question about this:
> 1) Client-Broker Protocol Compatibility: Ensure seamless interoperability
> between 5.x clients and brokers, as well as with clients and brokers in the
> [4.x,
Dear all,
> only support upgrades between 2 major version, and also
limit client-broker compatibility to two major version.
I wholeheartedly support this concept due to its inherent simplicity. However,
a primary concern arises regarding third-party client libraries implemented in
other langua
Thanks, Chia-Ping, for the comments. I’ve addressed comments chia00 and chia01
in the KIP!
Also, thanks to Sophie for reminding me that 2.4 introduced cooperative
rebalancing.
I’ve added a paragraph about Kafka Streams and provided an example. By the way,
I’ve also updated the Test Plan. We can
Thanks for the KIP!
I have to admit that all these different aspects of compatibility are
really confusing. When we talk about compatibility, we should try to be
precise about what kind of compatibility we talk about so that for users
and developers it is clear.
Basically, we have Java/Scala
Thanks for starting this. It's overdue that we define a public contract
how far back we want do guarantee for direct upgrades.
In general, a lot of information is in the docs
(https://kafka.apache.org/39/documentation/streams/upgrade-guide) but
the information is hard to extract. And it's not
Thanks for the KIP. I'm all for documenting the upgrade path like this.
Just want to chime in on the Streams side of things, since there is (at
least) one nuance which is probably worth accounting for:
Because of the introduction of cooperative rebalancing in 2.4 which was
enabled by default for K
hi Kuan-Po
thanks for raising this discussion. Some questions are listed below.
chia00:
in the `Upgrade Example`, could you pleas remind that the bridge version
must be compatible to server's version?
chia01:
could you please add link of KIP-896 in the "Proposed Changes"?
Thanks,
Chia-Ping
Hi all,
I'd like to initiate a discussion thread on *KIP-1124,* which proposes a
clear upgrade path for Kafka Client, including Kafka Streams.
You can find the details here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1124%3A+Providing+a+clear+Kafka+Client+upgrade+path
I’d appreciate it
60 matches
Mail list logo
