hi Matthias, I apologize, I used the wrong word in my previous response. The PR "deliberately" removed the public API, but it was an "oversight" that we weren't aware of the "3 prior releases" rule.[0]
To simplify the upgrade path, I suggest reverting this change in 4.x and creating a ticket to track its removal in 5.0.. [0] https://github.com/apache/kafka/pull/17412#issuecomment-2693221650 Best, Chia-Ping On 2025/03/02 20:35:27 Chia-Ping Tsai wrote: > > For this case, what was the reason that the 3-prior-release rule was > ignored? > > According to KIP-970 ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP-970%3A+Deprecate+and+remove+Connect%27s+redundant+task+configurations+endpoint), > it seems the main reason to avoid "a bit of a maintenance burden to have > two different endpoints for the same purpose". > > Best, > Chia-Ping > > > Matthias J. Sax <mj...@apache.org> 於 2025年3月3日 週一 上午3:57寫道： > > > >> deliberately. > > > > For this case, what was the reason that the 3-prior-release rule was > > ignored? > > > > > > -Matthias > > > > On 3/1/25 11:29 PM, Chia-Ping Tsai wrote: > > > hi > > > > > >> If this Connect case is the only one that deviates from that rule, I > > would > > > recommended reverting it. But if there are many more cases, we may need > > to > > > look into them in more detail. > > > > > > I briefly checked the commit history, and this is the only one that needs > > > to be highlighted. Fortunately, it is easy to revert. > > > > > >> why was it removed? Oversight (ie, accidentally > > > too early) or deliberately? > > > > > > deliberately. > > > > > > I have left a common on the PR ( > > > https://github.com/apache/kafka/pull/17412#issuecomment-2692600494) > > > > > > Best, > > > Chia-Ping > > > > > > > > > Matthias J. Sax <mj...@apache.org> 於 2025年3月2日 週日 上午6:32寫道： > > > > > >> Don't have a strong opinion about the Connect case. > > >> > > >> I guess the question is: why was it removed? Oversight (ie, accidentally > > >> too early) or deliberately? Also, what is the impact if we keep the API > > >> and revert the removal? > > >> > > >> IMHO, the 3-releases-rule can have exceptions, if there are good reason > > >> for it. I just don't have enough context about the change to judge. > > >> > > >> > > >> > > >>>> One more thing, 3.7 was released in Feb 2024 and hence it will be more > > >> than > > >>>> 12 months by the time 4.0 is released. Technically, it would be ok to > > >>>> remove APIs deprecated in 3.7. > > >> > > >> Not sure if this is a strong argument. We cannot time releases totally > > >> accurately, so the 3-prior-releases rule should be the "source of truth" > > >> IMHO, that just practically translated to "about 1 year". But as I said > > >> above, I also don't think we need to be dogmatic about it. > > >> > > >> > > >> > > >> -Matthias > > >> > > >> On 3/1/25 8:37 AM, Ismael Juma wrote: > > >>> One more thing, 3.7 was released in Feb 2024 and hence it will be more > > >> than > > >>> 12 months by the time 4.0 is released. Technically, it would be ok to > > >>> remove APIs deprecated in 3.7. > > >>> > > >>> Ismael > > >>> > > >>> On Sat, Mar 1, 2025, 8:34 AM Ismael Juma <m...@ismaeljuma.com> wrote: > > >>> > > >>>> Hi Chia-Ping, > > >>>> > > >>>> Is this Connect removal the only one that did not meet the stated > > >>>> requirement? > > >>>> > > >>>> We do indeed aim not to remove APIs deprecated within the last 12 > > months > > >>>> before the next major release, but I'm not sure how strictly this is > > >>>> followed (since it's not enforced). > > >>>> > > >>>> If this Connect case is the only one that deviates from that rule, I > > >> would > > >>>> recommended reverting it. But if there are many more cases, we may > > need > > >> to > > >>>> look into them in more detail. > > >>>> > > >>>> Ismael > > >>>> > > >>>> On Fri, Feb 28, 2025, 7:10 PM Chia-Ping Tsai <chia7...@apache.org> > > >> wrote: > > >>>> > > >>>>> hi Matthias > > >>>>> > > >>>>> thanks for your updates. it looks great and readable. > > >>>>> > > >>>>> Regarding API compatibility, we state "we only provide API backward > > >>>>> compatibility for version 3.7, 3.8, and 3.9" However, within Connect > > >> APIs, > > >>>>> we have removed a public API that was deprecated by 3.7 [0][1]. > > >>>>> > > >>>>> Since this KIP will likely include Connect APIs, it's important to > > >>>>> discuss this now. We could: 1) add extra descriptions to highlight > > the > > >>>>> difference for Connect APIs, or 2) revert the commit that removed the > > >>>>> public API for consistent documentation. > > >>>>> > > >>>>> WDYT? > > >>>>> > > >>>>> [0] > > >>>>> > > >> > > https://github.com/apache/kafka/commit/1c8bb61a43d3ad1fd7a10eb3947342ceba783c4e > > >>>>> [1] > > >>>>> > > >> > > https://github.com/apache/kafka/commit/4f1688742e75a147741fbdc307c5e85d2addb811 > > >>>>> > > >>>>> Best, > > >>>>> Chia-Ping > > >>>>> > > >>>>> On 2025/02/28 22:44:22 "Matthias J. Sax" wrote: > > >>>>>> Thanks for pointing me to test Ismael. Glad to see I just missed it, > > >>>>> and > > >>>>>> that we indeed have a test for it. > > >>>>>> > > >>>>>> > > >>>>>>>> Feel free to edit the KIP directly – I'm totally fine with it. > > >>>>> Thanks a lot! > > >>>>>> > > >>>>>> Thanks Kuan-Po! > > >>>>>> > > >>>>>> I did update the wiki page and added more details. I did follow > > >> Bruno's > > >>>>>> suggestion, to be a little bit more elaborate, as the KIP page is > > >>>>> mainly > > >>>>>> for Kafka maintainers. -- We might want to do it differently in user > > >>>>>> facing docs, but detailed doc updates seems to be out-of-scope for > > the > > >>>>>> KIP discussion and we can take it on the corresponding PR, to > > simplify > > >>>>>> it further. > > >>>>>> > > >>>>>> > > >>>>>> I did keep the 2.8.x bridge release for KS for now.; we can still > > >>>>> change > > >>>>>> the recommendation and skip 2.8.x as bridge release of course, if we > > >>>>>> think it would be better for users. > > >>>>>> > > >>>>>> > > >>>>>> Curious the get your feedback. > > >>>>>> > > >>>>>> > > >>>>>> -Matthias > > >>>>>> > > >>>>>> > > >>>>>> On 2/28/25 8:15 AM, Kuan Po Tseng wrote: > > >>>>>>> Hi Matthias, > > >>>>>>> > > >>>>>>> Feel free to edit the KIP directly – I'm totally fine with it. > > Thanks > > >>>>> a lot! > > >>>>>>> > > >>>>>>> Best, > > >>>>>>> Kuan-Po Tseng > > >>>>>>> > > >>>>>>> On 2025/02/28 07:35:10 "Matthias J. Sax" wrote: > > >>>>>>>> Thanks all. I am happy that we are making progress to close this > > >> out. > > >>>>>>>> > > >>>>>>>> As mentioned in my (long) email last Friday, I also think that the > > >>>>> KIP > > >>>>>>>> makes a lot of implicit assumptions, and it might benefit from > > begin > > >>>>>>>> more explicit, even explaining things that are "set in stone" > > >>>>> already, > > >>>>>>>> and the KIP does not change them (like API deprecation timelines > > and > > >>>>>>>> guarantees). But as Bruno said, it would help a lot to make it > > >>>>> easier to > > >>>>>>>> understand the KIP. > > >>>>>>>> > > >>>>>>>> Only a very small number of people seems to fully understand all > > the > > >>>>>>>> nuances, and it took quite a while to collect and curate all the > > >>>>> cases, > > >>>>>>>> to shape this KIP... And we know that a lot of people actually > > read > > >>>>>>>> KIPs, and the title for KIP-1124 is or sure click-bait... > > >>>>>>>> > > >>>>>>>> I am also happy to edit the wiki page directly if that's ok with > > >>>>> Kuan, > > >>>>>>>> to add background information that I believe is missing and > > >>>>> important. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> My overall understanding of the state of the KIP, ie, its actual > > >>>>> content > > >>>>>>>> seems to be, that there is a high level agreement, and the content > > >> of > > >>>>>>>> the KIP is ok. > > >>>>>>>> > > >>>>>>>> The only open question seems to be, if we want to recommend > > `2.8.x` > > >>>>> as a > > >>>>>>>> bridge release for Kafka Streams or not. As said in my email on > > >>>>> Friday, > > >>>>>>>> I am personally ok either way, with a preference to add `2.8.x` as > > >>>>>>>> bridge release, as it simplifies updating the code by avoiding > > >>>>>>>> compilation errors. With a 2.8.x bridge release, KS application > > code > > >>>>>>>> would _always_ compile, and the JavaDocs help a lot to guide users > > >>>>> how > > >>>>>>>> to migrate off deprecated APIs. -- With a direct upgrade from > > older > > >>>>>>>> versions to 3.9.x, code does not compile, and there is no guidance > > >> as > > >>>>>>>> the old methods (and corresponding JavaDocs) are already removed, > > >>>>> making > > >>>>>>>> it significant harder for user to update their code and make it > > >>>>> compile > > >>>>>>>> again. > > >>>>>>>> > > >>>>>>>> In the end, we would only _recommend_ 2.8.x as a bridge release to > > >>>>>>>> users, and there won't be any claim that an direct upgrade to > > 3.9.x > > >>>>> is > > >>>>>>>> not supported. If one wants to do this, it's tested, and ok. Knock > > >>>>>>>> yourself out -- there is for sure apps out there, that are just a > > >>>>> simple > > >>>>>>>> `builder.stream(...).filter(...).to(...)` the there won't be an > > API > > >>>>>>>> incompatibility issues. > > >>>>>>>> > > >>>>>>>> But I don't think it would be wise to _recommend_ a direct upgrade > > >> to > > >>>>>>>> 3.9.x for the reasons stated, as many apps are more complex. > > >>>>>>>> > > >>>>>>>> Thoughts? > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> About testing: I did look into the system tests, and it seems we > > >>>>>>>> actually don't really have a real upgrade test for clients, ie, a > > >>>>> test > > >>>>>>>> that is running an consumer or producer with some older versions, > > >> and > > >>>>>>>> executes a rolling bounce to upgrade the app... > > >>>>>>>> > > >>>>>>>> But maybe such a test is not necessary for clients (compare to > > Kafka > > >>>>>>>> Streams), and the existing client-broker forward/backward > > >>>>> compatibility > > >>>>>>>> test are good enough? (At least in the last 10 years, it seems > > this > > >>>>> test > > >>>>>>>> coverage was sufficient?) > > >>>>>>>> > > >>>>>>>> I want to point out though that I agree with Bruno: If we assume > > >> that > > >>>>>>>> the client-broker compatibility tests are sufficient to verify > > >> client > > >>>>>>>> upgrades, we do effectively/implicitly test upgrading 2.0 (and > > older > > >>>>>>>> clients) to 4.0 using 3.9 and older brokers. These test coverage > > >> does > > >>>>>>>> not exist in a single branch so, but test from different branches > > >>>>>>>> combined, implicitly cover this. Nevertheless, is it tested. > > >>>>>>>> > > >>>>>>>> So it seems both Bruno and Ismael are right, depending how you > > look > > >>>>> at > > >>>>>>>> the problem. In any case, even if we test such an upgrade > > >>>>> implicitly, I > > >>>>>>>> agree that it's good to exclude these upgrades officially as > > >>>>> unsupported. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> -Matthias > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On 2/27/25 6:10 AM, Ismael Juma wrote: > > >>>>>>>>> Hi Bruno, > > >>>>>>>>> > > >>>>>>>>> A comment below. > > >>>>>>>>> > > >>>>>>>>> On Thu, Feb 27, 2025, 1:20 AM Bruno Cadonna <cado...@apache.org> > > >>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> BC2: > > >>>>>>>>>> Another aspect that confuses me in the KIP is the bridge version > > >>>>> for > > >>>>>>>>>> clients. If my broker version is 3.9.x, I can do a direct > > upgrade > > >>>>> from > > >>>>>>>>>> 1.0.x to 4.0.x (let apart the Java API compatibility mentioned > > >>>>> above). > > >>>>>>>>>> The 4.0.x release tests 4.0.x clients against 3.9.x brokers and > > >> the > > >>>>>>>>>> 3.9.x release tests 1.0.x clients against 3.9.x brokers. It is > > not > > >>>>> clear > > >>>>>>>>>> to me why we need a bridge release in this case. > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> We don't test client upgrades from 1.0.x to 4.0.x. They may work, > > >>>>> but it's > > >>>>>>>>> untested. Again, we are focusing on simplicity versus flexibility > > >>>>> for edge > > >>>>>>>>> cases. The number of people upgrading from 1.0 to 4.0 is very low > > >>>>> (if not > > >>>>>>>>> zero). > > >>>>>>>>> > > >>>>>>>>> That's why I still think the simple description I provided in my > > >>>>> other > > >>>>>>>>> message is the way to go > > >>>>>>>>> > > >>>>>>>>> Deprecated APIs may or may not affect users, so that has to be > > >>>>> evaluated on > > >>>>>>>>> a case by case basis. > > >>>>>>>>> > > >>>>>>>>> Ismael > > >>>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > >> > > > > > > > >
