Hi Josep, Thanks for the KIP! I think that the motivations for having another feature release before 4.0 are valid, as it seems that 3.7 is not suitable as the last 3.x release. I had a concern about the "Scope" section, in particular the language about "must haves", and the prevention of further 3.x releases.
As I understand it, the Kafka project uses Time based releases [1], in which the release cadence has a fixed time between releases, and the set of features in a release is flexible. This is in contrast to feature-based releases where the release cadence is flexible, but the set of features in a release is fixed. If we are describing particular KIPs as "must-haves" for 3.8, does that mean that 3.8 is not a time-based release, but a feature-based release? For example, if KIP-853 was not ready by a deadline in the release plan, what would happen to the 3.8 release? Would it be delayed until KIP-853 was ready, or would KIP-853 be postponed until a 3.9 release? Is KIP-853 sufficiently low-risk that slipping from 3.8 is impossible? I am not familiar enough with that KIP to answer that question. Looking back at the discussion on KIP-833 [2], it looks like the "last 3.x version" was interpreted to be an estimation, and the KIP itself includes "Note: this timeline is very rough and subject to change." I'll take that to mean that accepting KIP-833 did not count as explicitly voting that 3.7 would be the last release. It looks like in the past, we have had loose discussion threads about bumping to the next major version (ex 1.0 [3] 2.0 [4] and 3.0 [5]) but I don't see anything labeled VOTE with formal consensus. It looks like each of the discussions reached informal consensus after the final minor release reached feature freeze. If we were to approve this KIP, we would be formally voting that 4.0 is the next release after 3.8. If we were to follow a similar process today, we would (1) vote down a bump to 4.0 after 3.7, (2) proceed with a 3.8 release, and (3) after 3.8 feature freeze vote whether to have 4.0. There are two separate things to reach consensus about here: Whether the 3.8 release will exist, and when the 4.0 release will appear. We can vote for a 3.8 release with or without also agreeing on the timing for 4.0, and I think that committing to 4.0 now is in violation of our time-based release policy. I am whole-heartedly +1 for a 3.8, but I'm hesitant about voting on 4.0 before the feature set of 3.8 is finalized. I think there is a risk that features that are on-time and eligible for a 3.8 release could be delayed by some KIPs which are given special treatment. [1] https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan [2] https://lists.apache.org/thread/90zkqvmmw3y8j6tkgbg3md78m7hs4yn6 [3] https://lists.apache.org/thread/vr1pwbz3q0dpkphb4m4b1snxc4cy8otm [4] https://lists.apache.org/thread/9vc5q3m6y68czt929tgnk0frcf12oypt [5] https://lists.apache.org/thread/76dt0s95s73j7rwtccwjf0nh6zkn2sft Thanks all! Greg On Fri, Jan 5, 2024 at 9:25 AM Justine Olshan <jols...@confluent.io.invalid> wrote: > > While I agree we should have this release and should vote on it soon, is it > worth determining the exact items we need before we vote? Just so we are > all in agreement? > There is still some discussion on the road to 4.0 thread that may be worth > having here. > > On Fri, Jan 5, 2024 at 1:25 AM Josep Prat <josep.p...@aiven.io.invalid> > wrote: > > > Hi Colin, > > Sorry for being quiet these last days (PTO). > > > > I will start the vote thread right away. > > > > Best, > > > > > > --- > > Josep Prat > > Open Source Engineering Director, aivenjosep.p...@aiven.io | > > +491715557497 | aiven.io > > Aiven Deutschland GmbH > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > Amtsgericht Charlottenburg, HRB 209739 B > > > > On Fri, Jan 5, 2024, 00:24 Colin McCabe <cmcc...@apache.org> wrote: > > > > > Hi all, > > > > > > Since this has been open for a few weeks, are there any objections to > > > starting the vote? What do you think, Josep? > > > > > > Since 3.8 is going to be the next release (according to the KIP) we > > should > > > really vote this in as soon as possible. > > > > > > Also, I created a wiki page about the 3.8 release with a tentative > > > schedule. > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0 > > > > > > Please let me know if these dates make sense -- they're just proposals > > > right now. > > > > > > best, > > > Colin > > > > > > > > > On Thu, Dec 28, 2023, at 20:14, Colin McCabe wrote: > > > > On Thu, Dec 28, 2023, at 18:17, Justine Olshan wrote: > > > >> Hey Colin, > > > >> > > > >> Some folks were concerned about the lack of automatic unclean leader > > > >> election. I mentioned that KIP-966 would actually be better with its > > > >> aggressive recovery option. > > > >> I think folks were hoping for some availability over durability > > solution > > > >> for KRaft, so if we don't do KIP-966 we should provide an alternative > > > or be > > > >> able to convince ourselves it is not needed. > > > > > > > > Hi Justine, > > > > > > > > That's a fair point. We should specify in KIP-1012 that we need to have > > > > some way to configure the system to automatically do unclean leader > > > > election. If we run out of time implementing KIP-966, this could be > > > > something quite simple, like honoring the static > > > > unclean.leader.election = true configuration. > > > > > > > >> > > > >> I think while many folks decided KIP-853 was a blocker, there were a > > > lot of > > > >> other features that many folks were expecting so I don't think we can > > > say > > > >> definitively the only must-have is KIP-853 (and hence the discussion > > > thread > > > >> here :) ) > > > >> > > > >> Also as an aside, I filed a ticket to remove ZK from the top of the > > > >> quickstart guide. https://issues.apache.org/jira/browse/KAFKA-15975 > > > >> > > > > > > > > Yeah. There is a bunch of docs and quickstart cleanup that we should > > > > do. I don't think any of it is a blocker for 3.8 or 4.0, but the new > > > > year is a good time to clean things up. > > > > > > > > best, > > > > Colin > > > > > > > > > > > >> Justine > > > >> > > > >> On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe <cmcc...@apache.org> > > > wrote: > > > >> > > > >>> Hi Josep, > > > >>> > > > >>> Thanks for the KIP. Based on the discussions we had previously, I > > agree > > > >>> that we need a 3.8. > > > >>> > > > >>> It would be good to link to KIP-833 in the motivation section, since > > > this > > > >>> KIP builds on that one. > > > >>> > > > >>> Also, I think we should mention in KIP-1012 that 3.8 will be a > > > >>> general-purpose release that may add some new features. This was > > > something > > > >>> that we were on the fence about previously, so it would be good to > > > clarify > > > >>> it here. > > > >>> > > > >>> On another note. I don't think KIP-966 is a "must-have" for Kafka > > 3.8, > > > as > > > >>> the KIP currently states. I certainly hope that it makes it for 3.8, > > > but if > > > >>> it doesn't, it can go into 4.0. It's not needed for migration, so it > > > could > > > >>> just as easily go into 4.0 as 3.8. The only thing that KIP-966 really > > > needs > > > >>> is "KIP-853: KRaft Controller Membership Changes." > > > >>> > > > >>> Along these lines, I think we should drop the language about > > "strategic > > > >>> feature parity with Zookeeper." Kafka isn't competing with ZooKeeper, > > > and > > > >>> doesn't need feature parity with it. For example, ZK implemented > > > >>> Netty-TcNative OpenSSL Support, but we don't have that (and probably > > > won't > > > >>> in 3.8). We probably won't add this -- or if we do, it won't be so > > > that we > > > >>> can have "parity with ZK." Really the only must-have in 3.8 is > > > KIP-853, and > > > >>> we should be clear about that. > > > >>> > > > >>> I think we should start issuing a deprecation log message at ERROR > > > level > > > >>> when brokers start up in ZK mode. This message could point out that > > > some > > > >>> safety mechanisms and new features will not be available in ZK mode, > > > and > > > >>> give a link to our documentation about migration. > > > >>> > > > >>> We should probably also move the example configurations for kraft > > from > > > >>> config/kraft to config. And move the zk ones into config/zk. Or maybe > > > even > > > >>> drop the ZK ones altogether, since they're not needed for migration > > or > > > >>> upgrade. > > > >>> > > > >>> best, > > > >>> Colin > > > >>> > > > >>> > > > >>> On Fri, Dec 22, 2023, at 04:37, Josep Prat wrote: > > > >>> > On this note, I'd like to add that I would volunteer to be the > > > release > > > >>> > manager of such release 3.8.0. > > > >>> > > > > >>> > Best, > > > >>> > > > > >>> > On Fri, Dec 22, 2023 at 1:31 PM Josep Prat <josep.p...@aiven.io> > > > wrote: > > > >>> > > > > >>> >> Hi all! > > > >>> >> As agreed on the "Road to Kafka 4.0" email thread, I created > > > KIP-1012 to > > > >>> >> discuss and I'd like to open it up for discussion: > > > >>> >> > > > >>> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release > > > >>> >> > > > >>> >> Let's use this KIP to: > > > >>> >> a) Leave a papertrail agreement for the need of a 3.8 version > > > >>> >> b) Define which KIPs are the must-haves in regards to KRaft that > > > should > > > >>> be > > > >>> >> included there. > > > >>> >> > > > >>> >> Please let me know your feedback and suggestions. > > > >>> >> > > > >>> >> Best, > > > >>> >> > > > >>> >> -- > > > >>> >> [image: Aiven] <https://www.aiven.io> > > > >>> >> > > > >>> >> *Josep Prat* > > > >>> >> Open Source Engineering Director, *Aiven* > > > >>> >> josep.p...@aiven.io | +491715557497 > > > >>> >> aiven.io <https://www.aiven.io> | > > > >>> >> <https://www.facebook.com/aivencloud> > > > >>> >> <https://www.linkedin.com/company/aiven/> < > > > >>> https://twitter.com/aiven_io> > > > >>> >> *Aiven Deutschland GmbH* > > > >>> >> Alexanderufer 3-7, 10117 Berlin > > > >>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > >>> >> Amtsgericht Charlottenburg, HRB 209739 B > > > >>> >> > > > >>> > > > > >>> > > > > >>> > -- > > > >>> > [image: Aiven] <https://www.aiven.io> > > > >>> > > > > >>> > *Josep Prat* > > > >>> > Open Source Engineering Director, *Aiven* > > > >>> > josep.p...@aiven.io | +491715557497 > > > >>> > aiven.io <https://www.aiven.io> | < > > > >>> https://www.facebook.com/aivencloud> > > > >>> > <https://www.linkedin.com/company/aiven/> < > > > >>> https://twitter.com/aiven_io> > > > >>> > *Aiven Deutschland GmbH* > > > >>> > Alexanderufer 3-7, 10117 Berlin > > > >>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > >>> > Amtsgericht Charlottenburg, HRB 209739 B > > > >>> > > > > >
