Hi, given that we passed 2.8 feature freeze, I wanted to restart this thread. Currently, `trunk` is at `2.9.0-SNAPSHOT` and I am wondering if the decision for the 3.0 release is final and if we should bump the version number?
I am asking particularly because there a many Jiras with a 3.0 target release version for breaking changes and we should ensure that we have enough time to work on those tickets. -- As long as we don't agree that the next release will indeed be 3.0, those tickets are effectively blocked/pending. Thoughts? -Matthias On 10/15/20 4:28 PM, Matthias J. Sax wrote: > Thanks for clarifying Colin. Works for me. Overall, 3.0 should be guided > by the ZK removal progress and if we are not there yet, it's better to > have a 2.8 first. > > > -Matthias > > > On 10/15/20 2:41 PM, Colin McCabe wrote: >> Hi all, >> >> Just to follow up on this... since we're not quite ready for 3.0 yet, it's >> probably best if we release a 2.8 next, and then go to 3.0 after that. >> Sorry for any confusion. >> >> best, >> Colin >> >> >> On Mon, Jul 20, 2020, at 12:52, Matthias J. Sax wrote: >>> Did we reach any conclusion on the subject? >>> >>> It seems we are aiming for 2.7 after 2.6 and plan the major version bump >>> to 3.0 after 2.7 (assuming we make progress on ZK removal as planned?) >>> >>> >>> -Matthias >>> >>> >>> On 5/18/20 1:11 PM, Boyang Chen wrote: >>>> One more thing I would like to see deprecated (hopefully no one mentioned >>>> before) is the zk based consumer offset support. >>>> >>>> On Mon, May 11, 2020 at 2:15 PM Colin McCabe <cmcc...@apache.org> wrote: >>>> >>>>> Hi Michael, >>>>> >>>>> It would be better to discuss the background behind KIP-500 in a separate >>>>> thread, since this thread is about the Kafka 3.0 release. As others have >>>>> said, your questions are answered in the KIP. For example, "what is the >>>>> actual goal?" is addressed in the motivation section. >>>>> >>>>> I agree that Kafka's usage of Apache ZooKeeper could be optimized. But >>>>> there are fundamental limitations to this approach compared to storing our >>>>> metadata internally. For example, having to contact a remote server to >>>>> reload all your metadata on a controller failover simply doesn't scale >>>>> past >>>>> a certain point. >>>>> >>>>> Apache Curator is a nice API, and if we were starting again today we would >>>>> certainly consider using it. But it doesn't allow us to do anything more >>>>> efficiently than ZooKeeper could already do it. >>>>> >>>>> Finally, Kafka's core competence is logs. While our replication protocol >>>>> is not Raft, it shares many similarities with that protocol. So I think >>>>> it's a bit unfair to say that it is "catastrophic hubris" to believe we >>>>> can >>>>> implement the protocol. >>>>> >>>>> best, >>>>> Colin >>>>> >>>>> >>>>> On Sun, May 10, 2020, at 11:02, Michael K. Edwards wrote: >>>>>> Yes, I've read the KIP. But all it really says to me is "we have never >>>>>> gotten around to using ZooKeeper properly." To the extent that any of >>>>> the >>>>>> distributed-state-maintenance problems discussed in "Metadata as an Event >>>>>> Log" can be solved — and some of them intrinsically can't, because CAP >>>>>> theorem — most of them are already implemented very effectively in >>>>> Curator >>>>>> recipes. (For instance, Curator's Tree Cache >>>>>> https://curator.apache.org/curator-recipes/tree-cache.html is a good >>>>> fit to >>>>>> some of the state-maintenance needs.) >>>>>> >>>>>> Kafka does have some usage patterns that don't map neatly onto existing >>>>>> Curator recipes. For instance, neither LeaderSelector nor LeaderLatch >>>>>> implements leader preference in the way that the existing Kafka partition >>>>>> leadership election procedure does. But why not handle that by improving >>>>>> and extending Curator? That way, other Curator users benefit, and we get >>>>>> additional highly experienced reviewers' eyes on the distributed >>>>>> algorithms, which are very very tricky to get right. >>>>>> >>>>>> >>>>>> On Sun, May 10, 2020 at 10:47 AM Ron Dagostino <rndg...@gmail.com> >>>>> wrote: >>>>>> >>>>>>> Hi Michael. This is discussed in the KIP. >>>>>>> >>>>>>> >>>>>>> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-Motivation >>>>>>> >>>>>>> Ron >>>>>>> >>>>>>>> On May 10, 2020, at 1:35 PM, Michael K. Edwards < >>>>> m.k.edwa...@gmail.com> >>>>>>> wrote: >>>>>>>> >>>>>>>> What is the actual goal of removing the ZooKeeper dependency? In my >>>>>>>> experience, if ZooKeeper is properly provisioned and deployed, it's >>>>>>> largely >>>>>>>> trouble-free. (You do need to know how to use observers properly.) >>>>>>> There >>>>>>>> are some subtleties about timeouts and leadership changes, but >>>>> they're >>>>>>>> pretty small stuff. Why go to all the trouble of building a new >>>>>>>> distributed-consensus system that's going to have catastrophic bugs >>>>> for >>>>>>>> years to come? It seems like such an act of hubris to me, as well >>>>> as a >>>>>>>> massive waste of engineering effort. What is there to be gained? >>>>>>>> >>>>>>>>> On Fri, May 8, 2020 at 4:11 PM Matthias J. Sax <mj...@apache.org> >>>>>>> wrote: >>>>>>>>> >>>>>>>>> Sure, we can compile a list for Kafka Streams. But the KIP would be >>>>> for >>>>>>>>> 3.0, so I don't think it's urgent to do it now? >>>>>>>>> >>>>>>>>> >>>>>>>>> -Matthias >>>>>>>>> >>>>>>>>>> On 5/8/20 3:47 PM, Colin McCabe wrote: >>>>>>>>>> Thanks, Guozhang-- sounds like a good plan. >>>>>>>>>> >>>>>>>>>> I think it would be good to have a list of deprecated streams APIs >>>>> that >>>>>>>>> we want to remove in 3.0. Maybe it's easiest to do that as its own >>>>> KIP? >>>>>>>>>> >>>>>>>>>> For MirrorMaker 1, we should have a KIP to deprecate its use in >>>>> 2.6 if >>>>>>>>> we want to remove it in 3.0. I don't have a good sense of how >>>>>>> practical it >>>>>>>>> is to deprecate this now, so I will defer to others here. But the >>>>> KIP >>>>>>>>> freeze for 2.6 is coming soon, so if we want to make the case, now >>>>> is >>>>>>> the >>>>>>>>> time. >>>>>>>>>> >>>>>>>>>> best, >>>>>>>>>> Colin >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Thu, May 7, 2020, at 16:28, Guozhang Wang wrote: >>>>>>>>>>> Hey folks, >>>>>>>>>>> >>>>>>>>>>> Sorry for stating that the bridge release would not break any >>>>>>>>> compatibility >>>>>>>>>>> before, which is incorrect and confused many people. >>>>>>>>>>> >>>>>>>>>>> I think one way to think about the versioning is that: >>>>>>>>>>> >>>>>>>>>>> 0) In a 2.x version moving ahead we would deprecate the >>>>> ZK-dependent >>>>>>>>> tools >>>>>>>>>>> such as --zookeeper flags from various scripts (KIP-555) >>>>>>>>>>> >>>>>>>>>>> 1) In 3.0 we would at least make one incompatible change for >>>>> example >>>>>>> to >>>>>>>>>>> remove the deprecated ZK flags. >>>>>>>>>>> >>>>>>>>>>> 2) In a future major version (e.g. 4.0) we would drop ZK entirely, >>>>>>>>>>> including usages such as security credentials / broker >>>>> registration / >>>>>>>>> etc >>>>>>>>>>> which are via ZK today as well. >>>>>>>>>>> >>>>>>>>>>> Then for the bridge release(s), it can be any or all of 3.x. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> For 1), I'd love to add a few more incompatibility changes in 3.0 >>>>> from >>>>>>>>>>> Kafka Streams: we evolve Streams public APIs by deprecating and >>>>> then >>>>>>>>> remove >>>>>>>>>>> in major releases, and since 2.0 we've accumulated quite a few >>>>>>>>> deprecated >>>>>>>>>>> APIs, and I can compile a list of KIPs that contain those if >>>>> people >>>>>>> are >>>>>>>>>>> interested. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Guozhang >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Thu, May 7, 2020 at 3:53 PM Colin McCabe <cmcc...@apache.org> >>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote: >>>>>>>>>>>>>> In fact, we know that the bridge release will involve at least >>>>> one >>>>>>>>>>>>>> incompatible change. We will need to drop support for the >>>>>>>>> --zookeeper >>>>>>>>>>>>>> flags in the command-line tools. >>>>>>>>>>>>> >>>>>>>>>>>>> If the bridge release(s) and the subsequent post-ZK release are >>>>>>> _both_ >>>>>>>>>>>>> breaking changes, I think we only have one option: the 3.x line >>>>> are >>>>>>>>> the >>>>>>>>>>>>> bridge release(s), and ZK is removed in 4.0, as suggested by >>>>> Andrew >>>>>>>>>>>>> Schofield. >>>>>>>>>>>>> >>>>>>>>>>>>> Specifically: >>>>>>>>>>>>> - in order to _remove_ (not merely deprecate) the --zookeeper >>>>> args, >>>>>>> we >>>>>>>>>>>> will >>>>>>>>>>>>> need a major release. >>>>>>>>>>>>> - in oder to drop support for ZK entirely (e.g. break a bunch of >>>>>>>>> external >>>>>>>>>>>>> tooling like Cruise Control), we will need a major release. >>>>>>>>>>>>> >>>>>>>>>>>>> I count two major releases. >>>>>>>>>>>> >>>>>>>>>>>> Hi Ryanne, >>>>>>>>>>>> >>>>>>>>>>>> I agree that dropping ZK completely will need a new major release >>>>>>> after >>>>>>>>>>>> 3.0. I think that's OK and in keeping with how we've handled >>>>>>>>> deprecation >>>>>>>>>>>> and removal in the past. It's important for users to have a >>>>> smooth >>>>>>>>> upgrade >>>>>>>>>>>> path. >>>>>>>>>>>> >>>>>>>>>>>> best, >>>>>>>>>>>> Colin >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Ryanne >>>>>>>>>>>>> >>>>>>>>>>>>> - >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, May 6, 2020 at 10:52 PM Colin McCabe < >>>>> cmcc...@apache.org> >>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: >>>>>>>>>>>>>>> Hey Colin, I think we should wait until after KIP-500's >>>>> "bridge >>>>>>>>>>>>>>> release" so there is a clean break from Zookeeper after 3.0. >>>>> The >>>>>>>>>>>>>>> bridge release by definition is an attempt to not break >>>>> anything, >>>>>>> so >>>>>>>>>>>>>>> it theoretically doesn't warrant a major release. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Ryanne, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think it's important to clarify this a little bit. The >>>>> bridge >>>>>>>>>>>> release >>>>>>>>>>>>>> (really, releases, plural) allow you to upgrade from a cluster >>>>> that >>>>>>>>> is >>>>>>>>>>>>>> using ZooKeeper to one that is not using ZooKeeper. But, that >>>>>>>>> doesn't >>>>>>>>>>>>>> imply that the bridge release itself doesn't break anything. >>>>>>>>> Upgrading >>>>>>>>>>>>>> to the bridge release itself might involve some minor >>>>>>>>> incompatibility. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Kafka does occasionally have incompatible changes. In those >>>>> cases, >>>>>>>>> we >>>>>>>>>>>>>> bump the major version number. One example is that when we >>>>> went >>>>>>> from >>>>>>>>>>>>>> Kafka 1.x to Kafka 2.0, we dropped support for JDK7. This is >>>>> an >>>>>>>>>>>>>> incompatible change. >>>>>>>>>>>>>> >>>>>>>>>>>>>> In fact, we know that the bridge release will involve at least >>>>> one >>>>>>>>>>>>>> incompatible change. We will need to drop support for the >>>>>>>>> --zookeeper >>>>>>>>>>>>>> flags in the command-line tools. >>>>>>>>>>>>>> >>>>>>>>>>>>>> We've been preparing for this change for a long time. People >>>>> have >>>>>>>>>>>> spent >>>>>>>>>>>>>> a lot of effort designing new APIs that can be used instead of >>>>> the >>>>>>>>> old >>>>>>>>>>>>>> zookeeper-based code that some of the command-line tools >>>>> used. We >>>>>>>>> have >>>>>>>>>>>>>> also deprecated the old ZK-based flags. But at the end of the >>>>> day, >>>>>>>>> it >>>>>>>>>>>>>> is still an incompatible change. So it's unfortunately not >>>>>>> possible >>>>>>>>>>>> for >>>>>>>>>>>>>> the >>>>>>>>>>>>>> bridge release to be a 2.x release. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> If that's not the case (i.e. if a single "bridge release" >>>>> turns >>>>>>> out >>>>>>>>>>>> to >>>>>>>>>>>>>>> be impractical), we should consider forking 3.0 while >>>>> maintaining >>>>>>> a >>>>>>>>>>>>>>> line of Zookeeper-dependent Kafka in 2.x. That way 3.x can >>>>> evolve >>>>>>>>>>>>>>> dramatically without breaking the 2.x line. In particular, >>>>>>> anything >>>>>>>>>>>>>>> related to removing Zookeeper could land in pre-3.0 while >>>>> every >>>>>>>>> other >>>>>>>>>>>>>>> feature targets 2.6. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Just to be super clear about this, what we want to do here is >>>>>>> support >>>>>>>>>>>>>> operating in __either__ KIP-500 mode and legacy mode for a >>>>> while. >>>>>>> So >>>>>>>>>>>> the >>>>>>>>>>>>>> same branch will have support for both the old way and the new >>>>> way >>>>>>> of >>>>>>>>>>>>>> managing metadata. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This will allow us to get an "alpha" version of the KIP-500 >>>>> mode >>>>>>> out >>>>>>>>>>>> early >>>>>>>>>>>>>> for people to experiment with. It also greatly reduces the >>>>> number >>>>>>> of >>>>>>>>>>>> Kafka >>>>>>>>>>>>>> releases we have to make, and the amount of backporting we >>>>> have to >>>>>>>>> do. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If you are proposing 2.6 should be the "bridge release", I >>>>> think >>>>>>>>> this >>>>>>>>>>>>>>> is premature given Kafka's time-based release schedule. If the >>>>>>>>> bridge >>>>>>>>>>>>>>> features happen to be merged before 2.6's feature freeze, then >>>>>>> sure >>>>>>>>>>>> -- >>>>>>>>>>>>>>> let's make that the bridge release in retrospect. And if we >>>>> get >>>>>>> all >>>>>>>>>>>>>>> the post-Zookeeper features merged before 2.7, I'm onboard >>>>> with >>>>>>>>>>>> naming >>>>>>>>>>>>>>> it "3.0" instead. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> That said, we should aim to remove legacy MirrorMaker before >>>>> 3.0 >>>>>>> as >>>>>>>>>>>>>>> well. I'm happy to drive that additional breaking change. >>>>> Maybe >>>>>>> 2.6 >>>>>>>>>>>>>>> can be the "bridge" for MM2 as well. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I don't have a strong opinion either way about this, but if we >>>>> want >>>>>>>>> to >>>>>>>>>>>>>> remove the original MirrorMaker, we have to deprecate it first, >>>>>>>>>>>> right? Are >>>>>>>>>>>>>> we ready to do that? >>>>>>>>>>>>>> >>>>>>>>>>>>>> best, >>>>>>>>>>>>>> Colin >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ryanne >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, May 4, 2020, 5:05 PM Colin McCabe <cmcc...@apache.org >>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We've had a few proposals recently for incompatible >>>>> changes. One >>>>>>>>>>>> of >>>>>>>>>>>>>>>> them is my KIP-604: Remove ZooKeeper Flags from the >>>>>>> Administrative >>>>>>>>>>>>>>>> Tools. The other is Boyang's KIP-590: Redirect ZK Mutation >>>>>>>>>>>>>>>> Protocols to the Controller. I think it's time to start >>>>> thinking >>>>>>>>>>>>>>>> about Kafka 3.0. Specifically, I think we should move to 3.0 >>>>>>> after >>>>>>>>>>>>>>>> the 2.6 release. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> From the perspective of KIP-500, in Kafka 3.x we'd like to >>>>> make >>>>>>>>>>>>>>>> running in a ZooKeeper-less mode possible (but not yet the >>>>>>>>>>>> default.) >>>>>>>>>>>>>>>> This is the motivation behind KIP-590 and KIP-604, as well as >>>>>>> some >>>>>>>>>>>>>>>> of the other KIPs we've done recently. Since it will take >>>>> some >>>>>>>>>>>> time >>>>>>>>>>>>>>>> to stabilize the new ZooKeeper-free Kafka code, we will hide >>>>> it >>>>>>>>>>>>>>>> behind an option initially. (We'll have a KIP describing >>>>> this all >>>>>>>>>>>> in >>>>>>>>>>>>>>>> detail soon.) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> What does everyone think about having Kafka 3.0 come up next >>>>>>> after >>>>>>>>>>>>>>>> 2.6? Are there any other things we should change in the 2.6 >>>>> -> >>>>>>> 3.0 >>>>>>>>>>>>>>>> transition? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> best, Colin >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> -- Guozhang >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >>> Attachments: >>> * signature.asc
