Hi all, Just to follow up on this... since we're not quite ready for 3.0 yet, it's probably best if we release a 2.8 next, and then go to 3.0 after that. Sorry for any confusion.
best, Colin On Mon, Jul 20, 2020, at 12:52, Matthias J. Sax wrote: > Did we reach any conclusion on the subject? > > It seems we are aiming for 2.7 after 2.6 and plan the major version bump > to 3.0 after 2.7 (assuming we make progress on ZK removal as planned?) > > > -Matthias > > > On 5/18/20 1:11 PM, Boyang Chen wrote: > > One more thing I would like to see deprecated (hopefully no one mentioned > > before) is the zk based consumer offset support. > > > > On Mon, May 11, 2020 at 2:15 PM Colin McCabe <cmcc...@apache.org> wrote: > > > >> Hi Michael, > >> > >> It would be better to discuss the background behind KIP-500 in a separate > >> thread, since this thread is about the Kafka 3.0 release. As others have > >> said, your questions are answered in the KIP. For example, "what is the > >> actual goal?" is addressed in the motivation section. > >> > >> I agree that Kafka's usage of Apache ZooKeeper could be optimized. But > >> there are fundamental limitations to this approach compared to storing our > >> metadata internally. For example, having to contact a remote server to > >> reload all your metadata on a controller failover simply doesn't scale past > >> a certain point. > >> > >> Apache Curator is a nice API, and if we were starting again today we would > >> certainly consider using it. But it doesn't allow us to do anything more > >> efficiently than ZooKeeper could already do it. > >> > >> Finally, Kafka's core competence is logs. While our replication protocol > >> is not Raft, it shares many similarities with that protocol. So I think > >> it's a bit unfair to say that it is "catastrophic hubris" to believe we can > >> implement the protocol. > >> > >> best, > >> Colin > >> > >> > >> On Sun, May 10, 2020, at 11:02, Michael K. Edwards wrote: > >>> Yes, I've read the KIP. But all it really says to me is "we have never > >>> gotten around to using ZooKeeper properly." To the extent that any of > >> the > >>> distributed-state-maintenance problems discussed in "Metadata as an Event > >>> Log" can be solved — and some of them intrinsically can't, because CAP > >>> theorem — most of them are already implemented very effectively in > >> Curator > >>> recipes. (For instance, Curator's Tree Cache > >>> https://curator.apache.org/curator-recipes/tree-cache.html is a good > >> fit to > >>> some of the state-maintenance needs.) > >>> > >>> Kafka does have some usage patterns that don't map neatly onto existing > >>> Curator recipes. For instance, neither LeaderSelector nor LeaderLatch > >>> implements leader preference in the way that the existing Kafka partition > >>> leadership election procedure does. But why not handle that by improving > >>> and extending Curator? That way, other Curator users benefit, and we get > >>> additional highly experienced reviewers' eyes on the distributed > >>> algorithms, which are very very tricky to get right. > >>> > >>> > >>> On Sun, May 10, 2020 at 10:47 AM Ron Dagostino <rndg...@gmail.com> > >> wrote: > >>> > >>>> Hi Michael. This is discussed in the KIP. > >>>> > >>>> > >>>> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-Motivation > >>>> > >>>> Ron > >>>> > >>>>> On May 10, 2020, at 1:35 PM, Michael K. Edwards < > >> m.k.edwa...@gmail.com> > >>>> wrote: > >>>>> > >>>>> What is the actual goal of removing the ZooKeeper dependency? In my > >>>>> experience, if ZooKeeper is properly provisioned and deployed, it's > >>>> largely > >>>>> trouble-free. (You do need to know how to use observers properly.) > >>>> There > >>>>> are some subtleties about timeouts and leadership changes, but > >> they're > >>>>> pretty small stuff. Why go to all the trouble of building a new > >>>>> distributed-consensus system that's going to have catastrophic bugs > >> for > >>>>> years to come? It seems like such an act of hubris to me, as well > >> as a > >>>>> massive waste of engineering effort. What is there to be gained? > >>>>> > >>>>>> On Fri, May 8, 2020 at 4:11 PM Matthias J. Sax <mj...@apache.org> > >>>> wrote: > >>>>>> > >>>>>> Sure, we can compile a list for Kafka Streams. But the KIP would be > >> for > >>>>>> 3.0, so I don't think it's urgent to do it now? > >>>>>> > >>>>>> > >>>>>> -Matthias > >>>>>> > >>>>>>> On 5/8/20 3:47 PM, Colin McCabe wrote: > >>>>>>> Thanks, Guozhang-- sounds like a good plan. > >>>>>>> > >>>>>>> I think it would be good to have a list of deprecated streams APIs > >> that > >>>>>> we want to remove in 3.0. Maybe it's easiest to do that as its own > >> KIP? > >>>>>>> > >>>>>>> For MirrorMaker 1, we should have a KIP to deprecate its use in > >> 2.6 if > >>>>>> we want to remove it in 3.0. I don't have a good sense of how > >>>> practical it > >>>>>> is to deprecate this now, so I will defer to others here. But the > >> KIP > >>>>>> freeze for 2.6 is coming soon, so if we want to make the case, now > >> is > >>>> the > >>>>>> time. > >>>>>>> > >>>>>>> best, > >>>>>>> Colin > >>>>>>> > >>>>>>> > >>>>>>>> On Thu, May 7, 2020, at 16:28, Guozhang Wang wrote: > >>>>>>>> Hey folks, > >>>>>>>> > >>>>>>>> Sorry for stating that the bridge release would not break any > >>>>>> compatibility > >>>>>>>> before, which is incorrect and confused many people. > >>>>>>>> > >>>>>>>> I think one way to think about the versioning is that: > >>>>>>>> > >>>>>>>> 0) In a 2.x version moving ahead we would deprecate the > >> ZK-dependent > >>>>>> tools > >>>>>>>> such as --zookeeper flags from various scripts (KIP-555) > >>>>>>>> > >>>>>>>> 1) In 3.0 we would at least make one incompatible change for > >> example > >>>> to > >>>>>>>> remove the deprecated ZK flags. > >>>>>>>> > >>>>>>>> 2) In a future major version (e.g. 4.0) we would drop ZK entirely, > >>>>>>>> including usages such as security credentials / broker > >> registration / > >>>>>> etc > >>>>>>>> which are via ZK today as well. > >>>>>>>> > >>>>>>>> Then for the bridge release(s), it can be any or all of 3.x. > >>>>>>>> > >>>>>>>> > >>>>>>>> For 1), I'd love to add a few more incompatibility changes in 3.0 > >> from > >>>>>>>> Kafka Streams: we evolve Streams public APIs by deprecating and > >> then > >>>>>> remove > >>>>>>>> in major releases, and since 2.0 we've accumulated quite a few > >>>>>> deprecated > >>>>>>>> APIs, and I can compile a list of KIPs that contain those if > >> people > >>>> are > >>>>>>>> interested. > >>>>>>>> > >>>>>>>> > >>>>>>>> Guozhang > >>>>>>>> > >>>>>>>> > >>>>>>>>> On Thu, May 7, 2020 at 3:53 PM Colin McCabe <cmcc...@apache.org> > >>>> wrote: > >>>>>>>>> > >>>>>>>>> On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote: > >>>>>>>>>>> In fact, we know that the bridge release will involve at least > >> one > >>>>>>>>>>> incompatible change. We will need to drop support for the > >>>>>> --zookeeper > >>>>>>>>>>> flags in the command-line tools. > >>>>>>>>>> > >>>>>>>>>> If the bridge release(s) and the subsequent post-ZK release are > >>>> _both_ > >>>>>>>>>> breaking changes, I think we only have one option: the 3.x line > >> are > >>>>>> the > >>>>>>>>>> bridge release(s), and ZK is removed in 4.0, as suggested by > >> Andrew > >>>>>>>>>> Schofield. > >>>>>>>>>> > >>>>>>>>>> Specifically: > >>>>>>>>>> - in order to _remove_ (not merely deprecate) the --zookeeper > >> args, > >>>> we > >>>>>>>>> will > >>>>>>>>>> need a major release. > >>>>>>>>>> - in oder to drop support for ZK entirely (e.g. break a bunch of > >>>>>> external > >>>>>>>>>> tooling like Cruise Control), we will need a major release. > >>>>>>>>>> > >>>>>>>>>> I count two major releases. > >>>>>>>>> > >>>>>>>>> Hi Ryanne, > >>>>>>>>> > >>>>>>>>> I agree that dropping ZK completely will need a new major release > >>>> after > >>>>>>>>> 3.0. I think that's OK and in keeping with how we've handled > >>>>>> deprecation > >>>>>>>>> and removal in the past. It's important for users to have a > >> smooth > >>>>>> upgrade > >>>>>>>>> path. > >>>>>>>>> > >>>>>>>>> best, > >>>>>>>>> Colin > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Ryanne > >>>>>>>>>> > >>>>>>>>>> - > >>>>>>>>>> > >>>>>>>>>> On Wed, May 6, 2020 at 10:52 PM Colin McCabe < > >> cmcc...@apache.org> > >>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>>> On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: > >>>>>>>>>>>> Hey Colin, I think we should wait until after KIP-500's > >> "bridge > >>>>>>>>>>>> release" so there is a clean break from Zookeeper after 3.0. > >> The > >>>>>>>>>>>> bridge release by definition is an attempt to not break > >> anything, > >>>> so > >>>>>>>>>>>> it theoretically doesn't warrant a major release. > >>>>>>>>>>> > >>>>>>>>>>> Hi Ryanne, > >>>>>>>>>>> > >>>>>>>>>>> I think it's important to clarify this a little bit. The > >> bridge > >>>>>>>>> release > >>>>>>>>>>> (really, releases, plural) allow you to upgrade from a cluster > >> that > >>>>>> is > >>>>>>>>>>> using ZooKeeper to one that is not using ZooKeeper. But, that > >>>>>> doesn't > >>>>>>>>>>> imply that the bridge release itself doesn't break anything. > >>>>>> Upgrading > >>>>>>>>>>> to the bridge release itself might involve some minor > >>>>>> incompatibility. > >>>>>>>>>>> > >>>>>>>>>>> Kafka does occasionally have incompatible changes. In those > >> cases, > >>>>>> we > >>>>>>>>>>> bump the major version number. One example is that when we > >> went > >>>> from > >>>>>>>>>>> Kafka 1.x to Kafka 2.0, we dropped support for JDK7. This is > >> an > >>>>>>>>>>> incompatible change. > >>>>>>>>>>> > >>>>>>>>>>> In fact, we know that the bridge release will involve at least > >> one > >>>>>>>>>>> incompatible change. We will need to drop support for the > >>>>>> --zookeeper > >>>>>>>>>>> flags in the command-line tools. > >>>>>>>>>>> > >>>>>>>>>>> We've been preparing for this change for a long time. People > >> have > >>>>>>>>> spent > >>>>>>>>>>> a lot of effort designing new APIs that can be used instead of > >> the > >>>>>> old > >>>>>>>>>>> zookeeper-based code that some of the command-line tools > >> used. We > >>>>>> have > >>>>>>>>>>> also deprecated the old ZK-based flags. But at the end of the > >> day, > >>>>>> it > >>>>>>>>>>> is still an incompatible change. So it's unfortunately not > >>>> possible > >>>>>>>>> for > >>>>>>>>>>> the > >>>>>>>>>>> bridge release to be a 2.x release. > >>>>>>>>>>> > >>>>>>>>>>>> If that's not the case (i.e. if a single "bridge release" > >> turns > >>>> out > >>>>>>>>> to > >>>>>>>>>>>> be impractical), we should consider forking 3.0 while > >> maintaining > >>>> a > >>>>>>>>>>>> line of Zookeeper-dependent Kafka in 2.x. That way 3.x can > >> evolve > >>>>>>>>>>>> dramatically without breaking the 2.x line. In particular, > >>>> anything > >>>>>>>>>>>> related to removing Zookeeper could land in pre-3.0 while > >> every > >>>>>> other > >>>>>>>>>>>> feature targets 2.6. > >>>>>>>>>>> > >>>>>>>>>>> Just to be super clear about this, what we want to do here is > >>>> support > >>>>>>>>>>> operating in __either__ KIP-500 mode and legacy mode for a > >> while. > >>>> So > >>>>>>>>> the > >>>>>>>>>>> same branch will have support for both the old way and the new > >> way > >>>> of > >>>>>>>>>>> managing metadata. > >>>>>>>>>>> > >>>>>>>>>>> This will allow us to get an "alpha" version of the KIP-500 > >> mode > >>>> out > >>>>>>>>> early > >>>>>>>>>>> for people to experiment with. It also greatly reduces the > >> number > >>>> of > >>>>>>>>> Kafka > >>>>>>>>>>> releases we have to make, and the amount of backporting we > >> have to > >>>>>> do. > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> If you are proposing 2.6 should be the "bridge release", I > >> think > >>>>>> this > >>>>>>>>>>>> is premature given Kafka's time-based release schedule. If the > >>>>>> bridge > >>>>>>>>>>>> features happen to be merged before 2.6's feature freeze, then > >>>> sure > >>>>>>>>> -- > >>>>>>>>>>>> let's make that the bridge release in retrospect. And if we > >> get > >>>> all > >>>>>>>>>>>> the post-Zookeeper features merged before 2.7, I'm onboard > >> with > >>>>>>>>> naming > >>>>>>>>>>>> it "3.0" instead. > >>>>>>>>>>>> > >>>>>>>>>>>> That said, we should aim to remove legacy MirrorMaker before > >> 3.0 > >>>> as > >>>>>>>>>>>> well. I'm happy to drive that additional breaking change. > >> Maybe > >>>> 2.6 > >>>>>>>>>>>> can be the "bridge" for MM2 as well. > >>>>>>>>>>> > >>>>>>>>>>> I don't have a strong opinion either way about this, but if we > >> want > >>>>>> to > >>>>>>>>>>> remove the original MirrorMaker, we have to deprecate it first, > >>>>>>>>> right? Are > >>>>>>>>>>> we ready to do that? > >>>>>>>>>>> > >>>>>>>>>>> best, > >>>>>>>>>>> Colin > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Ryanne > >>>>>>>>>>>> > >>>>>>>>>>>> On Mon, May 4, 2020, 5:05 PM Colin McCabe <cmcc...@apache.org > >>> > >>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>> > >>>>>>>>>>>>> We've had a few proposals recently for incompatible > >> changes. One > >>>>>>>>> of > >>>>>>>>>>>>> them is my KIP-604: Remove ZooKeeper Flags from the > >>>> Administrative > >>>>>>>>>>>>> Tools. The other is Boyang's KIP-590: Redirect ZK Mutation > >>>>>>>>>>>>> Protocols to the Controller. I think it's time to start > >> thinking > >>>>>>>>>>>>> about Kafka 3.0. Specifically, I think we should move to 3.0 > >>>> after > >>>>>>>>>>>>> the 2.6 release. > >>>>>>>>>>>>> > >>>>>>>>>>>>> From the perspective of KIP-500, in Kafka 3.x we'd like to > >> make > >>>>>>>>>>>>> running in a ZooKeeper-less mode possible (but not yet the > >>>>>>>>> default.) > >>>>>>>>>>>>> This is the motivation behind KIP-590 and KIP-604, as well as > >>>> some > >>>>>>>>>>>>> of the other KIPs we've done recently. Since it will take > >> some > >>>>>>>>> time > >>>>>>>>>>>>> to stabilize the new ZooKeeper-free Kafka code, we will hide > >> it > >>>>>>>>>>>>> behind an option initially. (We'll have a KIP describing > >> this all > >>>>>>>>> in > >>>>>>>>>>>>> detail soon.) > >>>>>>>>>>>>> > >>>>>>>>>>>>> What does everyone think about having Kafka 3.0 come up next > >>>> after > >>>>>>>>>>>>> 2.6? Are there any other things we should change in the 2.6 > >> -> > >>>> 3.0 > >>>>>>>>>>>>> transition? > >>>>>>>>>>>>> > >>>>>>>>>>>>> best, Colin > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> -- Guozhang > >>>>>>>> > >>>>>> > >>>>>> > >>>> > >>> > >> > > > > > Attachments: > * signature.asc
