Hi all, First, I agree with what has been discussed. Having 3.x as the bridge releases and entirely removing ZK in 4.0 makes total sense.
Second, what would you think about removing the auto topics creation in 3.0? It is not recommended to use it anymore and that could simplify a bit our path towards removing ZK. We could deprecate it in 2.6 already and remove it in 3.0. If you guys generally agree, I could draft a KIP and kick off the discussion. Best, David On Sun, May 10, 2020 at 8:02 PM Michael K. Edwards <m.k.edwa...@gmail.com> wrote: > Yes, I've read the KIP. But all it really says to me is "we have never > gotten around to using ZooKeeper properly." To the extent that any of the > distributed-state-maintenance problems discussed in "Metadata as an Event > Log" can be solved — and some of them intrinsically can't, because CAP > theorem — most of them are already implemented very effectively in Curator > recipes. (For instance, Curator's Tree Cache > https://curator.apache.org/curator-recipes/tree-cache.html is a good fit > to > some of the state-maintenance needs.) > > Kafka does have some usage patterns that don't map neatly onto existing > Curator recipes. For instance, neither LeaderSelector nor LeaderLatch > implements leader preference in the way that the existing Kafka partition > leadership election procedure does. But why not handle that by improving > and extending Curator? That way, other Curator users benefit, and we get > additional highly experienced reviewers' eyes on the distributed > algorithms, which are very very tricky to get right. > > > On Sun, May 10, 2020 at 10:47 AM Ron Dagostino <rndg...@gmail.com> wrote: > > > Hi Michael. This is discussed in the KIP. > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-Motivation > > > > Ron > > > > > On May 10, 2020, at 1:35 PM, Michael K. Edwards <m.k.edwa...@gmail.com > > > > wrote: > > > > > > What is the actual goal of removing the ZooKeeper dependency? In my > > > experience, if ZooKeeper is properly provisioned and deployed, it's > > largely > > > trouble-free. (You do need to know how to use observers properly.) > > There > > > are some subtleties about timeouts and leadership changes, but they're > > > pretty small stuff. Why go to all the trouble of building a new > > > distributed-consensus system that's going to have catastrophic bugs for > > > years to come? It seems like such an act of hubris to me, as well as a > > > massive waste of engineering effort. What is there to be gained? > > > > > >> On Fri, May 8, 2020 at 4:11 PM Matthias J. Sax <mj...@apache.org> > > wrote: > > >> > > >> Sure, we can compile a list for Kafka Streams. But the KIP would be > for > > >> 3.0, so I don't think it's urgent to do it now? > > >> > > >> > > >> -Matthias > > >> > > >>> On 5/8/20 3:47 PM, Colin McCabe wrote: > > >>> Thanks, Guozhang-- sounds like a good plan. > > >>> > > >>> I think it would be good to have a list of deprecated streams APIs > that > > >> we want to remove in 3.0. Maybe it's easiest to do that as its own > KIP? > > >>> > > >>> For MirrorMaker 1, we should have a KIP to deprecate its use in 2.6 > if > > >> we want to remove it in 3.0. I don't have a good sense of how > > practical it > > >> is to deprecate this now, so I will defer to others here. But the KIP > > >> freeze for 2.6 is coming soon, so if we want to make the case, now is > > the > > >> time. > > >>> > > >>> best, > > >>> Colin > > >>> > > >>> > > >>>> On Thu, May 7, 2020, at 16:28, Guozhang Wang wrote: > > >>>> Hey folks, > > >>>> > > >>>> Sorry for stating that the bridge release would not break any > > >> compatibility > > >>>> before, which is incorrect and confused many people. > > >>>> > > >>>> I think one way to think about the versioning is that: > > >>>> > > >>>> 0) In a 2.x version moving ahead we would deprecate the ZK-dependent > > >> tools > > >>>> such as --zookeeper flags from various scripts (KIP-555) > > >>>> > > >>>> 1) In 3.0 we would at least make one incompatible change for example > > to > > >>>> remove the deprecated ZK flags. > > >>>> > > >>>> 2) In a future major version (e.g. 4.0) we would drop ZK entirely, > > >>>> including usages such as security credentials / broker registration > / > > >> etc > > >>>> which are via ZK today as well. > > >>>> > > >>>> Then for the bridge release(s), it can be any or all of 3.x. > > >>>> > > >>>> > > >>>> For 1), I'd love to add a few more incompatibility changes in 3.0 > from > > >>>> Kafka Streams: we evolve Streams public APIs by deprecating and then > > >> remove > > >>>> in major releases, and since 2.0 we've accumulated quite a few > > >> deprecated > > >>>> APIs, and I can compile a list of KIPs that contain those if people > > are > > >>>> interested. > > >>>> > > >>>> > > >>>> Guozhang > > >>>> > > >>>> > > >>>>> On Thu, May 7, 2020 at 3:53 PM Colin McCabe <cmcc...@apache.org> > > wrote: > > >>>>> > > >>>>> On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote: > > >>>>>>> In fact, we know that the bridge release will involve at least > one > > >>>>>>> incompatible change. We will need to drop support for the > > >> --zookeeper > > >>>>>>> flags in the command-line tools. > > >>>>>> > > >>>>>> If the bridge release(s) and the subsequent post-ZK release are > > _both_ > > >>>>>> breaking changes, I think we only have one option: the 3.x line > are > > >> the > > >>>>>> bridge release(s), and ZK is removed in 4.0, as suggested by > Andrew > > >>>>>> Schofield. > > >>>>>> > > >>>>>> Specifically: > > >>>>>> - in order to _remove_ (not merely deprecate) the --zookeeper > args, > > we > > >>>>> will > > >>>>>> need a major release. > > >>>>>> - in oder to drop support for ZK entirely (e.g. break a bunch of > > >> external > > >>>>>> tooling like Cruise Control), we will need a major release. > > >>>>>> > > >>>>>> I count two major releases. > > >>>>> > > >>>>> Hi Ryanne, > > >>>>> > > >>>>> I agree that dropping ZK completely will need a new major release > > after > > >>>>> 3.0. I think that's OK and in keeping with how we've handled > > >> deprecation > > >>>>> and removal in the past. It's important for users to have a smooth > > >> upgrade > > >>>>> path. > > >>>>> > > >>>>> best, > > >>>>> Colin > > >>>>> > > >>>>>> > > >>>>>> Ryanne > > >>>>>> > > >>>>>> - > > >>>>>> > > >>>>>> On Wed, May 6, 2020 at 10:52 PM Colin McCabe <cmcc...@apache.org> > > >> wrote: > > >>>>>> > > >>>>>>>> On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: > > >>>>>>>> Hey Colin, I think we should wait until after KIP-500's "bridge > > >>>>>>>> release" so there is a clean break from Zookeeper after 3.0. The > > >>>>>>>> bridge release by definition is an attempt to not break > anything, > > so > > >>>>>>>> it theoretically doesn't warrant a major release. > > >>>>>>> > > >>>>>>> Hi Ryanne, > > >>>>>>> > > >>>>>>> I think it's important to clarify this a little bit. The bridge > > >>>>> release > > >>>>>>> (really, releases, plural) allow you to upgrade from a cluster > that > > >> is > > >>>>>>> using ZooKeeper to one that is not using ZooKeeper. But, that > > >> doesn't > > >>>>>>> imply that the bridge release itself doesn't break anything. > > >> Upgrading > > >>>>>>> to the bridge release itself might involve some minor > > >> incompatibility. > > >>>>>>> > > >>>>>>> Kafka does occasionally have incompatible changes. In those > cases, > > >> we > > >>>>>>> bump the major version number. One example is that when we went > > from > > >>>>>>> Kafka 1.x to Kafka 2.0, we dropped support for JDK7. This is an > > >>>>>>> incompatible change. > > >>>>>>> > > >>>>>>> In fact, we know that the bridge release will involve at least > one > > >>>>>>> incompatible change. We will need to drop support for the > > >> --zookeeper > > >>>>>>> flags in the command-line tools. > > >>>>>>> > > >>>>>>> We've been preparing for this change for a long time. People > have > > >>>>> spent > > >>>>>>> a lot of effort designing new APIs that can be used instead of > the > > >> old > > >>>>>>> zookeeper-based code that some of the command-line tools used. > We > > >> have > > >>>>>>> also deprecated the old ZK-based flags. But at the end of the > day, > > >> it > > >>>>>>> is still an incompatible change. So it's unfortunately not > > possible > > >>>>> for > > >>>>>>> the > > >>>>>>> bridge release to be a 2.x release. > > >>>>>>> > > >>>>>>>> If that's not the case (i.e. if a single "bridge release" turns > > out > > >>>>> to > > >>>>>>>> be impractical), we should consider forking 3.0 while > maintaining > > a > > >>>>>>>> line of Zookeeper-dependent Kafka in 2.x. That way 3.x can > evolve > > >>>>>>>> dramatically without breaking the 2.x line. In particular, > > anything > > >>>>>>>> related to removing Zookeeper could land in pre-3.0 while every > > >> other > > >>>>>>>> feature targets 2.6. > > >>>>>>> > > >>>>>>> Just to be super clear about this, what we want to do here is > > support > > >>>>>>> operating in __either__ KIP-500 mode and legacy mode for a while. > > So > > >>>>> the > > >>>>>>> same branch will have support for both the old way and the new > way > > of > > >>>>>>> managing metadata. > > >>>>>>> > > >>>>>>> This will allow us to get an "alpha" version of the KIP-500 mode > > out > > >>>>> early > > >>>>>>> for people to experiment with. It also greatly reduces the > number > > of > > >>>>> Kafka > > >>>>>>> releases we have to make, and the amount of backporting we have > to > > >> do. > > >>>>>>> > > >>>>>>>> > > >>>>>>>> If you are proposing 2.6 should be the "bridge release", I think > > >> this > > >>>>>>>> is premature given Kafka's time-based release schedule. If the > > >> bridge > > >>>>>>>> features happen to be merged before 2.6's feature freeze, then > > sure > > >>>>> -- > > >>>>>>>> let's make that the bridge release in retrospect. And if we get > > all > > >>>>>>>> the post-Zookeeper features merged before 2.7, I'm onboard with > > >>>>> naming > > >>>>>>>> it "3.0" instead. > > >>>>>>>> > > >>>>>>>> That said, we should aim to remove legacy MirrorMaker before 3.0 > > as > > >>>>>>>> well. I'm happy to drive that additional breaking change. Maybe > > 2.6 > > >>>>>>>> can be the "bridge" for MM2 as well. > > >>>>>>> > > >>>>>>> I don't have a strong opinion either way about this, but if we > want > > >> to > > >>>>>>> remove the original MirrorMaker, we have to deprecate it first, > > >>>>> right? Are > > >>>>>>> we ready to do that? > > >>>>>>> > > >>>>>>> best, > > >>>>>>> Colin > > >>>>>>> > > >>>>>>>> > > >>>>>>>> Ryanne > > >>>>>>>> > > >>>>>>>> On Mon, May 4, 2020, 5:05 PM Colin McCabe <cmcc...@apache.org> > > >>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Hi all, > > >>>>>>>>> > > >>>>>>>>> We've had a few proposals recently for incompatible changes. > One > > >>>>> of > > >>>>>>>>> them is my KIP-604: Remove ZooKeeper Flags from the > > Administrative > > >>>>>>>>> Tools. The other is Boyang's KIP-590: Redirect ZK Mutation > > >>>>>>>>> Protocols to the Controller. I think it's time to start > thinking > > >>>>>>>>> about Kafka 3.0. Specifically, I think we should move to 3.0 > > after > > >>>>>>>>> the 2.6 release. > > >>>>>>>>> > > >>>>>>>>> From the perspective of KIP-500, in Kafka 3.x we'd like to make > > >>>>>>>>> running in a ZooKeeper-less mode possible (but not yet the > > >>>>> default.) > > >>>>>>>>> This is the motivation behind KIP-590 and KIP-604, as well as > > some > > >>>>>>>>> of the other KIPs we've done recently. Since it will take some > > >>>>> time > > >>>>>>>>> to stabilize the new ZooKeeper-free Kafka code, we will hide it > > >>>>>>>>> behind an option initially. (We'll have a KIP describing this > all > > >>>>> in > > >>>>>>>>> detail soon.) > > >>>>>>>>> > > >>>>>>>>> What does everyone think about having Kafka 3.0 come up next > > after > > >>>>>>>>> 2.6? Are there any other things we should change in the 2.6 -> > > 3.0 > > >>>>>>>>> transition? > > >>>>>>>>> > > >>>>>>>>> best, Colin > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>>> > > >>>> -- > > >>>> -- Guozhang > > >>>> > > >> > > >> > > >
