Did we reach any conclusion on the subject? It seems we are aiming for 2.7 after 2.6 and plan the major version bump to 3.0 after 2.7 (assuming we make progress on ZK removal as planned?)
-Matthias On 5/18/20 1:11 PM, Boyang Chen wrote: > One more thing I would like to see deprecated (hopefully no one mentioned > before) is the zk based consumer offset support. > > On Mon, May 11, 2020 at 2:15 PM Colin McCabe <cmcc...@apache.org> wrote: > >> Hi Michael, >> >> It would be better to discuss the background behind KIP-500 in a separate >> thread, since this thread is about the Kafka 3.0 release. As others have >> said, your questions are answered in the KIP. For example, "what is the >> actual goal?" is addressed in the motivation section. >> >> I agree that Kafka's usage of Apache ZooKeeper could be optimized. But >> there are fundamental limitations to this approach compared to storing our >> metadata internally. For example, having to contact a remote server to >> reload all your metadata on a controller failover simply doesn't scale past >> a certain point. >> >> Apache Curator is a nice API, and if we were starting again today we would >> certainly consider using it. But it doesn't allow us to do anything more >> efficiently than ZooKeeper could already do it. >> >> Finally, Kafka's core competence is logs. While our replication protocol >> is not Raft, it shares many similarities with that protocol. So I think >> it's a bit unfair to say that it is "catastrophic hubris" to believe we can >> implement the protocol. >> >> best, >> Colin >> >> >> On Sun, May 10, 2020, at 11:02, Michael K. Edwards wrote: >>> Yes, I've read the KIP. But all it really says to me is "we have never >>> gotten around to using ZooKeeper properly." To the extent that any of >> the >>> distributed-state-maintenance problems discussed in "Metadata as an Event >>> Log" can be solved — and some of them intrinsically can't, because CAP >>> theorem — most of them are already implemented very effectively in >> Curator >>> recipes. (For instance, Curator's Tree Cache >>> https://curator.apache.org/curator-recipes/tree-cache.html is a good >> fit to >>> some of the state-maintenance needs.) >>> >>> Kafka does have some usage patterns that don't map neatly onto existing >>> Curator recipes. For instance, neither LeaderSelector nor LeaderLatch >>> implements leader preference in the way that the existing Kafka partition >>> leadership election procedure does. But why not handle that by improving >>> and extending Curator? That way, other Curator users benefit, and we get >>> additional highly experienced reviewers' eyes on the distributed >>> algorithms, which are very very tricky to get right. >>> >>> >>> On Sun, May 10, 2020 at 10:47 AM Ron Dagostino <rndg...@gmail.com> >> wrote: >>> >>>> Hi Michael. This is discussed in the KIP. >>>> >>>> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-Motivation >>>> >>>> Ron >>>> >>>>> On May 10, 2020, at 1:35 PM, Michael K. Edwards < >> m.k.edwa...@gmail.com> >>>> wrote: >>>>> >>>>> What is the actual goal of removing the ZooKeeper dependency? In my >>>>> experience, if ZooKeeper is properly provisioned and deployed, it's >>>> largely >>>>> trouble-free. (You do need to know how to use observers properly.) >>>> There >>>>> are some subtleties about timeouts and leadership changes, but >> they're >>>>> pretty small stuff. Why go to all the trouble of building a new >>>>> distributed-consensus system that's going to have catastrophic bugs >> for >>>>> years to come? It seems like such an act of hubris to me, as well >> as a >>>>> massive waste of engineering effort. What is there to be gained? >>>>> >>>>>> On Fri, May 8, 2020 at 4:11 PM Matthias J. Sax <mj...@apache.org> >>>> wrote: >>>>>> >>>>>> Sure, we can compile a list for Kafka Streams. But the KIP would be >> for >>>>>> 3.0, so I don't think it's urgent to do it now? >>>>>> >>>>>> >>>>>> -Matthias >>>>>> >>>>>>> On 5/8/20 3:47 PM, Colin McCabe wrote: >>>>>>> Thanks, Guozhang-- sounds like a good plan. >>>>>>> >>>>>>> I think it would be good to have a list of deprecated streams APIs >> that >>>>>> we want to remove in 3.0. Maybe it's easiest to do that as its own >> KIP? >>>>>>> >>>>>>> For MirrorMaker 1, we should have a KIP to deprecate its use in >> 2.6 if >>>>>> we want to remove it in 3.0. I don't have a good sense of how >>>> practical it >>>>>> is to deprecate this now, so I will defer to others here. But the >> KIP >>>>>> freeze for 2.6 is coming soon, so if we want to make the case, now >> is >>>> the >>>>>> time. >>>>>>> >>>>>>> best, >>>>>>> Colin >>>>>>> >>>>>>> >>>>>>>> On Thu, May 7, 2020, at 16:28, Guozhang Wang wrote: >>>>>>>> Hey folks, >>>>>>>> >>>>>>>> Sorry for stating that the bridge release would not break any >>>>>> compatibility >>>>>>>> before, which is incorrect and confused many people. >>>>>>>> >>>>>>>> I think one way to think about the versioning is that: >>>>>>>> >>>>>>>> 0) In a 2.x version moving ahead we would deprecate the >> ZK-dependent >>>>>> tools >>>>>>>> such as --zookeeper flags from various scripts (KIP-555) >>>>>>>> >>>>>>>> 1) In 3.0 we would at least make one incompatible change for >> example >>>> to >>>>>>>> remove the deprecated ZK flags. >>>>>>>> >>>>>>>> 2) In a future major version (e.g. 4.0) we would drop ZK entirely, >>>>>>>> including usages such as security credentials / broker >> registration / >>>>>> etc >>>>>>>> which are via ZK today as well. >>>>>>>> >>>>>>>> Then for the bridge release(s), it can be any or all of 3.x. >>>>>>>> >>>>>>>> >>>>>>>> For 1), I'd love to add a few more incompatibility changes in 3.0 >> from >>>>>>>> Kafka Streams: we evolve Streams public APIs by deprecating and >> then >>>>>> remove >>>>>>>> in major releases, and since 2.0 we've accumulated quite a few >>>>>> deprecated >>>>>>>> APIs, and I can compile a list of KIPs that contain those if >> people >>>> are >>>>>>>> interested. >>>>>>>> >>>>>>>> >>>>>>>> Guozhang >>>>>>>> >>>>>>>> >>>>>>>>> On Thu, May 7, 2020 at 3:53 PM Colin McCabe <cmcc...@apache.org> >>>> wrote: >>>>>>>>> >>>>>>>>> On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote: >>>>>>>>>>> In fact, we know that the bridge release will involve at least >> one >>>>>>>>>>> incompatible change. We will need to drop support for the >>>>>> --zookeeper >>>>>>>>>>> flags in the command-line tools. >>>>>>>>>> >>>>>>>>>> If the bridge release(s) and the subsequent post-ZK release are >>>> _both_ >>>>>>>>>> breaking changes, I think we only have one option: the 3.x line >> are >>>>>> the >>>>>>>>>> bridge release(s), and ZK is removed in 4.0, as suggested by >> Andrew >>>>>>>>>> Schofield. >>>>>>>>>> >>>>>>>>>> Specifically: >>>>>>>>>> - in order to _remove_ (not merely deprecate) the --zookeeper >> args, >>>> we >>>>>>>>> will >>>>>>>>>> need a major release. >>>>>>>>>> - in oder to drop support for ZK entirely (e.g. break a bunch of >>>>>> external >>>>>>>>>> tooling like Cruise Control), we will need a major release. >>>>>>>>>> >>>>>>>>>> I count two major releases. >>>>>>>>> >>>>>>>>> Hi Ryanne, >>>>>>>>> >>>>>>>>> I agree that dropping ZK completely will need a new major release >>>> after >>>>>>>>> 3.0. I think that's OK and in keeping with how we've handled >>>>>> deprecation >>>>>>>>> and removal in the past. It's important for users to have a >> smooth >>>>>> upgrade >>>>>>>>> path. >>>>>>>>> >>>>>>>>> best, >>>>>>>>> Colin >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Ryanne >>>>>>>>>> >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> On Wed, May 6, 2020 at 10:52 PM Colin McCabe < >> cmcc...@apache.org> >>>>>> wrote: >>>>>>>>>> >>>>>>>>>>>> On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: >>>>>>>>>>>> Hey Colin, I think we should wait until after KIP-500's >> "bridge >>>>>>>>>>>> release" so there is a clean break from Zookeeper after 3.0. >> The >>>>>>>>>>>> bridge release by definition is an attempt to not break >> anything, >>>> so >>>>>>>>>>>> it theoretically doesn't warrant a major release. >>>>>>>>>>> >>>>>>>>>>> Hi Ryanne, >>>>>>>>>>> >>>>>>>>>>> I think it's important to clarify this a little bit. The >> bridge >>>>>>>>> release >>>>>>>>>>> (really, releases, plural) allow you to upgrade from a cluster >> that >>>>>> is >>>>>>>>>>> using ZooKeeper to one that is not using ZooKeeper. But, that >>>>>> doesn't >>>>>>>>>>> imply that the bridge release itself doesn't break anything. >>>>>> Upgrading >>>>>>>>>>> to the bridge release itself might involve some minor >>>>>> incompatibility. >>>>>>>>>>> >>>>>>>>>>> Kafka does occasionally have incompatible changes. In those >> cases, >>>>>> we >>>>>>>>>>> bump the major version number. One example is that when we >> went >>>> from >>>>>>>>>>> Kafka 1.x to Kafka 2.0, we dropped support for JDK7. This is >> an >>>>>>>>>>> incompatible change. >>>>>>>>>>> >>>>>>>>>>> In fact, we know that the bridge release will involve at least >> one >>>>>>>>>>> incompatible change. We will need to drop support for the >>>>>> --zookeeper >>>>>>>>>>> flags in the command-line tools. >>>>>>>>>>> >>>>>>>>>>> We've been preparing for this change for a long time. People >> have >>>>>>>>> spent >>>>>>>>>>> a lot of effort designing new APIs that can be used instead of >> the >>>>>> old >>>>>>>>>>> zookeeper-based code that some of the command-line tools >> used. We >>>>>> have >>>>>>>>>>> also deprecated the old ZK-based flags. But at the end of the >> day, >>>>>> it >>>>>>>>>>> is still an incompatible change. So it's unfortunately not >>>> possible >>>>>>>>> for >>>>>>>>>>> the >>>>>>>>>>> bridge release to be a 2.x release. >>>>>>>>>>> >>>>>>>>>>>> If that's not the case (i.e. if a single "bridge release" >> turns >>>> out >>>>>>>>> to >>>>>>>>>>>> be impractical), we should consider forking 3.0 while >> maintaining >>>> a >>>>>>>>>>>> line of Zookeeper-dependent Kafka in 2.x. That way 3.x can >> evolve >>>>>>>>>>>> dramatically without breaking the 2.x line. In particular, >>>> anything >>>>>>>>>>>> related to removing Zookeeper could land in pre-3.0 while >> every >>>>>> other >>>>>>>>>>>> feature targets 2.6. >>>>>>>>>>> >>>>>>>>>>> Just to be super clear about this, what we want to do here is >>>> support >>>>>>>>>>> operating in __either__ KIP-500 mode and legacy mode for a >> while. >>>> So >>>>>>>>> the >>>>>>>>>>> same branch will have support for both the old way and the new >> way >>>> of >>>>>>>>>>> managing metadata. >>>>>>>>>>> >>>>>>>>>>> This will allow us to get an "alpha" version of the KIP-500 >> mode >>>> out >>>>>>>>> early >>>>>>>>>>> for people to experiment with. It also greatly reduces the >> number >>>> of >>>>>>>>> Kafka >>>>>>>>>>> releases we have to make, and the amount of backporting we >> have to >>>>>> do. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> If you are proposing 2.6 should be the "bridge release", I >> think >>>>>> this >>>>>>>>>>>> is premature given Kafka's time-based release schedule. If the >>>>>> bridge >>>>>>>>>>>> features happen to be merged before 2.6's feature freeze, then >>>> sure >>>>>>>>> -- >>>>>>>>>>>> let's make that the bridge release in retrospect. And if we >> get >>>> all >>>>>>>>>>>> the post-Zookeeper features merged before 2.7, I'm onboard >> with >>>>>>>>> naming >>>>>>>>>>>> it "3.0" instead. >>>>>>>>>>>> >>>>>>>>>>>> That said, we should aim to remove legacy MirrorMaker before >> 3.0 >>>> as >>>>>>>>>>>> well. I'm happy to drive that additional breaking change. >> Maybe >>>> 2.6 >>>>>>>>>>>> can be the "bridge" for MM2 as well. >>>>>>>>>>> >>>>>>>>>>> I don't have a strong opinion either way about this, but if we >> want >>>>>> to >>>>>>>>>>> remove the original MirrorMaker, we have to deprecate it first, >>>>>>>>> right? Are >>>>>>>>>>> we ready to do that? >>>>>>>>>>> >>>>>>>>>>> best, >>>>>>>>>>> Colin >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Ryanne >>>>>>>>>>>> >>>>>>>>>>>> On Mon, May 4, 2020, 5:05 PM Colin McCabe <cmcc...@apache.org >>> >>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi all, >>>>>>>>>>>>> >>>>>>>>>>>>> We've had a few proposals recently for incompatible >> changes. One >>>>>>>>> of >>>>>>>>>>>>> them is my KIP-604: Remove ZooKeeper Flags from the >>>> Administrative >>>>>>>>>>>>> Tools. The other is Boyang's KIP-590: Redirect ZK Mutation >>>>>>>>>>>>> Protocols to the Controller. I think it's time to start >> thinking >>>>>>>>>>>>> about Kafka 3.0. Specifically, I think we should move to 3.0 >>>> after >>>>>>>>>>>>> the 2.6 release. >>>>>>>>>>>>> >>>>>>>>>>>>> From the perspective of KIP-500, in Kafka 3.x we'd like to >> make >>>>>>>>>>>>> running in a ZooKeeper-less mode possible (but not yet the >>>>>>>>> default.) >>>>>>>>>>>>> This is the motivation behind KIP-590 and KIP-604, as well as >>>> some >>>>>>>>>>>>> of the other KIPs we've done recently. Since it will take >> some >>>>>>>>> time >>>>>>>>>>>>> to stabilize the new ZooKeeper-free Kafka code, we will hide >> it >>>>>>>>>>>>> behind an option initially. (We'll have a KIP describing >> this all >>>>>>>>> in >>>>>>>>>>>>> detail soon.) >>>>>>>>>>>>> >>>>>>>>>>>>> What does everyone think about having Kafka 3.0 come up next >>>> after >>>>>>>>>>>>> 2.6? Are there any other things we should change in the 2.6 >> -> >>>> 3.0 >>>>>>>>>>>>> transition? >>>>>>>>>>>>> >>>>>>>>>>>>> best, Colin >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> -- Guozhang >>>>>>>> >>>>>> >>>>>> >>>> >>> >> >
signature.asc
Description: OpenPGP digital signature
