On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: > Hey Colin, I think we should wait until after KIP-500's "bridge > release" so there is a clean break from Zookeeper after 3.0. The > bridge release by definition is an attempt to not break anything, so > it theoretically doesn't warrant a major release.
Hi Ryanne, I think it's important to clarify this a little bit. The bridge release (really, releases, plural) allow you to upgrade from a cluster that is using ZooKeeper to one that is not using ZooKeeper. But, that doesn't imply that the bridge release itself doesn't break anything. Upgrading to the bridge release itself might involve some minor incompatibility. Kafka does occasionally have incompatible changes. In those cases, we bump the major version number. One example is that when we went from Kafka 1.x to Kafka 2.0, we dropped support for JDK7. This is an incompatible change. In fact, we know that the bridge release will involve at least one incompatible change. We will need to drop support for the --zookeeper flags in the command-line tools. We've been preparing for this change for a long time. People have spent a lot of effort designing new APIs that can be used instead of the old zookeeper-based code that some of the command-line tools used. We have also deprecated the old ZK-based flags. But at the end of the day, it is still an incompatible change. So it's unfortunately not possible for the bridge release to be a 2.x release. > If that's not the case (i.e. if a single "bridge release" turns out to > be impractical), we should consider forking 3.0 while maintaining a > line of Zookeeper-dependent Kafka in 2.x. That way 3.x can evolve > dramatically without breaking the 2.x line. In particular, anything > related to removing Zookeeper could land in pre-3.0 while every other > feature targets 2.6. Just to be super clear about this, what we want to do here is support operating in __either__ KIP-500 mode and legacy mode for a while. So the same branch will have support for both the old way and the new way of managing metadata. This will allow us to get an "alpha" version of the KIP-500 mode out early for people to experiment with. It also greatly reduces the number of Kafka releases we have to make, and the amount of backporting we have to do. > > If you are proposing 2.6 should be the "bridge release", I think this > is premature given Kafka's time-based release schedule. If the bridge > features happen to be merged before 2.6's feature freeze, then sure -- > let's make that the bridge release in retrospect. And if we get all > the post-Zookeeper features merged before 2.7, I'm onboard with naming > it "3.0" instead. > > That said, we should aim to remove legacy MirrorMaker before 3.0 as > well. I'm happy to drive that additional breaking change. Maybe 2.6 > can be the "bridge" for MM2 as well. I don't have a strong opinion either way about this, but if we want to remove the original MirrorMaker, we have to deprecate it first, right? Are we ready to do that? best, Colin > > Ryanne > > On Mon, May 4, 2020, 5:05 PM Colin McCabe <cmcc...@apache.org> wrote: > > > Hi all, > > > > We've had a few proposals recently for incompatible changes. One of > > them is my KIP-604: Remove ZooKeeper Flags from the Administrative > > Tools. The other is Boyang's KIP-590: Redirect ZK Mutation > > Protocols to the Controller. I think it's time to start thinking > > about Kafka 3.0. Specifically, I think we should move to 3.0 after > > the 2.6 release. > > > > From the perspective of KIP-500, in Kafka 3.x we'd like to make > > running in a ZooKeeper-less mode possible (but not yet the default.) > > This is the motivation behind KIP-590 and KIP-604, as well as some > > of the other KIPs we've done recently. Since it will take some time > > to stabilize the new ZooKeeper-free Kafka code, we will hide it > > behind an option initially. (We'll have a KIP describing this all in > > detail soon.) > > > > What does everyone think about having Kafka 3.0 come up next after > > 2.6? Are there any other things we should change in the 2.6 -> 3.0 > > transition? > > > > best, Colin > > >
