On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote: > > In fact, we know that the bridge release will involve at least one > > incompatible change. We will need to drop support for the --zookeeper > > flags in the command-line tools. > > If the bridge release(s) and the subsequent post-ZK release are _both_ > breaking changes, I think we only have one option: the 3.x line are the > bridge release(s), and ZK is removed in 4.0, as suggested by Andrew > Schofield. > > Specifically: > - in order to _remove_ (not merely deprecate) the --zookeeper args, we will > need a major release. > - in oder to drop support for ZK entirely (e.g. break a bunch of external > tooling like Cruise Control), we will need a major release. > > I count two major releases.
Hi Ryanne, I agree that dropping ZK completely will need a new major release after 3.0. I think that's OK and in keeping with how we've handled deprecation and removal in the past. It's important for users to have a smooth upgrade path. best, Colin > > Ryanne > > - > > On Wed, May 6, 2020 at 10:52 PM Colin McCabe <cmcc...@apache.org> wrote: > > > On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: > > > Hey Colin, I think we should wait until after KIP-500's "bridge > > > release" so there is a clean break from Zookeeper after 3.0. The > > > bridge release by definition is an attempt to not break anything, so > > > it theoretically doesn't warrant a major release. > > > > Hi Ryanne, > > > > I think it's important to clarify this a little bit. The bridge release > > (really, releases, plural) allow you to upgrade from a cluster that is > > using ZooKeeper to one that is not using ZooKeeper. But, that doesn't > > imply that the bridge release itself doesn't break anything. Upgrading > > to the bridge release itself might involve some minor incompatibility. > > > > Kafka does occasionally have incompatible changes. In those cases, we > > bump the major version number. One example is that when we went from > > Kafka 1.x to Kafka 2.0, we dropped support for JDK7. This is an > > incompatible change. > > > > In fact, we know that the bridge release will involve at least one > > incompatible change. We will need to drop support for the --zookeeper > > flags in the command-line tools. > > > > We've been preparing for this change for a long time. People have spent > > a lot of effort designing new APIs that can be used instead of the old > > zookeeper-based code that some of the command-line tools used. We have > > also deprecated the old ZK-based flags. But at the end of the day, it > > is still an incompatible change. So it's unfortunately not possible for > > the > > bridge release to be a 2.x release. > > > > > If that's not the case (i.e. if a single "bridge release" turns out to > > > be impractical), we should consider forking 3.0 while maintaining a > > > line of Zookeeper-dependent Kafka in 2.x. That way 3.x can evolve > > > dramatically without breaking the 2.x line. In particular, anything > > > related to removing Zookeeper could land in pre-3.0 while every other > > > feature targets 2.6. > > > > Just to be super clear about this, what we want to do here is support > > operating in __either__ KIP-500 mode and legacy mode for a while. So the > > same branch will have support for both the old way and the new way of > > managing metadata. > > > > This will allow us to get an "alpha" version of the KIP-500 mode out early > > for people to experiment with. It also greatly reduces the number of Kafka > > releases we have to make, and the amount of backporting we have to do. > > > > > > > > If you are proposing 2.6 should be the "bridge release", I think this > > > is premature given Kafka's time-based release schedule. If the bridge > > > features happen to be merged before 2.6's feature freeze, then sure -- > > > let's make that the bridge release in retrospect. And if we get all > > > the post-Zookeeper features merged before 2.7, I'm onboard with naming > > > it "3.0" instead. > > > > > > That said, we should aim to remove legacy MirrorMaker before 3.0 as > > > well. I'm happy to drive that additional breaking change. Maybe 2.6 > > > can be the "bridge" for MM2 as well. > > > > I don't have a strong opinion either way about this, but if we want to > > remove the original MirrorMaker, we have to deprecate it first, right? Are > > we ready to do that? > > > > best, > > Colin > > > > > > > > Ryanne > > > > > > On Mon, May 4, 2020, 5:05 PM Colin McCabe <cmcc...@apache.org> wrote: > > > > > > > Hi all, > > > > > > > > We've had a few proposals recently for incompatible changes. One of > > > > them is my KIP-604: Remove ZooKeeper Flags from the Administrative > > > > Tools. The other is Boyang's KIP-590: Redirect ZK Mutation > > > > Protocols to the Controller. I think it's time to start thinking > > > > about Kafka 3.0. Specifically, I think we should move to 3.0 after > > > > the 2.6 release. > > > > > > > > From the perspective of KIP-500, in Kafka 3.x we'd like to make > > > > running in a ZooKeeper-less mode possible (but not yet the default.) > > > > This is the motivation behind KIP-590 and KIP-604, as well as some > > > > of the other KIPs we've done recently. Since it will take some time > > > > to stabilize the new ZooKeeper-free Kafka code, we will hide it > > > > behind an option initially. (We'll have a KIP describing this all in > > > > detail soon.) > > > > > > > > What does everyone think about having Kafka 3.0 come up next after > > > > 2.6? Are there any other things we should change in the 2.6 -> 3.0 > > > > transition? > > > > > > > > best, Colin > > > > > > > > > >
