On Wed, May 6, 2020, at 21:40, Ryanne Dolan wrote: > > This will allow us to get an "alpha" version of the KIP-500 mode out > > early for people to experiment with > > I think this is a non-sequitur. It's not a requirement that KIP-500 be > merged to master and released in order for people to experiment with it. >
Hi Ryanne, I agree that it is not a requirement that KIP-500 be merged to master in order for people to experiment with it. The main reason for merging to master is to avoid maintaining lots of branches and doing lots of backports. > > Nor does it sound great to call for a major release (3.0) in order to get > an "alpha version ... out early". > As I said earlier, the reason for the new major release is to make certain incompatible changes, not in order to get an alpha version of KIP-500 out. For example, dropping the zookeeper flags is a step forward for security and encapsulation which also advances KIP-500. Another example is that removing the kafka-preferred-replica-election.sh command removes a duplicate command that has been deprecated for a while. best, Colin > > On Wed, May 6, 2020 at 10:52 PM Colin McCabe <cmcc...@apache.org> wrote: > > > On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: > > > Hey Colin, I think we should wait until after KIP-500's "bridge > > > release" so there is a clean break from Zookeeper after 3.0. The > > > bridge release by definition is an attempt to not break anything, so > > > it theoretically doesn't warrant a major release. > > > > Hi Ryanne, > > > > I think it's important to clarify this a little bit. The bridge release > > (really, releases, plural) allow you to upgrade from a cluster that is > > using ZooKeeper to one that is not using ZooKeeper. But, that doesn't > > imply that the bridge release itself doesn't break anything. Upgrading > > to the bridge release itself might involve some minor incompatibility. > > > > Kafka does occasionally have incompatible changes. In those cases, we > > bump the major version number. One example is that when we went from > > Kafka 1.x to Kafka 2.0, we dropped support for JDK7. This is an > > incompatible change. > > > > In fact, we know that the bridge release will involve at least one > > incompatible change. We will need to drop support for the --zookeeper > > flags in the command-line tools. > > > > We've been preparing for this change for a long time. People have spent > > a lot of effort designing new APIs that can be used instead of the old > > zookeeper-based code that some of the command-line tools used. We have > > also deprecated the old ZK-based flags. But at the end of the day, it > > is still an incompatible change. So it's unfortunately not possible for > > the > > bridge release to be a 2.x release. > > > > > If that's not the case (i.e. if a single "bridge release" turns out to > > > be impractical), we should consider forking 3.0 while maintaining a > > > line of Zookeeper-dependent Kafka in 2.x. That way 3.x can evolve > > > dramatically without breaking the 2.x line. In particular, anything > > > related to removing Zookeeper could land in pre-3.0 while every other > > > feature targets 2.6. > > > > Just to be super clear about this, what we want to do here is support > > operating in __either__ KIP-500 mode and legacy mode for a while. So the > > same branch will have support for both the old way and the new way of > > managing metadata. > > > > This will allow us to get an "alpha" version of the KIP-500 mode out early > > for people to experiment with. It also greatly reduces the number of Kafka > > releases we have to make, and the amount of backporting we have to do. > > > > > > > > If you are proposing 2.6 should be the "bridge release", I think this > > > is premature given Kafka's time-based release schedule. If the bridge > > > features happen to be merged before 2.6's feature freeze, then sure -- > > > let's make that the bridge release in retrospect. And if we get all > > > the post-Zookeeper features merged before 2.7, I'm onboard with naming > > > it "3.0" instead. > > > > > > That said, we should aim to remove legacy MirrorMaker before 3.0 as > > > well. I'm happy to drive that additional breaking change. Maybe 2.6 > > > can be the "bridge" for MM2 as well. > > > > I don't have a strong opinion either way about this, but if we want to > > remove the original MirrorMaker, we have to deprecate it first, right? Are > > we ready to do that? > > > > best, > > Colin > > > > > > > > Ryanne > > > > > > On Mon, May 4, 2020, 5:05 PM Colin McCabe <cmcc...@apache.org> wrote: > > > > > > > Hi all, > > > > > > > > We've had a few proposals recently for incompatible changes. One of > > > > them is my KIP-604: Remove ZooKeeper Flags from the Administrative > > > > Tools. The other is Boyang's KIP-590: Redirect ZK Mutation > > > > Protocols to the Controller. I think it's time to start thinking > > > > about Kafka 3.0. Specifically, I think we should move to 3.0 after > > > > the 2.6 release. > > > > > > > > From the perspective of KIP-500, in Kafka 3.x we'd like to make > > > > running in a ZooKeeper-less mode possible (but not yet the default.) > > > > This is the motivation behind KIP-590 and KIP-604, as well as some > > > > of the other KIPs we've done recently. Since it will take some time > > > > to stabilize the new ZooKeeper-free Kafka code, we will hide it > > > > behind an option initially. (We'll have a KIP describing this all in > > > > detail soon.) > > > > > > > > What does everyone think about having Kafka 3.0 come up next after > > > > 2.6? Are there any other things we should change in the 2.6 -> 3.0 > > > > transition? > > > > > > > > best, Colin > > > > > > > > > >
