Hello Colin, Thank you for the response!
1. I have attached the compatibility matrix in the KIP under the section Compatibility, Deprecation, and Migration Plan. 2. I believe the answer to how many bridge releases (for Kafka) will be needed to upgrade from 2.0 to 4.0 based on the compatibility matrix is 2 - one from 2.0 to any of 2.4.x to 3.5.x (now that we are no longer considering this KIP for 3.5.x) and one from that version to the bridge release mentioned in KIP-500 and KIP-866 (assuming that bridge release has a dependency on Zookeeper 3.8.1). 3. What determines whether you need your Zookeeper cluster to first be upgraded to 3.4.6 is "Upgrading a running ZooKeeper ensemble to 3.5.0 should be done only after upgrading your ensemble to the 3.4.6 release." (source: https://zookeeper.apache.org/doc/r3.8.1/zookeeperReconfig.html#ch_reconfig_upgrade). Continuing from the example in point 2, since Kafka 2.0 had Zookeeper 3.4.13 no second bridge upgrade for Zookeeper is needed. To clarify, you would go from Zookeeper 3.4.13 to any between 3.5.x and 3.6.x and then to 3.8.1. 4. Ideally, if users make an error since they will be carrying out a rolling restart of their Zookeeper cluster the errors should start appearing with the first Zookeeper instance which is rebooted, so if they have sufficient monitoring in place they should be able to catch it before it takes down their whole Kafka cluster. To be honest, I have never had to downgrade a Zookeeper cluster, but I suspect the procedure is the same as upgrading but in reverse i.e. stop the new binary, remove the new binary, put the old binary, start the old binary. 5. Fair point, I meant to say that Zookeeper will no longer be a thing when Kafka 4.0 arrives. 6. I believe Ismael's response answers your last concern better than I could. Best, Christo On Mon, 10 Apr 2023 at 00:53, Colin McCabe <cmcc...@apache.org> wrote: > On Wed, Mar 15, 2023, at 04:58, Christo Lolov wrote: > > Hello Colin, > > > > Thank you for taking the time to review the proposal! > > > > I have attached a compatibility matrix to aid the explanation below - if > the mailing system rejects it I will find another way to share it. > > Hi Christo, > > The mailing list doesn't take attachments. So perhaps you could share this > in textual form? > > > For the avoidance of doubt, I am not proposing to drop support for > rolling upgrade from old Kafka versions to new ones. What I am saying is > that additional care will need to be taken when upgrading to the latest > Kafka versions depending on the version of the accompanying Zookeeper > cluster. This additional care means one might have to upgrade to a Kafka > version which falls in the intersection of the two sets in the accompanying > diagram before upgrading the accompanying Zookeeper cluster. > > I think we are talking about the same thing, just using different > terminology. If you have to go through multiple upgrades to get from > version X to version Y, I would not say there is "support for rolling > upgrade from X to Y." In particular if you have to go through some other > version B, I would say that B is the "bridge release." > > This is different than having an "upgrade path" -- I think everyone agrees > that there should be an upgrade path between any two kafka versions (well, > ones that are 0.8 or newer, at least). > > So I'd like to understand what the bridge release would be for this kind > of change, and how many "hops" would be required to get from, say, 2.0 to > 4.0. Keeping in mind that 4.0 won't have ZK at all. > > > As a concrete example let's say you want to upgrade to Kafka 3.5 from > Kafka 2.3 and Zookeeper 3.4. You will have to: > > 1. Carry out a rolling upgrade of your Kafka cluster to a version > between 2.4 and 3.4. > > 2. Carry out a rolling upgrade of your Zookeeper cluster to 3.8.1 (with > a possible stop at 3.4.6 due to > https://zookeeper.apache.org/doc/r3.8.1/zookeeperReconfig.html#ch_reconfig_upgrade > ). > > Hmm, what determines whether I have to make the stop or not? > > One thing we haven't discussed in this thread is that a lot of users don't > upgrade ZK when they do a Kafka upgrade. So I'd also like to understand in > what situations ZK upgrades would be required as part of Kafka upgrades, if > we bump this version. Also, what will happen if they forget? I assume the > cluster would be down for a while. Does ZK have a downgrade procedure? > > > 3. Carry out a rolling upgrade of your Kafka cluster from 3.4 to 3.5. > > > > It is true that Zookeeper is to be deprecated in Kafka 4.0, but as far > as I looked there is no concrete release date for that version yet. > > ZK is not going to be deprecated in AK 4.0, but removed in 4.0. > > > > > Until this is the case and unless we carry out a Zookeeper version > upgrade we leave users to run on an end-of-life version with unpatched CVEs > addressed in later versions. > > > > Some users have compliance requirements to only run on stable versions > of a software and its dependencies and not keeping the dependencies up to > date renders them unable to use Kafka. > > We are going to deprecate ZK mode soon. So if this is indeed a requirement > (no deprecated software in prod), perhaps those users will have to move to > KRaft mode. (Independently of what we decide here) > > best, > Colin > > > Please, let me know your thoughts on the matter! > > > > Best, > > Christo > > > > On Thu, 9 Mar 2023 at 21:56, Colin McCabe <cmcc...@apache.org> wrote: > >> Hi, > >> > >> I'm struggling a bit with this KIP, because dropping support for > rolling upgrades from old Kafka versions doesn't seem like something we > should do in a minor release. But on the other hand, the next Kafka release > won't have ZK at all. Maybe we should punt on this until and unless there > is a security issue that requires some action from us. > >> > >> I would also add, that a major ZK version bump is pretty risky. Last > time we did this we hit several bugs. I remember we hit one where there was > an incompatible change with regard to formatting (sorry, I can't seem to > find the JIRA right now). > >> > >> Sorry, but for now I have to vote -1 until I can understand this better > >> > >> best, > >> Colin > >> > >> > >> On Thu, Feb 23, 2023, at 06:48, Divij Vaidya wrote: > >> > Thanks for the KIP Christo. > >> > > >> > Having Zk 3.6 reach EOL in Dec 2022 is a good enough reason to > upgrade, > >> > hence I completely agree with the motivation. Your experiments have > >> > demonstrated that the new version of Zk is stable at scale and the > backward > >> > compatibility risks are acceptable since Apache Kafka 2.4.x is an EOL > >> > version. > >> > > >> > Vote +1 (non binding) > >> > > >> > -- > >> > Divij Vaidya > >> > > >> > > >> > > >> > On Thu, Feb 23, 2023 at 3:32 PM Christo Lolov <christolo...@gmail.com > > > >> > wrote: > >> > > >> >> Hello! > >> >> > >> >> I would like to start the vote for KIP-902, which upgrades Zookeeper > to > >> >> version 3.8.1. > >> >> > >> >> The KIP can be found at > >> >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784 > >> >> > >> >> The discussion thread is > >> >> https://lists.apache.org/thread/5jbn2x0rtmqz5scyoygbdbj4vo0mpbw1 > >> >> > >> >> Thanks > >> >> Christo >
