Hi all, I am working on upgrading the Kafka cluster from the version 2.0.1 to 3.5.2. The main concern in that update is passing version 2.1 which has the following note in the upgrade document: ``` Note that 2.1.x contains a change to the internal schema used to store consumer offsets. Once the upgrade is complete, it will not be possible to downgrade to previous versions. ``` In the rolling upgrade section the documentation says the following: ``` Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version. ``` So, as soon as inter.broker.protocol.version is set to 2.1+ you should not be able to rollback to the old version. I did reproduce the issue with the rollback with the following steps: 1. Start cluster with Kafka 2.0.1 binaries and inter.broker.protocol.version=2.0. Produce some messages in the test topic 2. Restart cluster with Kafka 3.5.2 binaries and inter.broker.protocol.version=3.5. Consume messages. 3. Restart cluster once again with Kafka 2.0.1 binaries and inter.broker.protocol.version=2.0. During start you will get error like this: ``` ERROR [GroupMetadataManager brokerId=0] Error loading offsets from __consumer_offsets-43 (kafka.coordinator.group.GroupMetadataManager) org.apache.kafka.common.KafkaException: Unknown group metadata version 3 ```
The interesting thing is that this doesn't happen if I rollback inter.broker.protocol.version to 2.0 using Kafka 3.5.2 binaries. Cluster is able to start and read all the data from the __consumer_offsets partitions with metadata version 3. The question is that expected behavior? Below are the steps that I have used: 1. Start cluster with Kafka 2.0.1 binaries and inter.broker.protocol.version=2.0. Produce some messages in the test topic 2. Restart cluster with Kafka 3.5.2 binaries and inter.broker.protocol.version=3.5. Consume messages. 3. Restart cluster once again with Kafka 3.5.2 binaries and inter.broker.protocol.version=2.0. Cluster starts properly and is able to read partitions from the __consumer_offsets topic. I did double check Kafka startup logs and make sure that it uses configured inter.broker.protocol.version versions. if this is okay, then I guess that can be used as a rollback in case of problems after switching to the inter.broker.protocol.version 3.X Thanks in advance, Dmytro
