Hi Nicolas, Did you follow the upgrade steps here?
https://kafka.apache.org/documentation/#upgrade_2_8_1 Also, the recommended Zookeeper versions for 2.0 (3.4.13) is different from that of 2.8 (3.5.9) https://github.com/apache/kafka/blob/2.0/gradle/dependencies.gradle#L87 https://github.com/apache/kafka/blob/2.8/gradle/dependencies.gradle#L122 It looks like your Zookeeper quorum was not in sync before the upgrade. 2 of them seem to agree with the topic id but 1 is different. If this is production, you may have to decide which one to keep and then you may need to upgrade Zookeeper first make sure they are all in sync with your approved data, then you can bring Kafka online. I generally recommend to back up your cluster data (in remote storage) and config data in version control before any upgrades so that you can always restore the state if all does not go well. This could be due to Zookeeper version incompatibility or bad state of data in Zookeeper before the upgrade. Israel Ekpo Lead Instructor, IzzyAcademy.com https://www.youtube.com/c/izzyacademy https://izzyacademy.com/ On Tue, Jan 25, 2022 at 7:21 AM Nicolas Carlot <nicolas.car...@chronopost.fr.invalid> wrote: > Hello everyone, > > I just had a major failure while upgrading a kafka cluster from 2.0 to > 2.8.1 following the provided migration process. > I understand that a topicId is now given to each topic within zookeeper and > meta.properties of each partition. > While describing the topic, it seems I have different topicId depending on > the zk node i'm querying: > > [kafkaadm@lyn3e154(PFI):~ 13:16:28]$ > /opt/java/j2ee/kafka/bin/kafka-topics.sh --zookeeper satezookeeperi1:62181 > --describe --topic PARCEL360.LT > Topic: PARCEL360.LT TopicId: XjIuCqy2TcKu-M5smrz9iA PartitionCount: 10 > ReplicationFactor: 3 Configs: compression.type=lz4 > Topic: PARCEL360.LT Partition: 0 Leader: 3 Replicas: > 1,2,3 Isr: 3 > Topic: PARCEL360.LT Partition: 1 Leader: 3 Replicas: > 2,3,1 Isr: 3 > Topic: PARCEL360.LT Partition: 2 Leader: 3 Replicas: > 3,1,2 Isr: 3 > Topic: PARCEL360.LT Partition: 3 Leader: 3 Replicas: > 1,2,3 Isr: 3 > Topic: PARCEL360.LT Partition: 4 Leader: 3 Replicas: > 2,3,1 Isr: 3 > Topic: PARCEL360.LT Partition: 5 Leader: 3 Replicas: > 3,1,2 Isr: 3 > Topic: PARCEL360.LT Partition: 6 Leader: 3 Replicas: > 1,2,3 Isr: 3 > Topic: PARCEL360.LT Partition: 7 Leader: 3 Replicas: > 2,3,1 Isr: 3 > Topic: PARCEL360.LT Partition: 8 Leader: 3 Replicas: > 3,1,2 Isr: 3 > Topic: PARCEL360.LT Partition: 9 Leader: 3 Replicas: > 1,2,3 Isr: 3 > [kafkaadm@lyn3e154(PFI):~ 13:17:06]$ > /opt/java/j2ee/kafka/bin/kafka-topics.sh --zookeeper satezookeeperi2:62181 > --describe --topic PARCEL360.LT > Topic: PARCEL360.LT TopicId: zwbQDd9NRjGwq-v2twHfIQ PartitionCount: 10 > ReplicationFactor: 3 Configs: compression.type=lz4 > Topic: PARCEL360.LT Partition: 0 Leader: 3 Replicas: > 1,2,3 Isr: 3 > Topic: PARCEL360.LT Partition: 1 Leader: 3 Replicas: > 2,3,1 Isr: 3 > Topic: PARCEL360.LT Partition: 2 Leader: 3 Replicas: > 3,1,2 Isr: 3 > Topic: PARCEL360.LT Partition: 3 Leader: 3 Replicas: > 1,2,3 Isr: 3 > Topic: PARCEL360.LT Partition: 4 Leader: 3 Replicas: > 2,3,1 Isr: 3 > Topic: PARCEL360.LT Partition: 5 Leader: 3 Replicas: > 3,1,2 Isr: 3 > Topic: PARCEL360.LT Partition: 6 Leader: 3 Replicas: > 1,2,3 Isr: 3 > Topic: PARCEL360.LT Partition: 7 Leader: 3 Replicas: > 2,3,1 Isr: 3 > Topic: PARCEL360.LT Partition: 8 Leader: 3 Replicas: > 3,1,2 Isr: 3 > Topic: PARCEL360.LT Partition: 9 Leader: 3 Replicas: > 1,2,3 Isr: 3 > [kafkaadm@lyn3e154(PFI):~ 13:17:11]$ > /opt/java/j2ee/kafka/bin/kafka-topics.sh --zookeeper satezookeeperi3:62181 > --describe --topic PARCEL360.LT > Topic: PARCEL360.LT TopicId: XjIuCqy2TcKu-M5smrz9iA PartitionCount: 10 > ReplicationFactor: 3 Configs: compression.type=lz4 > Topic: PARCEL360.LT Partition: 0 Leader: 3 Replicas: > 1,2,3 Isr: 3 > Topic: PARCEL360.LT Partition: 1 Leader: 3 Replicas: > 2,3,1 Isr: 3 > Topic: PARCEL360.LT Partition: 2 Leader: 3 Replicas: > 3,1,2 Isr: 3 > Topic: PARCEL360.LT Partition: 3 Leader: 3 Replicas: > 1,2,3 Isr: 3 > Topic: PARCEL360.LT Partition: 4 Leader: 3 Replicas: > 2,3,1 Isr: 3 > Topic: PARCEL360.LT Partition: 5 Leader: 3 Replicas: > 3,1,2 Isr: 3 > Topic: PARCEL360.LT Partition: 6 Leader: 3 Replicas: > 1,2,3 Isr: 3 > Topic: PARCEL360.LT Partition: 7 Leader: 3 Replicas: > 2,3,1 Isr: 3 > Topic: PARCEL360.LT Partition: 8 Leader: 3 Replicas: > 3,1,2 Isr: 3 > Topic: PARCEL360.LT Partition: 9 Leader: 3 Replicas: > 1,2,3 Isr: 3 > > > Any idea what's happening here ? > > > > > -- > <https://www.chronopost.fr/fr?xtatc=INT-149> > > > *Nicolas Carlot* > *Lead dev*Direction des Systèmes d'Information > > > 3 boulevard Romain Rolland > 75014 Paris > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ> > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ> >
