Hi Israel, I followed these steps but instead of setting inter.broker.protocol.version to 2.8 in step 3 I just removed it. Dunno if that changes anything ? Regarding zookeeper we're using 3.5.8 Luckily this was a pre-production cluster, we were testing the upgrade here and we do have backups.
Anyway to ensute that all zk nodes are in sync when performing the upgrade ? I fail to understand how this can lead to such situation. Shouldn't the unsync zk node data be discared automatically ? Le mar. 25 janv. 2022 à 15:32, Israel Ekpo <israele...@gmail.com> a écrit : > Hi Nicolas, > > Did you follow the upgrade steps here? > > https://kafka.apache.org/documentation/#upgrade_2_8_1 > > Also, the recommended Zookeeper versions for 2.0 (3.4.13) is different from > that of 2.8 (3.5.9) > > https://github.com/apache/kafka/blob/2.0/gradle/dependencies.gradle#L87 > https://github.com/apache/kafka/blob/2.8/gradle/dependencies.gradle#L122 > > It looks like your Zookeeper quorum was not in sync before the upgrade. 2 > of them seem to agree with the topic id but 1 is different. > > If this is production, you may have to decide which one to keep and then > you may need to upgrade Zookeeper first make sure they are all in sync with > your approved data, then you can bring Kafka online. > > I generally recommend to back up your cluster data (in remote storage) and > config data in version control before any upgrades so that you can always > restore the state if all does not go well. > > This could be due to Zookeeper version incompatibility or bad state of data > in Zookeeper before the upgrade. > > Israel Ekpo > Lead Instructor, IzzyAcademy.com > https://www.youtube.com/c/izzyacademy > https://izzyacademy.com/ > > > On Tue, Jan 25, 2022 at 7:21 AM Nicolas Carlot > <nicolas.car...@chronopost.fr.invalid> wrote: > > > Hello everyone, > > > > I just had a major failure while upgrading a kafka cluster from 2.0 to > > 2.8.1 following the provided migration process. > > I understand that a topicId is now given to each topic within zookeeper > and > > meta.properties of each partition. > > While describing the topic, it seems I have different topicId depending > on > > the zk node i'm querying: > > > > [kafkaadm@lyn3e154(PFI):~ 13:16:28]$ > > /opt/java/j2ee/kafka/bin/kafka-topics.sh --zookeeper > satezookeeperi1:62181 > > --describe --topic PARCEL360.LT > > Topic: PARCEL360.LT TopicId: XjIuCqy2TcKu-M5smrz9iA PartitionCount: > 10 > > ReplicationFactor: 3 Configs: compression.type=lz4 > > Topic: PARCEL360.LT Partition: 0 Leader: 3 > Replicas: > > 1,2,3 Isr: 3 > > Topic: PARCEL360.LT Partition: 1 Leader: 3 > Replicas: > > 2,3,1 Isr: 3 > > Topic: PARCEL360.LT Partition: 2 Leader: 3 > Replicas: > > 3,1,2 Isr: 3 > > Topic: PARCEL360.LT Partition: 3 Leader: 3 > Replicas: > > 1,2,3 Isr: 3 > > Topic: PARCEL360.LT Partition: 4 Leader: 3 > Replicas: > > 2,3,1 Isr: 3 > > Topic: PARCEL360.LT Partition: 5 Leader: 3 > Replicas: > > 3,1,2 Isr: 3 > > Topic: PARCEL360.LT Partition: 6 Leader: 3 > Replicas: > > 1,2,3 Isr: 3 > > Topic: PARCEL360.LT Partition: 7 Leader: 3 > Replicas: > > 2,3,1 Isr: 3 > > Topic: PARCEL360.LT Partition: 8 Leader: 3 > Replicas: > > 3,1,2 Isr: 3 > > Topic: PARCEL360.LT Partition: 9 Leader: 3 > Replicas: > > 1,2,3 Isr: 3 > > [kafkaadm@lyn3e154(PFI):~ 13:17:06]$ > > /opt/java/j2ee/kafka/bin/kafka-topics.sh --zookeeper > satezookeeperi2:62181 > > --describe --topic PARCEL360.LT > > Topic: PARCEL360.LT TopicId: zwbQDd9NRjGwq-v2twHfIQ PartitionCount: > 10 > > ReplicationFactor: 3 Configs: compression.type=lz4 > > Topic: PARCEL360.LT Partition: 0 Leader: 3 > Replicas: > > 1,2,3 Isr: 3 > > Topic: PARCEL360.LT Partition: 1 Leader: 3 > Replicas: > > 2,3,1 Isr: 3 > > Topic: PARCEL360.LT Partition: 2 Leader: 3 > Replicas: > > 3,1,2 Isr: 3 > > Topic: PARCEL360.LT Partition: 3 Leader: 3 > Replicas: > > 1,2,3 Isr: 3 > > Topic: PARCEL360.LT Partition: 4 Leader: 3 > Replicas: > > 2,3,1 Isr: 3 > > Topic: PARCEL360.LT Partition: 5 Leader: 3 > Replicas: > > 3,1,2 Isr: 3 > > Topic: PARCEL360.LT Partition: 6 Leader: 3 > Replicas: > > 1,2,3 Isr: 3 > > Topic: PARCEL360.LT Partition: 7 Leader: 3 > Replicas: > > 2,3,1 Isr: 3 > > Topic: PARCEL360.LT Partition: 8 Leader: 3 > Replicas: > > 3,1,2 Isr: 3 > > Topic: PARCEL360.LT Partition: 9 Leader: 3 > Replicas: > > 1,2,3 Isr: 3 > > [kafkaadm@lyn3e154(PFI):~ 13:17:11]$ > > /opt/java/j2ee/kafka/bin/kafka-topics.sh --zookeeper > satezookeeperi3:62181 > > --describe --topic PARCEL360.LT > > Topic: PARCEL360.LT TopicId: XjIuCqy2TcKu-M5smrz9iA PartitionCount: > 10 > > ReplicationFactor: 3 Configs: compression.type=lz4 > > Topic: PARCEL360.LT Partition: 0 Leader: 3 > Replicas: > > 1,2,3 Isr: 3 > > Topic: PARCEL360.LT Partition: 1 Leader: 3 > Replicas: > > 2,3,1 Isr: 3 > > Topic: PARCEL360.LT Partition: 2 Leader: 3 > Replicas: > > 3,1,2 Isr: 3 > > Topic: PARCEL360.LT Partition: 3 Leader: 3 > Replicas: > > 1,2,3 Isr: 3 > > Topic: PARCEL360.LT Partition: 4 Leader: 3 > Replicas: > > 2,3,1 Isr: 3 > > Topic: PARCEL360.LT Partition: 5 Leader: 3 > Replicas: > > 3,1,2 Isr: 3 > > Topic: PARCEL360.LT Partition: 6 Leader: 3 > Replicas: > > 1,2,3 Isr: 3 > > Topic: PARCEL360.LT Partition: 7 Leader: 3 > Replicas: > > 2,3,1 Isr: 3 > > Topic: PARCEL360.LT Partition: 8 Leader: 3 > Replicas: > > 3,1,2 Isr: 3 > > Topic: PARCEL360.LT Partition: 9 Leader: 3 > Replicas: > > 1,2,3 Isr: 3 > > > > > > Any idea what's happening here ? > > > > > > > > > > -- > > <https://www.chronopost.fr/fr?xtatc=INT-149> > > > > > > *Nicolas Carlot* > > *Lead dev*Direction des Systèmes d'Information > > > > > > 3 boulevard Romain Rolland > > 75014 Paris > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ> > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ> > > > -- <https://www.chronopost.fr/fr?xtatc=INT-149> *Nicolas Carlot* *Lead dev*Direction des Systèmes d'Information 3 boulevard Romain Rolland 75014 Paris <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
