Hello Ismael, my installations of Zookeeper and Kafka are independent, before and after the upgrade. My previous situation was :
Zookeeper 3.4.13 and Kafka 2.12-2.1.0 I wanted to update to the latest versions (sounds not unusual) and had this plan : - Zookeeper 3.4.13 --> 3.5.6 - Kafka 2.12-2.1.0 --> 2.12-2.3.0 So on the broken cluster I run Zookeeper 3.5.6 with Kafka 2.12-2.3.0. Not sure to understand the question precisely. I start zookeeper from its /appl/zookeeper dir, and Kafka from /appl/kafka. I ran a ps and see this in the command line kafka: ........:/appl/kafka/bin/../libs/zookeeper-3.4.14.jar:......... so I guess Kafka uses its own zookeeper lib 3.4.14 and talks to the Zookeeper server which itself is running 3.5.6. Do you think it could be a reason for the issue ? I can try to downgrade ZK and see how it goes. Regs, Charles Le 05.12.19 à 17:33, Ismael Juma a écrit : > Are you using ZooKeeper 3.5.6 client libraries with Kafka 2.3.1? Kafka > 2.3.1 ships with ZooKeeper 3.4.x. > > Ismael > > On Thu, Dec 5, 2019 at 8:18 AM Charles Bueche <cbue...@gmail.com> wrote: > >> Hello all, >> >> I do have two Kafka clusters in action, test and prod. The two are formed >> by 3 nodes each, are independent and run their own zookeeper setups. My >> prod cluster is running fine. My test cluster is half-broken and I'm >> struggling to fix it. I could wipe it but I prefer to understand what's >> wrong and fix it. >> >> I'm not sure what broke my test cluster. I had several network >> disconnections / split-brains but Kafka always recovered fine. The reasons >> for the network issues are independent and still being investigated >> (layer-2 storms, etc). >> >> So I upgraded my zookeeper and kafka to the latest versions and when >> trying to rebalance a topic across brokers I started to notice the >> problems. Not sure really when they started, before or after the upgrade. >> >> I ran the upgrade as for the official doc (rolling upgrade, moving up >> inter.broker.protocol.version and log.message.format.version gradually). >> >> ------------------------------------------------------ >> cat /etc/os-release >> NAME="SLES" >> VERSION="12-SP4" >> VERSION_ID="12.4" >> PRETTY_NAME="SUSE Linux Enterprise Server 12 SP4" >> ID="sles" >> ANSI_COLOR="0;32" >> CPE_NAME="cpe:/o:suse:sles:12:sp4" >> >> ------------------------------------------------------ >> versions : >> - zookeeper 3.5.6 >> - kafka 2.12-2.3.1 >> >> ------------------------------------------------------ >> >> many rapid log entries on brokers 1 & 2 (we have 1, 2, 3) >> >> [2019-12-05 09:56:54,967] ERROR [KafkaApi-1] Number of alive brokers '0' >> does not meet the required replication factor '1' for the offsets topic >> (configured via 'offsets.topic.replication.factor'). This error can be >> ignored if the cluster is starting up and not all brokers are up yet. >> (kafka.server.KafkaApis) >> >> ------------------------------------------------------ >> java is java-1_8_0-openjdk-1.8.0.222-27.35.2.x86_64 from SLES. I have >> tried Oracle Java jdk1.8.0_231 with the same issue. >> >> ------------------------------------------------------ >> when trying to see a reassignment I have this very suspect error : >> >> root@vmgato701a01:/appl/kafka/bin # ./kafka-reassign-partitions.sh >> --zookeeper $ZKLIST --reassignment-json-file /tmp/r7.json --verify >> >> Status of partition reassignment: >> Partitions reassignment failed due to >> com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer; >> java.lang.AbstractMethodError: >> com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer; >> at >> com.fasterxml.jackson.databind.ext.OptionalHandlerFactory.findDeserializer(OptionalHandlerFactory.java:122) >> at >> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findOptionalStdDeserializer(BasicDeserializerFactory.java:1589) >> at >> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findDefaultDeserializer(BasicDeserializerFactory.java:1812) >> at >> com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.findStdDeserializer(BeanDeserializerFactory.java:161) >> at >> com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:125) >> at >> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411) >> at >> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349) >> at >> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264) >> at >> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) >> at >> com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) >> at >> com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477) >> at >> com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4178) >> at >> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997) >> at >> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3079) >> at kafka.utils.Json$.parseBytesAs(Json.scala:73) >> at kafka.zk.ReassignPartitionsZNode$.decode(ZkData.scala:407) >> at >> kafka.zk.KafkaZkClient.getPartitionReassignment(KafkaZkClient.scala:795) >> at >> kafka.admin.ReassignPartitionsCommand$.checkIfPartitionReassignmentSucceeded(ReassignPartitionsCommand.scala:355) >> at >> kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:97) >> at >> kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90) >> at >> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61) >> at >> kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala) >> >> Help appreciated. >>
