Are you using ZooKeeper 3.5.6 client libraries with Kafka 2.3.1? Kafka 2.3.1 ships with ZooKeeper 3.4.x.
Ismael On Thu, Dec 5, 2019 at 8:18 AM Charles Bueche <cbue...@gmail.com> wrote: > Hello all, > > I do have two Kafka clusters in action, test and prod. The two are formed > by 3 nodes each, are independent and run their own zookeeper setups. My > prod cluster is running fine. My test cluster is half-broken and I'm > struggling to fix it. I could wipe it but I prefer to understand what's > wrong and fix it. > > I'm not sure what broke my test cluster. I had several network > disconnections / split-brains but Kafka always recovered fine. The reasons > for the network issues are independent and still being investigated > (layer-2 storms, etc). > > So I upgraded my zookeeper and kafka to the latest versions and when > trying to rebalance a topic across brokers I started to notice the > problems. Not sure really when they started, before or after the upgrade. > > I ran the upgrade as for the official doc (rolling upgrade, moving up > inter.broker.protocol.version and log.message.format.version gradually). > > ------------------------------------------------------ > cat /etc/os-release > NAME="SLES" > VERSION="12-SP4" > VERSION_ID="12.4" > PRETTY_NAME="SUSE Linux Enterprise Server 12 SP4" > ID="sles" > ANSI_COLOR="0;32" > CPE_NAME="cpe:/o:suse:sles:12:sp4" > > ------------------------------------------------------ > versions : > - zookeeper 3.5.6 > - kafka 2.12-2.3.1 > > ------------------------------------------------------ > > many rapid log entries on brokers 1 & 2 (we have 1, 2, 3) > > [2019-12-05 09:56:54,967] ERROR [KafkaApi-1] Number of alive brokers '0' > does not meet the required replication factor '1' for the offsets topic > (configured via 'offsets.topic.replication.factor'). This error can be > ignored if the cluster is starting up and not all brokers are up yet. > (kafka.server.KafkaApis) > > ------------------------------------------------------ > java is java-1_8_0-openjdk-1.8.0.222-27.35.2.x86_64 from SLES. I have > tried Oracle Java jdk1.8.0_231 with the same issue. > > ------------------------------------------------------ > when trying to see a reassignment I have this very suspect error : > > root@vmgato701a01:/appl/kafka/bin # ./kafka-reassign-partitions.sh > --zookeeper $ZKLIST --reassignment-json-file /tmp/r7.json --verify > > Status of partition reassignment: > Partitions reassignment failed due to > com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer; > java.lang.AbstractMethodError: > com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer; > at > com.fasterxml.jackson.databind.ext.OptionalHandlerFactory.findDeserializer(OptionalHandlerFactory.java:122) > at > com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findOptionalStdDeserializer(BasicDeserializerFactory.java:1589) > at > com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findDefaultDeserializer(BasicDeserializerFactory.java:1812) > at > com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.findStdDeserializer(BeanDeserializerFactory.java:161) > at > com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:125) > at > com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411) > at > com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349) > at > com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264) > at > com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) > at > com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) > at > com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477) > at > com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4178) > at > com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997) > at > com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3079) > at kafka.utils.Json$.parseBytesAs(Json.scala:73) > at kafka.zk.ReassignPartitionsZNode$.decode(ZkData.scala:407) > at > kafka.zk.KafkaZkClient.getPartitionReassignment(KafkaZkClient.scala:795) > at > kafka.admin.ReassignPartitionsCommand$.checkIfPartitionReassignmentSucceeded(ReassignPartitionsCommand.scala:355) > at > kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:97) > at > kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90) > at > kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61) > at > kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala) > > Help appreciated. >
