Hello again,
spending hours debugging this and having no clue...
* Kafka 2.12-2.1.0 : kafka-reassign-partitions.sh is working fine
* Anything newer (2.2.2, 2.3.1, trunk) breaks any operation involving
JSON. Stack Dump below.
Help please.
Charles
Le 11.12.19 à 16:58, Charles Bueche a écrit :
> Hello again,
>
> Problem is still present, something is badly broken with the handling of
> JSON files. Details :
>
> I have made multiple downgrades and rolling-upgrades, and the situation
> is now :
>
> * Zookeeper 3.4.14
> * Kafka 2.12-2.3.1 (contains libs/zookeeper-3.4.14.jar)
>
> The cluster is all "happy", with balanced, no skew, etc. I have
> kafka-manager and everything is ok.
>
> I then create a "testing" topic :
>
> export ZKLIST="srv1:2181,srv2:2181,srv3:2181"
> /appl/kafka/bin/kafka-topics.sh --zookeeper $ZKLIST --create
> --replication-factor 3 --partitions 1 --topic testing
>
> I can rise the partitions fine :
>
> /appl/kafka/bin/kafka-topics.sh --zookeeper $ZKLIST --alter --topic
> testing --partitions 3
>
> It then looks ok :
>
> /appl/kafka/bin/kafka-topics.sh --zookeeper $ZKLIST --describe
> --topic testing
> Topic:testing PartitionCount:3 ReplicationFactor:3 Configs:
> Topic: testing Partition: 0 Leader: 1 Replicas: 1,2,3
> Isr: 1,2,3
> Topic: testing Partition: 1 Leader: 2 Replicas: 2,3,1
> Isr: 2,3,1
> Topic: testing Partition: 2 Leader: 3 Replicas: 3,1,2
> Isr: 3,1,2
>
> Attempting any operation involving a json file breaks, when it worked
> fine on Kafka 2.12-2.1.0. I shall precise my json files are passing lint
> tests and work ok.
>
> Helped needed !!!
>
> Thanks,
> Charles
>
> /appl/kafka/bin/kafka-reassign-partitions.sh --zookeeper $ZKLIST
> --reassignment-json-file ~buec/tmp/r7.json --execute
> Current partition replica assignment
>
> {"version":1,"partitions":[{"topic":"testing","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"testing","partition":2,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"testing","partition":1,"replicas":[2,3,1],"log_dirs":["any","any","any"]}]}
>
> Save this to use as the --reassignment-json-file option during rollback
> Partitions reassignment failed due to
> com.fasterxml.jackson.databind.ext.Java7Support.getSerializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonSerializer;
> java.lang.AbstractMethodError:
> com.fasterxml.jackson.databind.ext.Java7Support.getSerializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonSerializer;
> at
> com.fasterxml.jackson.databind.ext.OptionalHandlerFactory.findSerializer(OptionalHandlerFactory.java:92)
> at
> com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findOptionalStdSerializer(BasicSerializerFactory.java:439)
> at
> com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByPrimaryType(BasicSerializerFactory.java:374)
> at
> com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:226)
> at
> com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:165)
> at
> com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1385)
> at
> com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1336)
> at
> com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:510)
> at
> com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:713)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:308)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3893)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3231)
> at kafka.utils.Json$.encodeAsBytes(Json.scala:115)
> at kafka.zk.ReassignPartitionsZNode$.encode(ZkData.scala:403)
> at
> kafka.zk.KafkaZkClient.createPartitionReassignment(KafkaZkClient.scala:843)
> at
> kafka.admin.ReassignPartitionsCommand.reassignPartitions(ReassignPartitionsCommand.scala:635)
> at
> kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:221)
> at
> kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:205)
> at
> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:65)
> at
> kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
>
>
> Le 05.12.19 à 17:33, Ismael Juma a écrit :
>> Are you using ZooKeeper 3.5.6 client libraries with Kafka 2.3.1? Kafka
>> 2.3.1 ships with ZooKeeper 3.4.x.
>>
>> Ismael
>>
>> On Thu, Dec 5, 2019 at 8:18 AM Charles Bueche <cbue...@gmail.com> wrote:
>>
>>> Hello all,
>>>
>>> I do have two Kafka clusters in action, test and prod. The two are formed
>>> by 3 nodes each, are independent and run their own zookeeper setups. My
>>> prod cluster is running fine. My test cluster is half-broken and I'm
>>> struggling to fix it. I could wipe it but I prefer to understand what's
>>> wrong and fix it.
>>>
>>> I'm not sure what broke my test cluster. I had several network
>>> disconnections / split-brains but Kafka always recovered fine. The reasons
>>> for the network issues are independent and still being investigated
>>> (layer-2 storms, etc).
>>>
>>> So I upgraded my zookeeper and kafka to the latest versions and when
>>> trying to rebalance a topic across brokers I started to notice the
>>> problems. Not sure really when they started, before or after the upgrade.
>>>
>>> I ran the upgrade as for the official doc (rolling upgrade, moving up
>>> inter.broker.protocol.version and log.message.format.version gradually).
>>>
>>> ------------------------------------------------------
>>> cat /etc/os-release
>>> NAME="SLES"
>>> VERSION="12-SP4"
>>> VERSION_ID="12.4"
>>> PRETTY_NAME="SUSE Linux Enterprise Server 12 SP4"
>>> ID="sles"
>>> ANSI_COLOR="0;32"
>>> CPE_NAME="cpe:/o:suse:sles:12:sp4"
>>>
>>> ------------------------------------------------------
>>> versions :
>>> - zookeeper 3.5.6
>>> - kafka 2.12-2.3.1
>>>
>>> ------------------------------------------------------
>>>
>>> many rapid log entries on brokers 1 & 2 (we have 1, 2, 3)
>>>
>>> [2019-12-05 09:56:54,967] ERROR [KafkaApi-1] Number of alive brokers '0'
>>> does not meet the required replication factor '1' for the offsets topic
>>> (configured via 'offsets.topic.replication.factor'). This error can be
>>> ignored if the cluster is starting up and not all brokers are up yet.
>>> (kafka.server.KafkaApis)
>>>
>>> ------------------------------------------------------
>>> java is java-1_8_0-openjdk-1.8.0.222-27.35.2.x86_64 from SLES. I have
>>> tried Oracle Java jdk1.8.0_231 with the same issue.
>>>
>>> ------------------------------------------------------
>>> when trying to see a reassignment I have this very suspect error :
>>>
>>> root@vmgato701a01:/appl/kafka/bin # ./kafka-reassign-partitions.sh
>>> --zookeeper $ZKLIST --reassignment-json-file /tmp/r7.json --verify
>>>
>>> Status of partition reassignment:
>>> Partitions reassignment failed due to
>>> com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
>>> java.lang.AbstractMethodError:
>>> com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
>>> at
>>> com.fasterxml.jackson.databind.ext.OptionalHandlerFactory.findDeserializer(OptionalHandlerFactory.java:122)
>>> at
>>> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findOptionalStdDeserializer(BasicDeserializerFactory.java:1589)
>>> at
>>> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findDefaultDeserializer(BasicDeserializerFactory.java:1812)
>>> at
>>> com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.findStdDeserializer(BeanDeserializerFactory.java:161)
>>> at
>>> com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:125)
>>> at
>>> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411)
>>> at
>>> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
>>> at
>>> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
>>> at
>>> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
>>> at
>>> com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
>>> at
>>> com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
>>> at
>>> com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4178)
>>> at
>>> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997)
>>> at
>>> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3079)
>>> at kafka.utils.Json$.parseBytesAs(Json.scala:73)
>>> at kafka.zk.ReassignPartitionsZNode$.decode(ZkData.scala:407)
>>> at
>>> kafka.zk.KafkaZkClient.getPartitionReassignment(KafkaZkClient.scala:795)
>>> at
>>> kafka.admin.ReassignPartitionsCommand$.checkIfPartitionReassignmentSucceeded(ReassignPartitionsCommand.scala:355)
>>> at
>>> kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:97)
>>> at
>>> kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90)
>>> at
>>> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)
>>> at
>>> kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
>>>
>>> Help appreciated.
>>>
--
Charles Bueche <cbli...@bueche.ch>
