Jackson was updated to 2.10 in the latest Kafka release. The method
mentioned no longer exists in 2.10.
Do you have multiple versions of Jackson on the ckasspath?
On Thu, 12 Dec 2019, 11:09 Charles Bueche, <cbli...@bueche.ch> wrote:
> Hello again,
>
> spending hours debugging this and having no clue...
>
> * Kafka 2.12-2.1.0 : kafka-reassign-partitions.sh is working fine
> * Anything newer (2.2.2, 2.3.1, trunk) breaks any operation involving
> JSON. Stack Dump below.
>
> Help please.
>
> Charles
>
> Le 11.12.19 à 16:58, Charles Bueche a écrit :
> > Hello again,
> >
> > Problem is still present, something is badly broken with the handling of
> > JSON files. Details :
> >
> > I have made multiple downgrades and rolling-upgrades, and the situation
> > is now :
> >
> > * Zookeeper 3.4.14
> > * Kafka 2.12-2.3.1 (contains libs/zookeeper-3.4.14.jar)
> >
> > The cluster is all "happy", with balanced, no skew, etc. I have
> > kafka-manager and everything is ok.
> >
> > I then create a "testing" topic :
> >
> > export ZKLIST="srv1:2181,srv2:2181,srv3:2181"
> > /appl/kafka/bin/kafka-topics.sh --zookeeper $ZKLIST --create
> > --replication-factor 3 --partitions 1 --topic testing
> >
> > I can rise the partitions fine :
> >
> > /appl/kafka/bin/kafka-topics.sh --zookeeper $ZKLIST --alter --topic
> > testing --partitions 3
> >
> > It then looks ok :
> >
> > /appl/kafka/bin/kafka-topics.sh --zookeeper $ZKLIST --describe
> > --topic testing
> > Topic:testing PartitionCount:3 ReplicationFactor:3
> Configs:
> > Topic: testing Partition: 0 Leader: 1 Replicas: 1,2,3
> > Isr: 1,2,3
> > Topic: testing Partition: 1 Leader: 2 Replicas: 2,3,1
> > Isr: 2,3,1
> > Topic: testing Partition: 2 Leader: 3 Replicas: 3,1,2
> > Isr: 3,1,2
> >
> > Attempting any operation involving a json file breaks, when it worked
> > fine on Kafka 2.12-2.1.0. I shall precise my json files are passing lint
> > tests and work ok.
> >
> > Helped needed !!!
> >
> > Thanks,
> > Charles
> >
> > /appl/kafka/bin/kafka-reassign-partitions.sh --zookeeper $ZKLIST
> > --reassignment-json-file ~buec/tmp/r7.json --execute
> > Current partition replica assignment
> >
> >
> {"version":1,"partitions":[{"topic":"testing","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"testing","partition":2,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"testing","partition":1,"replicas":[2,3,1],"log_dirs":["any","any","any"]}]}
> >
> > Save this to use as the --reassignment-json-file option during rollback
> > Partitions reassignment failed due to
> >
> com.fasterxml.jackson.databind.ext.Java7Support.getSerializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonSerializer;
> > java.lang.AbstractMethodError:
> >
> com.fasterxml.jackson.databind.ext.Java7Support.getSerializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonSerializer;
> > at
> >
> com.fasterxml.jackson.databind.ext.OptionalHandlerFactory.findSerializer(OptionalHandlerFactory.java:92)
> > at
> >
> com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findOptionalStdSerializer(BasicSerializerFactory.java:439)
> > at
> >
> com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByPrimaryType(BasicSerializerFactory.java:374)
> > at
> >
> com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:226)
> > at
> >
> com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:165)
> > at
> >
> com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1385)
> > at
> >
> com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1336)
> > at
> >
> com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:510)
> > at
> >
> com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:713)
> > at
> >
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:308)
> > at
> >
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3893)
> > at
> >
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3231)
> > at kafka.utils.Json$.encodeAsBytes(Json.scala:115)
> > at kafka.zk.ReassignPartitionsZNode$.encode(ZkData.scala:403)
> > at
> >
> kafka.zk.KafkaZkClient.createPartitionReassignment(KafkaZkClient.scala:843)
> > at
> >
> kafka.admin.ReassignPartitionsCommand.reassignPartitions(ReassignPartitionsCommand.scala:635)
> > at
> >
> kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:221)
> > at
> >
> kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:205)
> > at
> >
> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:65)
> > at
> >
> kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
> >
> >
> > Le 05.12.19 à 17:33, Ismael Juma a écrit :
> >> Are you using ZooKeeper 3.5.6 client libraries with Kafka 2.3.1? Kafka
> >> 2.3.1 ships with ZooKeeper 3.4.x.
> >>
> >> Ismael
> >>
> >> On Thu, Dec 5, 2019 at 8:18 AM Charles Bueche <cbue...@gmail.com>
> wrote:
> >>
> >>> Hello all,
> >>>
> >>> I do have two Kafka clusters in action, test and prod. The two are
> formed
> >>> by 3 nodes each, are independent and run their own zookeeper setups. My
> >>> prod cluster is running fine. My test cluster is half-broken and I'm
> >>> struggling to fix it. I could wipe it but I prefer to understand what's
> >>> wrong and fix it.
> >>>
> >>> I'm not sure what broke my test cluster. I had several network
> >>> disconnections / split-brains but Kafka always recovered fine. The
> reasons
> >>> for the network issues are independent and still being investigated
> >>> (layer-2 storms, etc).
> >>>
> >>> So I upgraded my zookeeper and kafka to the latest versions and when
> >>> trying to rebalance a topic across brokers I started to notice the
> >>> problems. Not sure really when they started, before or after the
> upgrade.
> >>>
> >>> I ran the upgrade as for the official doc (rolling upgrade, moving up
> >>> inter.broker.protocol.version and log.message.format.version
> gradually).
> >>>
> >>> ------------------------------------------------------
> >>> cat /etc/os-release
> >>> NAME="SLES"
> >>> VERSION="12-SP4"
> >>> VERSION_ID="12.4"
> >>> PRETTY_NAME="SUSE Linux Enterprise Server 12 SP4"
> >>> ID="sles"
> >>> ANSI_COLOR="0;32"
> >>> CPE_NAME="cpe:/o:suse:sles:12:sp4"
> >>>
> >>> ------------------------------------------------------
> >>> versions :
> >>> - zookeeper 3.5.6
> >>> - kafka 2.12-2.3.1
> >>>
> >>> ------------------------------------------------------
> >>>
> >>> many rapid log entries on brokers 1 & 2 (we have 1, 2, 3)
> >>>
> >>> [2019-12-05 09:56:54,967] ERROR [KafkaApi-1] Number of alive brokers
> '0'
> >>> does not meet the required replication factor '1' for the offsets topic
> >>> (configured via 'offsets.topic.replication.factor'). This error can be
> >>> ignored if the cluster is starting up and not all brokers are up yet.
> >>> (kafka.server.KafkaApis)
> >>>
> >>> ------------------------------------------------------
> >>> java is java-1_8_0-openjdk-1.8.0.222-27.35.2.x86_64 from SLES. I have
> >>> tried Oracle Java jdk1.8.0_231 with the same issue.
> >>>
> >>> ------------------------------------------------------
> >>> when trying to see a reassignment I have this very suspect error :
> >>>
> >>> root@vmgato701a01:/appl/kafka/bin # ./kafka-reassign-partitions.sh
> >>> --zookeeper $ZKLIST --reassignment-json-file /tmp/r7.json --verify
> >>>
> >>> Status of partition reassignment:
> >>> Partitions reassignment failed due to
> >>>
> com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
> >>> java.lang.AbstractMethodError:
> >>>
> com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
> >>> at
> >>>
> com.fasterxml.jackson.databind.ext.OptionalHandlerFactory.findDeserializer(OptionalHandlerFactory.java:122)
> >>> at
> >>>
> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findOptionalStdDeserializer(BasicDeserializerFactory.java:1589)
> >>> at
> >>>
> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findDefaultDeserializer(BasicDeserializerFactory.java:1812)
> >>> at
> >>>
> com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.findStdDeserializer(BeanDeserializerFactory.java:161)
> >>> at
> >>>
> com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:125)
> >>> at
> >>>
> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411)
> >>> at
> >>>
> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
> >>> at
> >>>
> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
> >>> at
> >>>
> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
> >>> at
> >>>
> com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
> >>> at
> >>>
> com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
> >>> at
> >>>
> com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4178)
> >>> at
> >>>
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997)
> >>> at
> >>>
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3079)
> >>> at kafka.utils.Json$.parseBytesAs(Json.scala:73)
> >>> at kafka.zk.ReassignPartitionsZNode$.decode(ZkData.scala:407)
> >>> at
> >>>
> kafka.zk.KafkaZkClient.getPartitionReassignment(KafkaZkClient.scala:795)
> >>> at
> >>>
> kafka.admin.ReassignPartitionsCommand$.checkIfPartitionReassignmentSucceeded(ReassignPartitionsCommand.scala:355)
> >>> at
> >>>
> kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:97)
> >>> at
> >>>
> kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90)
> >>> at
> >>>
> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)
> >>> at
> >>>
> kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
> >>>
> >>> Help appreciated.
> >>>
> --
> Charles Bueche <cbli...@bueche.ch>
>
>
