Hi Team,
In my production setup, I’m getting below exception(for 4-5
__consumer_offsets partitions) while restarting kafka broker(s).
Before restart, my producers(sync) are very slow. for some messages its
taking 5 seconds to acknowledge.
But after restart, all messages wishing 2 milliseconds.
After sometime once again same producer slowness.
Before restart was able to produce 7k messages, but after restart able to
push 1lakh messages in 10 minutes.
Kafka broker and client version- 10.0.0.1
Using latest producer java client for this version.
But consumers are old scala api.
Exception:
[2019-11-27 05:54:43,724] INFO [Group Metadata Manager on Broker 3]:
Loading offsets and group metadata from [__consumer_offsets,34]
(kafka.coordinator.GroupMetadataManager)
[2019-11-27 05:54:43,735] ERROR [Group Metadata Manager on Broker 3]: Error
in loading offsets from [__consumer_offsets,34]
(kafka.coordinator.GroupMetadataManager)
kafka.common.KafkaException: Unknown group metadata version 1
at
kafka.coordinator.GroupMetadataManager$.schemaForGroup(GroupMetadataManager.scala:764)
at
kafka.coordinator.GroupMetadataManager$.readGroupMessageValue(GroupMetadataManager.scala:934)
at
kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1$$anonfun$apply$mcV$sp$1.apply(GroupMetadataManager.scala:412)
at
kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1$$anonfun$apply$mcV$sp$1.apply(GroupMetadataManager.scala:383)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
at
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at kafka.message.MessageSet.foreach(MessageSet.scala:71)
at
kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply$mcV$sp(GroupMetadataManager.scala:383)
at
kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply(GroupMetadataManager.scala:374)
at
kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply(GroupMetadataManager.scala:374)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:231)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:239)
at
kafka.coordinator.GroupMetadataManager.kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1(GroupMetadataManager.scala:374)
at
kafka.coordinator.GroupMetadataManager$$anonfun$loadGroupsForPartition$1.apply$mcV$sp(GroupMetadataManager.scala:353)
at
kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:56)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
