Hi, We also did version upgrade and then degraded to old due to some issue. But that we done ~1 year before. Do I need to follow this to solve this issue?
http://mail-archives.apache.org/mod_mbox/kafka-dev/201907.mbox/%3cjira.13246826.1563975667000.40318.1563975720...@atlassian.jira%3E In this he follow below steps to evicted the broken records with temporarily setting the segment time to a very low value and deactivation of compaction. /opt/kafka/bin/kafka-topics.sh --alter --config segment.ms=900000 --topic __consumer_offsets --zookeeper localhost:2181 /opt/kafka/bin/kafka-topics.sh --alter --config cleanup.policy=delete --topic __consumer_offsets --zookeeper localhost:2181 < wait for the cleaner to clean up > /opt/kafka/bin/kafka-topics.sh --alter --config segment.ms=604800000 --topic __consumer_offsets --zookeeper localhost:2181 /opt/kafka/bin/kafka-topics.sh --alter --config cleanup.policy=compact --topic __consumer_offsets --zookeeper localhost:2181{code} On Wed, Nov 27, 2019 at 10:33 PM Upendra Yadav <upendra1...@gmail.com> wrote: > Hi Team, > > In my production setup, I’m getting below exception(for 4-5 > __consumer_offsets partitions) while restarting kafka broker(s). > Before restart, my producers(sync) are very slow. for some messages its > taking 5 seconds to acknowledge. > But after restart, all messages wishing 2 milliseconds. > After sometime once again same producer slowness. > Before restart was able to produce 7k messages, but after restart able to > push 1lakh messages in 10 minutes. > > Kafka broker and client version- 10.0.0.1 > Using latest producer java client for this version. > But consumers are old scala api. > > Exception: > > [2019-11-27 05:54:43,724] INFO [Group Metadata Manager on Broker 3]: > Loading offsets and group metadata from [__consumer_offsets,34] > (kafka.coordinator.GroupMetadataManager) > [2019-11-27 05:54:43,735] ERROR [Group Metadata Manager on Broker 3]: > Error in loading offsets from [__consumer_offsets,34] > (kafka.coordinator.GroupMetadataManager) > kafka.common.KafkaException: Unknown group metadata version 1 > at > kafka.coordinator.GroupMetadataManager$.schemaForGroup(GroupMetadataManager.scala:764) > at > kafka.coordinator.GroupMetadataManager$.readGroupMessageValue(GroupMetadataManager.scala:934) > at > kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1$$anonfun$apply$mcV$sp$1.apply(GroupMetadataManager.scala:412) > at > kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1$$anonfun$apply$mcV$sp$1.apply(GroupMetadataManager.scala:383) > at scala.collection.Iterator$class.foreach(Iterator.scala:893) > at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30) > at > scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at kafka.message.MessageSet.foreach(MessageSet.scala:71) > at > kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply$mcV$sp(GroupMetadataManager.scala:383) > at > kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply(GroupMetadataManager.scala:374) > at > kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply(GroupMetadataManager.scala:374) > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:231) > at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:239) > at > kafka.coordinator.GroupMetadataManager.kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1(GroupMetadataManager.scala:374) > at > kafka.coordinator.GroupMetadataManager$$anonfun$loadGroupsForPartition$1.apply$mcV$sp(GroupMetadataManager.scala:353) > at > kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110) > at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:56) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) >
