Hi Megh Many thanks for taking the time to get back to me. It sounds like we've had a similar issue although I've checked all of the topic_id's in all of the partition.metadata relating to the problematic topic __consumer_offsets (e.g. grep -r 'topic_id' /data/*/kafka/data/__consumer_offsets-*) and all of the topic ID's are the same on every partition, on all of the nodes for that topic. It sounds like some of your topic_id's were different and therefore you've got a slightly different issue and resolution?
Kind regards, Tom On Tue, 10 Jan 2023 at 02:08, megh vidani <vidanimeg...@gmail.com> wrote: > Hi Tom, > > We faced similar problem wherein there was an issue with isr and we were > also getting NotLeaderOrFollowerException on consumer end. Also, it was not > getting fixed automatically with broker restarts. > > We eventually found out that the topicId for a few partitions in the topic > (in the partition.metadata file) was different from the actual topicId in > zookeeper. I'd suggest you to check that as well. > > The way we fixed it was to remove the partition.metadata file (only this > file alone!!) from all the partition directories of that topic and then > restarting the brokers. This was the safest option we found as it doesn't > incur any data loss. Before figuring this out we used to delete and > re-create the topic which resulted into data being lost. > > Hope this helps. > > Thanks, > Megh > > On Mon, 9 Jan 2023, 22:28 Tom Bolitho, <tboli...@gmail.com> wrote: > > > Dear Kafka Community, > > > > I'm hoping you can help with kafka topic partition that is missing a > > leader. The topic in question is the '__consumer_offsets' topic > > > > The output of a '--describe' on that topic looks like: > > > > Topic: __consumer_offsets Partition: 7 Leader: none Replicas 5 > Isr: > > 5 > > Topic: __consumer_offsets Partition: 11 Leader: none Replicas 5 > Isr: > > 5 > > > > The other 48 partitions are all ok and have an assigned leader (some > with 5 > > as the leader). > > > > I have tried running a --reassignment-json-file against the topic .e.g > > > > kafka-reassign-partitons.sh --bootstrap-server localhost:9092 > > --reassignment-json-file /<reassignmentfilename>.json --execute > > > > but the reassignment just hangs, with the two partitions that are > missing a > > leader reporting: > > 'Reassignment of partition __consumer_offsets-7 is still in progress' > > > > I've since had to --cancel that reassignment > > > > Can anyone advise on how I can overcome the issue of this missing leader > > please? > > > > My eventual goal is to reassign this __consumer_offsets topic with a > > replication factor of 3 to increase resiliency now that the cluster is in > > production. I realise we should have set the > > offets.topic.replication.factor to a value higher than 1 before we spun > up > > the prod cluster but this was missed so we're now looking to manually > > reassign the __consumer_offsets with a higher replication factor. > > > > Any advice on how to overcome this 'Leader: none' issue would be greatly > > appreciated. > > > > Many thanks, > > > > Tom > > >
