Hi, That sounds similar to this issue: KAFKA-14190 <https://issues.apache.org/jira/browse/KAFKA-14190>. Could you help confirm it and comment in the JIRA? That will let contributors know this is an important issue bothering many users.
Thank you. Luke On Tue, Jan 10, 2023 at 10:08 AM megh vidani <vidanimeg...@gmail.com> wrote: > Hi Tom, > > We faced similar problem wherein there was an issue with isr and we were > also getting NotLeaderOrFollowerException on consumer end. Also, it was not > getting fixed automatically with broker restarts. > > We eventually found out that the topicId for a few partitions in the topic > (in the partition.metadata file) was different from the actual topicId in > zookeeper. I'd suggest you to check that as well. > > The way we fixed it was to remove the partition.metadata file (only this > file alone!!) from all the partition directories of that topic and then > restarting the brokers. This was the safest option we found as it doesn't > incur any data loss. Before figuring this out we used to delete and > re-create the topic which resulted into data being lost. > > Hope this helps. > > Thanks, > Megh > > On Mon, 9 Jan 2023, 22:28 Tom Bolitho, <tboli...@gmail.com> wrote: > > > Dear Kafka Community, > > > > I'm hoping you can help with kafka topic partition that is missing a > > leader. The topic in question is the '__consumer_offsets' topic > > > > The output of a '--describe' on that topic looks like: > > > > Topic: __consumer_offsets Partition: 7 Leader: none Replicas 5 > Isr: > > 5 > > Topic: __consumer_offsets Partition: 11 Leader: none Replicas 5 > Isr: > > 5 > > > > The other 48 partitions are all ok and have an assigned leader (some > with 5 > > as the leader). > > > > I have tried running a --reassignment-json-file against the topic .e.g > > > > kafka-reassign-partitons.sh --bootstrap-server localhost:9092 > > --reassignment-json-file /<reassignmentfilename>.json --execute > > > > but the reassignment just hangs, with the two partitions that are > missing a > > leader reporting: > > 'Reassignment of partition __consumer_offsets-7 is still in progress' > > > > I've since had to --cancel that reassignment > > > > Can anyone advise on how I can overcome the issue of this missing leader > > please? > > > > My eventual goal is to reassign this __consumer_offsets topic with a > > replication factor of 3 to increase resiliency now that the cluster is in > > production. I realise we should have set the > > offets.topic.replication.factor to a value higher than 1 before we spun > up > > the prod cluster but this was missed so we're now looking to manually > > reassign the __consumer_offsets with a higher replication factor. > > > > Any advice on how to overcome this 'Leader: none' issue would be greatly > > appreciated. > > > > Many thanks, > > > > Tom > > >
