Hi Meghana, We also faced similar issue and found that it returned ConsumerCoordinatorNotAvailableCode always for one broker (server id 3) and leader for all partitions of __consumer_offsets topic is same broker id 3.
So wiped off kafka data dir on that broker and restarted it. After that ConsumerOffsetChecker started working. We had replication enabled so losing kafka data dir is not big deal for us. If you can't delete complete kafka data dir, you can try deleting just __consumer_offsets data. Thanks, Raja. On Thu, May 14, 2015 at 10:46 AM, Meghana Narasimhan < mnarasim...@bandwidth.com> wrote: > Hi Mayuresh, > A few more inputs that I can provide at the moment after some testing are > as follows. > 1. The error returned by the consumer offset checker's > ConsumerMetadataResponse is "ConsumerCoordinatorNotAvailableCode". Could it > somehow be related to the offsets being written to zookeeper and not the > internal kafka offsets topic ? > 2. I see that the __consumer_offsets topic has been created on all the > brokers. > 3. I also tried the steps for migrating from zookeeper to kafka offset > topic as specified in the documentation [using the offsets.storage and > dual.commit.enabled configs] > 4. Also based on a few other links I tried to commit offset > using OffsetCommitRequest method and to get ConsumerMetadataResponse . But > the OffsetCommitResponse also returned error and could not commit the > offset successfully. > > On the other hand the producer and consumer are working fine and able to > produce and consume data from the topic. The issue is only with the > consumer offset checker tool. > > Thanks, > Meghana > > > On Mon, May 11, 2015 at 7:34 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi Meghana, > > > > Let me try this out on my cluster that has latest trunk deployed. > > > > Thanks, > > > > Mayuresh > > > > On Mon, May 11, 2015 at 1:53 PM, Meghana Narasimhan < > > mnarasim...@bandwidth.com> wrote: > > > > > Hi Mayuresh, > > > A small update. The Kafka version I'm currently using is 2.10-0.8.2.1 > > (not > > > 2.11 as previously mentioned). The cluster looks fine. Not sure why the > > > consumer offset checker does not return a valid output and gets stuck. > > > > > > bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test > > > Topic:test PartitionCount:3 ReplicationFactor:3 > > > Configs:min.insync.replicas=2 > > > Topic: test Partition: 0 Leader: 1 Replicas: 1,2,0 > Isr: > > > 1,2,0 > > > Topic: test Partition: 1 Leader: 2 Replicas: 2,0,1 > Isr: > > > 1,2,0 > > > Topic: test Partition: 2 Leader: 0 Replicas: 0,1,2 > Isr: > > > 1,2,0 > > > > > > > > > > > > > > > On Fri, May 8, 2015 at 12:52 PM, Meghana Narasimhan < > > > mnarasim...@bandwidth.com> wrote: > > > > > > > Hi Mayuresh, > > > > > > > > Yes, the broker is up and accepting connections. Multiple consumers > are > > > > consuming off topics on the broker. > > > > Also I am seeing the issue only with this particular version ( > > > > 2.11-0.8.2.1). It worked fine with the beta that I was using earlier. > > > > > > > > > > > > On Fri, May 8, 2015 at 12:45 PM, Mayuresh Gharat < > > > > gharatmayures...@gmail.com> wrote: > > > > > > > >> Is X.X.X.X:9092 up and accepting connections? > > > >> I am confused aas in why is it not connecting some other broker if > > > >> connection to this broker fails. Can you check if the broker is up? > > > >> > > > >> The way it works is the consumer will send a ConsumerMetadataRequest > > to > > > >> one > > > >> of the brokers and get the offsetmanager for its group and then > > perform > > > >> the > > > >> offset management. > > > >> > > > >> Thanks, > > > >> > > > >> Mayuresh > > > >> > > > >> On Fri, May 8, 2015 at 9:22 AM, Meghana Narasimhan < > > > >> mnarasim...@bandwidth.com> wrote: > > > >> > > > >> > Hi, > > > >> > I'm using the Kafka 8.2.1 version(kafka_2.11-0.8.2.1) and the > > consumer > > > >> > offset checker hangs indefinitely and does not return any > results. I > > > >> > enabled the debug for tools and below is the debug statements as > > seen > > > on > > > >> > the stdout. Any thoughts or inputs on this will be much > appreciated. > > > >> > > > > >> > command used : > > > >> > bin/kafka-consumer-offset-checker.sh --zookeeper localhost:2181 > > > --group > > > >> > test-consumer-group > > > >> > or > > > >> > ./kafka-consumer-offset-checker.sh --zookeeper > > > >> > broker1:2181,broker2:2181,broker3:2181 --group test-consumer-group > > > >> > > > > >> > DEBUG Querying X.X.X.X:9092 to locate offset manager for > > > >> > test-consumer-group. (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:23:55,090] DEBUG Consumer metadata response: > > > >> > ConsumerMetadataResponse(None,15,0) (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:23:55,091] DEBUG Query to X.X.X.X:9092 to locate > > offset > > > >> > manager for test-consumer-group failed - will retry in 3000 > > > >> milliseconds. > > > >> > (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:23:58,093] DEBUG Querying X.X.X.X:9092 to locate > > offset > > > >> > manager for test-consumer-group. (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:23:58,102] DEBUG Consumer metadata response: > > > >> > ConsumerMetadataResponse(None,15,0) (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:23:58,103] DEBUG Query to X.X.X.X:9092 to locate > > offset > > > >> > manager for test-consumer-group failed - will retry in 3000 > > > >> milliseconds. > > > >> > (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:01,107] DEBUG Querying X.X.X.X:9092 to locate > > offset > > > >> > manager for test-consumer-group. (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:01,115] DEBUG Consumer metadata response: > > > >> > ConsumerMetadataResponse(None,15,0) (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:01,116] DEBUG Query to X.X.X.X:9092 to locate > > offset > > > >> > manager for test-consumer-group failed - will retry in 3000 > > > >> milliseconds. > > > >> > (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:04,119] DEBUG Querying X.X.X.X:9092 to locate > > offset > > > >> > manager for test-consumer-group. (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:04,124] DEBUG Consumer metadata response: > > > >> > ConsumerMetadataResponse(None,15,0) (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:04,126] DEBUG Query to X.X.X.X:9092 to locate > > offset > > > >> > manager for test-consumer-group failed - will retry in 3000 > > > >> milliseconds. > > > >> > (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:04,993] DEBUG Got ping response for sessionid: > > > >> > 0x14d33e7fbc80002 after 3ms (org.apache.zookeeper.ClientCnxn) > > > >> > [2015-05-08 10:24:07,127] DEBUG Querying X.X.X.X:9092 to locate > > offset > > > >> > manager for test-consumer-group. (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:07,131] DEBUG Consumer metadata response: > > > >> > ConsumerMetadataResponse(None,15,0) (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:07,132] DEBUG Query to X.X.X.X:9092 to locate > > offset > > > >> > manager for test-consumer-group failed - will retry in 3000 > > > >> milliseconds. > > > >> > (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:10,132] DEBUG Querying X.X.X.X:9092 to locate > > offset > > > >> > manager for test-consumer-group. (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:10,138] DEBUG Consumer metadata response: > > > >> > ConsumerMetadataResponse(None,15,0) (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:10,139] DEBUG Query to X.X.X.X:9092 to locate > > offset > > > >> > manager for test-consumer-group failed - will retry in 3000 > > > >> milliseconds. > > > >> > (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:13,143] DEBUG Querying X.X.X.X:9092 to locate > > offset > > > >> > manager for test-consumer-group. (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:13,151] DEBUG Consumer metadata response: > > > >> > ConsumerMetadataResponse(None,15,0) (kafka.client.ClientUtils$) > > > >> > [2015-05-08 10:24:13,152] DEBUG Query to X.X.X.X:9092 to locate > > offset > > > >> > manager for test-consumer-group failed - will retry in 3000 > > > >> milliseconds. > > > >> > (kafka.client.ClientUtils$) > > > >> > > > > >> > Thanks, > > > >> > Meghana > > > >> > > > > >> > > > >> > > > >> > > > > > -- Thanks, Raja.
