Is this what you are looking for - 2014-10-28 21:46:24,146 pool-13-thread-1 INFO kafka.producer.SyncProducer-68: Connected to 172.31.16.31:9092 for producing 2014-10-28 21:46:24,149 pool-13-thread-1 INFO kafka.producer.SyncProducer-68: Disconnecting from 172.31.16.31:9092 2014-10-28 21:46:24,149 pool-13-thread-1 DEBUG kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s) Set(Test) 2014-10-28 21:46:24,150 pool-13-thread-1 WARN kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata [{TopicMetadata for topic Test -> No partition metadata for topic Test due to kafka.common.LeaderNotAvailableException}] for topic [Test]: class kafka.common.LeaderNotAvailableException 2014-10-28 21:46:24,150 pool-13-thread-1 DEBUG kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for topic Test 2014-10-28 21:46:24,151 pool-13-thread-1 INFO kafka.client.ClientUtils$-68: Fetching metadata from broker id:0,host:172.31.16.31,port:9092 with correlation id 9 for 1 topic(s) Set(Test) 2014-10-28 21:46:24,151 pool-13-thread-1 DEBUG kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000 (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400 (requested 102400). 2014-10-28 21:46:24,152 pool-13-thread-1 INFO kafka.producer.SyncProducer-68: Connected to 172.31.16.31:9092 for producing 2014-10-28 21:46:24,154 pool-13-thread-1 INFO kafka.producer.SyncProducer-68: Disconnecting from 172.31.16.31:9092 2014-10-28 21:46:24,155 pool-13-thread-1 DEBUG kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s) Set(Test) 2014-10-28 21:46:24,155 pool-13-thread-1 WARN kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata [{TopicMetadata for topic Test -> No partition metadata for topic Test due to kafka.common.LeaderNotAvailableException}] for topic [Test]: class kafka.common.LeaderNotAvailableException 2014-10-28 21:46:24,155 pool-13-thread-1 ERROR kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: Test 2014-10-28 21:46:24,156 pool-13-thread-1 INFO kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before retrying send. Remaining retries = 6 2014-10-28 21:46:25,156 pool-13-thread-1 INFO kafka.client.ClientUtils$-68: Fetching metadata from broker id:0,host:172.31.16.31,port:9092 with correlation id 10 for 1 topic(s) Set(Test) 2014-10-28 21:46:25,157 pool-13-thread-1 DEBUG kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000 (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400 (requested 102400). 2014-10-28 21:46:25,157 pool-13-thread-1 INFO kafka.producer.SyncProducer-68: Connected to 172.31.16.31:9092 for producing 2014-10-28 21:46:25,160 pool-13-thread-1 INFO kafka.producer.SyncProducer-68: Disconnecting from 172.31.16.31:9092 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s) Set(Test) 2014-10-28 21:46:25,161 pool-13-thread-1 WARN kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata [{TopicMetadata for topic Test -> No partition metadata for topic Test due to kafka.common.LeaderNotAvailableException}] for topic [Test]: class kafka.common.LeaderNotAvailableException 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for topic Test 2014-10-28 21:46:25,162 pool-13-thread-1 INFO kafka.client.ClientUtils$-68: Fetching metadata from broker id:0,host:172.31.16.31,port:9092 with correlation id 11 for 1 topic(s) Set(Test) 2014-10-28 21:46:25,162 pool-13-thread-1 DEBUG kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000 (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400 (requested 102400). 2014-10-28 21:46:25,163 pool-13-thread-1 INFO kafka.producer.SyncProducer-68: Connected to 172.31.16.31:9092 for producing 2014-10-28 21:46:25,166 pool-13-thread-1 INFO kafka.producer.SyncProducer-68: Disconnecting from 172.31.16.31:9092 2014-10-28 21:46:25,166 pool-13-thread-1 DEBUG kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s) Set(Test) 2014-10-28 21:46:25,166 pool-13-thread-1 WARN kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata [{TopicMetadata for topic Test -> No partition metadata for topic Test due to kafka.common.LeaderNotAvailableException}] for topic [Test]: class kafka.common.LeaderNotAvailableException 2014-10-28 21:46:25,167 pool-13-thread-1 ERROR kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: Test 2014-10-28 21:46:25,167 pool-13-thread-1 INFO kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before retrying send. Remaining retries = 5
As for /tmp/kafka-logs, I can see only these files - recovery-point-offset-checkpoint replication-offset-checkpoint I do not see the usual Kafka log files here. thanks Sameer On Tue, Oct 28, 2014 at 1:23 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Sameer, could you check broker logs and see if it has similar > LeaderNotAvailable exceptions? > > Guozhang > > On Tue, Oct 28, 2014 at 10:32 AM, Sameer Yami <sy...@ailive.net> wrote: > > > Yes. I can telnet from outside using the public ip of the instance and > port > > of 9092 > > > > On Tue, Oct 28, 2014 at 10:08 AM, Roger Hoover <roger.hoo...@gmail.com> > > wrote: > > > > > Just a guess but could it be a firewall issue? Did you enable > > connections > > > to port 9092 from outside EC2 in a security group? Can you telnet to > > each > > > broker IP and port? > > > > > > On Tue, Oct 28, 2014 at 10:01 AM, Sameer Yami <sy...@ailive.net> > wrote: > > > > > > > There was a typo earlier. > > > > > > > > This is the output - > > > > > > > > Topic:Test PartitionCount:2 ReplicationFactor:1 Configs: > > > > Topic: Test Partition: 0 Leader: 0 Replicas: 0 > > Isr: 0 > > > > Topic: Test Partition: 1 Leader: 0 Replicas: 0 > > Isr: 0 > > > > > > > > thanks > > > > Sameer > > > > > > > > > > > > > > > > > > > > On Tue, Oct 28, 2014 at 9:40 AM, Guozhang Wang <wangg...@gmail.com> > > > wrote: > > > > > > > > > Sameer, > > > > > > > > > > The topic name is not consistent from your describe outputs. > Normally > > > > they > > > > > should be either > > > > > > > > > > Topic:*Test* PartitionCount:2 ReplicationFactor:1 > Configs: > > > > > Topic: *Test* Partition: 0 Leader: 0 Replicas: 0 > > > > Isr: 0 > > > > > Topic: *Test* Partition: 1 Leader: 0 Replicas: 0 > > > > Isr: 0 > > > > > > > > > > Or > > > > > > > > > > Topic:*Heartbeat* PartitionCount:2 ReplicationFactor:1 > > > Configs: > > > > > Topic: *Heartbeat* Partition: 0 Leader: 0 > Replicas: > > 0 > > > > > Isr: 0 > > > > > Topic: *Heartbeat* Partition: 1 Leader: 0 > Replicas: > > 0 > > > > > Isr: 0 > > > > > > > > > > Did you manually modify the topic name in ZK? > > > > > > > > > > Guozhang > > > > > > > > > > On Tue, Oct 28, 2014 at 8:53 AM, Sameer Yami <sy...@ailive.net> > > wrote: > > > > > > > > > > > Hi Guozhang, > > > > > > > > > > > > I was wondering if there is anything wrong in the above > --describe > > > > data. > > > > > > > > > > > > thanks > > > > > > > > > > > > > > > > > > On Mon, Oct 27, 2014 at 2:56 PM, Sameer Yami <sy...@ailive.net> > > > wrote: > > > > > > > > > > > > > This is what I get - > > > > > > > bin/kafka-topics.sh --zookeeper localhost:2181 --describe > > > > > > > Topic:Heartbeat PartitionCount:2 ReplicationFactor:1 > > > > Configs: > > > > > > > Topic: Test Partition: 0 Leader: 0 Replicas: 0 > > > > > Isr: 0 > > > > > > > Topic: Test Partition: 1 Leader: 0 Replicas: 0 > > > > > Isr: 0 > > > > > > > > > > > > > > The topics are being created at run-time. > > > > > > > > > > > > > > thanks > > > > > > > > > > > > > > > > > > > > > On Sun, Oct 26, 2014 at 4:49 PM, Guozhang Wang < > > wangg...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > >> Could you use the topic tools "--describe" to check if the > topic > > > > > Test's > > > > > > >> partitions leaders are available? > > > > > > >> > > > > > > >> > http://kafka.apache.org/documentation.html#basic_ops_add_topic > > > > > > >> > > > > > > >> Guozhang > > > > > > >> > > > > > > >> On Fri, Oct 24, 2014 at 2:43 PM, Sameer Yami < > sy...@ailive.net> > > > > > wrote: > > > > > > >> > > > > > > >> > That is not the case. > > > > > > >> > This detailed error is below. Due to this error , the > producer > > > is > > > > > not > > > > > > >> able > > > > > > >> > to send messages > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main DEBUG > > > > > > >> kafka.producer.BrokerPartitionInfo-52: > > > > > > >> > Getting broker partition info for topic Test > > > > > > >> > 2014-10-24 14:31:47,209 main INFO > > > kafka.client.ClientUtils$-68: > > > > > > >> Fetching > > > > > > >> > metadata from broker id:0,host:54.215.200.61,port:9092 with > > > > > > correlation > > > > > > >> id > > > > > > >> > 1 for 1 topic(s) Set(Test) > > > > > > >> > 2014-10-24 14:31:47,230 main DEBUG > > > > > kafka.network.BlockingChannel-52: > > > > > > >> > Created socket with SO_TIMEOUT = 10000 (requested 10000), > > > > SO_RCVBUF > > > > > = > > > > > > >> > 187200 (requested -1), SO_SNDBUF = 102400 (requested > 102400). > > > > > > >> > 2014-10-24 14:31:47,231 main INFO > > > > kafka.producer.SyncProducer-68: > > > > > > >> > Connected to 54.215.200.61:9092 for producing > > > > > > >> > 2014-10-24 14:31:47,256 main INFO > > > > kafka.producer.SyncProducer-68: > > > > > > >> > Disconnecting from 54.215.200.61:9092 > > > > > > >> > 2014-10-24 14:31:47,256 main DEBUG > > > kafka.client.ClientUtils$-52: > > > > > > >> > Successfully fetched metadata for 1 topic(s) Set(Test) > > > > > > >> > 2014-10-24 14:31:47,257 main WARN > > > > > > >> kafka.producer.BrokerPartitionInfo-83: > > > > > > >> > Error while fetching metadata [{TopicMetadata for topic Test > > -> > > > > > > >> > No partition metadata for topic Test due to > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]: > > > class > > > > > > >> > kafka.common.LeaderNotAvailableException > > > > > > >> > 2014-10-24 14:31:47,258 main ERROR > > > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to > collate > > > > > > messages > > > > > > >> by > > > > > > >> > topic, partition due to: Failed to fetch topic metadata for > > > topic: > > > > > > Test > > > > > > >> > > > > > > > >> > On Fri, Oct 24, 2014 at 1:11 PM, Guozhang Wang < > > > > wangg...@gmail.com> > > > > > > >> wrote: > > > > > > >> > > > > > > > >> > > Could you see if this is the case: > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers > > > > > > >> > > ? > > > > > > >> > > > > > > > > >> > > On Fri, Oct 24, 2014 at 11:35 AM, Sameer Yami < > > > sy...@ailive.net > > > > > > > > > > > >> wrote: > > > > > > >> > > > > > > > > >> > > > Hi, > > > > > > >> > > > > > > > > > >> > > > We have a Kafka / Zookeeper test setup on a single EC2 > > > > machine. > > > > > > The > > > > > > >> > > > host.name is the private ip address and the > > > > > advertised.host.name > > > > > > is > > > > > > >> > the > > > > > > >> > > > public address in server.properties. > > > > > > >> > > > > > > > > > >> > > > We are trying to publish from outside EC2 using the > public > > > ip > > > > of > > > > > > the > > > > > > >> > EC2 > > > > > > >> > > > machine but get this error on the client side - > > > > > > >> > > > No partition metadata for topic test due to > > > > > > >> > > > kafka.common.LeaderNotAvailableException}] for topic > > [test]: > > > > > class > > > > > > >> > > > kafka.common.LeaderNotAvailableException > > > > > > >> > > > > > > > > > >> > > > The zookeeper has the public ip stored in > /brokers/ids/0. > > > > > > >> > > > We have tried multiple combinations but it works ONLY > when > > > the > > > > > > >> > advertised > > > > > > >> > > > host name is not used and the client is internal to EC2. > > > > > > >> > > > > > > > > > >> > > > Please let me know what the problem could be. > > > > > > >> > > > > > > > > > >> > > > thanks > > > > > > >> > > > Sameer > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > -- > > > > > > >> > > -- Guozhang > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> -- > > > > > > >> -- Guozhang > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > -- > -- Guozhang >