Sameer,
Yes, this is the server log. But there seems no abnormal entries in it, and
it does not cover the same time range as the producer client throwing
LeaderNotAvailableException
(it was 10/24, 14:30).
The reason that I want to check the server log at that same reason is
that LeaderNotAvailableException
is set by the brokers while handling requests, and its server log will show
what is the cause it set this error code.
Guozhang
On Tue, Nov 4, 2014 at 12:47 PM, Sameer Yami <sy...@ailive.net> wrote:
> Hi Guozhang,
>
> This is the server.log -
>
> [2014-11-04 20:21:57,510] INFO Verifying properties
> (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,545] INFO Property advertised.host.name is overridden
> to x.x.x.x (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,545] INFO Property broker.id is overridden to 0
> (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,545] INFO Property controlled.shutdown.enable is
> overridden to true (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,545] INFO Property host.name is overridden to
> 172.31.25.198 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,545] INFO Property log.cleaner.enable is overridden to
> false (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,546] INFO Property log.dirs is overridden to
> /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,546] INFO Property log.retention.check.interval.ms is
> overridden to 60000 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,546] INFO Property log.retention.hours is overridden
> to 168 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,546] INFO Property log.segment.bytes is overridden to
> 536870912 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,546] INFO Property num.io.threads is overridden to 8
> (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,547] INFO Property num.network.threads is overridden
> to 2 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,547] INFO Property num.partitions is overridden to 2
> (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,547] INFO Property port is overridden to 9092
> (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,547] INFO Property socket.receive.buffer.bytes is
> overridden to 1048576 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,547] INFO Property socket.request.max.bytes is
> overridden to 104857600 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,547] INFO Property socket.send.buffer.bytes is
> overridden to 1048576 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,548] INFO Property zookeeper.connect is overridden to
> localhost:2181 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,548] INFO Property zookeeper.connection.timeout.ms is
> overridden to 1000000 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,563] INFO [Kafka Server 0], starting
> (kafka.server.KafkaServer)
> [2014-11-04 20:21:57,565] INFO [Kafka Server 0], Connecting to zookeeper on
> localhost:2181 (kafka.server.KafkaServer)
> [2014-11-04 20:21:57,738] INFO Log directory '/tmp/kafka-logs' not found,
> creating it. (kafka.log.LogManager)
> [2014-11-04 20:21:57,748] INFO Starting log cleanup with a period of 60000
> ms. (kafka.log.LogManager)
> [2014-11-04 20:21:57,752] INFO Starting log flusher with a default period
> of 9223372036854775807 ms. (kafka.log.LogManager)
> [2014-11-04 20:21:57,783] INFO Awaiting socket connections on
> ip-172-31-25-198.us-west-1.compute.internal:9092. (kafka.network.Acceptor)
> [2014-11-04 20:21:57,784] INFO [Socket Server on Broker 0], Started
> (kafka.network.SocketServer)
> [2014-11-04 20:21:57,852] INFO Will not load MX4J, mx4j-tools.jar is not in
> the classpath (kafka.utils.Mx4jLoader$)
> [2014-11-04 20:21:57,884] INFO 0 successfully elected as leader
> (kafka.server.ZookeeperLeaderElector)
> [2014-11-04 20:21:57,967] INFO Registered broker 0 at path /brokers/ids/0
> with address x.x.x.x:9092. (kafka.utils.ZkUtils$)
> [2014-11-04 20:21:57,981] INFO [Kafka Server 0], started
> (kafka.server.KafkaServer)
> [2014-11-04 20:21:58,050] INFO New leader is 0
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> [2014-11-04 20:22:01,809] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:22:11,812] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:29,084] INFO Topic creation
> {"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
> [2014-11-04 20:30:29,088] INFO [KafkaApi-0] Auto creation of topic Test
> with 2 partitions and replication factor 1 is successful!
> (kafka.server.KafkaApis)
> [2014-11-04 20:30:29,280] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:29,301] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:30,310] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:30,315] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:31,320] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:31,325] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:32,331] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:32,336] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:33,343] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:33,347] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:34,356] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:34,360] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:35,366] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:35,372] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:36,377] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:36,382] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:37,387] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:37,394] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:38,399] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:38,403] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:39,410] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:39,414] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:40,419] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:41,602] INFO Topic creation
> {"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
>
>
>
> thanks
>
>
> On Fri, Oct 31, 2014 at 9:15 AM, Guozhang Wang <wangg...@gmail.com> wrote:
>
> > The server.log should be the broker logs.
> >
> > Guozhang
> >
> > On Thu, Oct 30, 2014 at 11:32 AM, Sameer Yami <sy...@ailive.net> wrote:
> >
> > > I do not see any other logs besides these.
> > > I checked the log4j.properties and where the logs are getting stored.
> > > The only other logs (apart from the controller logs) in the directory
> are
> > > zookeeper-gc.log,kafkaServer-gc.log, server.log and state-change.log
> > >
> > > Thanks
> > >
> > > On Wed, Oct 29, 2014 at 4:13 PM, Guozhang Wang <wangg...@gmail.com>
> > wrote:
> > >
> > > > Sorry for getting late on this Sameer.
> > > >
> > > > The logs you showed above are from controller logs, do you see
> brokers
> > > logs
> > > > besides that?
> > > >
> > > > Guozhang
> > > >
> > > > On Wed, Oct 29, 2014 at 3:40 PM, Sameer Yami <sy...@ailive.net>
> wrote:
> > > >
> > > > > Hi Guozhang,
> > > > >
> > > > > Is there any clue in the above data as to what is wrong?
> > > > >
> > > > > thanks
> > > > >
> > > > >
> > > > > On Tue, Oct 28, 2014 at 3:45 PM, Sameer Yami <sy...@ailive.net>
> > wrote:
> > > > >
> > > > > > This is the one from the file specified in log4j.properties -
> > > > > >
> > > > > > [2014-10-28 22:38:36,082] DEBUG [TopicChangeListener on
> Controller
> > > 0]:
> > > > > > Topic change listener fired for path /brokers/topics with
> children
> > > Test
> > > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > > [2014-10-28 22:38:36,094] INFO [TopicChangeListener on Controller
> > 0]:
> > > > New
> > > > > > topics: [Set(Test)], deleted topics: [Set()], new partition
> replica
> > > > > > assignment [Map([Test,0] -> List(0), [Test,1] -> List(0))]
> > > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > > [2014-10-28 22:38:36,094] INFO [Controller 0]: New topic creation
> > > > > callback
> > > > > > for [Test,0],[Test,1] (kafka.controller.KafkaController)
> > > > > > [2014-10-28 22:38:36,095] INFO [Controller 0]: New partition
> > creation
> > > > > > callback for [Test,0],[Test,1] (kafka.controller.KafkaController)
> > > > > > [2014-10-28 22:38:36,095] INFO [Partition state machine on
> > Controller
> > > > 0]:
> > > > > > Invoking state change to NewPartition for partitions
> > > [Test,0],[Test,1]
> > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > [2014-10-28 22:38:36,109] INFO [Replica state machine on
> controller
> > > 0]:
> > > > > > Invoking state change to NewReplica for replicas
> > > > > >
> > [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > > > > > (kafka.controller.ReplicaStateMachine)
> > > > > > [2014-10-28 22:38:36,111] INFO [Partition state machine on
> > Controller
> > > > 0]:
> > > > > > Invoking state change to OnlinePartition for partitions
> > > > [Test,0],[Test,1]
> > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > [2014-10-28 22:38:36,111] DEBUG [Partition state machine on
> > > Controller
> > > > > 0]:
> > > > > > Live assigned replicas for partition [Test,0] are: [List(0)]
> > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > [2014-10-28 22:38:36,112] DEBUG [Partition state machine on
> > > Controller
> > > > > 0]:
> > > > > > Initializing leader and isr for partition [Test,0] to
> > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > [2014-10-28 22:38:36,141] DEBUG [Partition state machine on
> > > Controller
> > > > > 0]:
> > > > > > Live assigned replicas for partition [Test,1] are: [List(0)]
> > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > [2014-10-28 22:38:36,142] DEBUG [Partition state machine on
> > > Controller
> > > > > 0]:
> > > > > > Initializing leader and isr for partition [Test,1] to
> > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > [2014-10-28 22:38:36,150] INFO [Replica state machine on
> controller
> > > 0]:
> > > > > > Invoking state change to OnlineReplica for replicas
> > > > > >
> > [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > > > > > (kafka.controller.ReplicaStateMachine)
> > > > > > [2014-10-28 22:39:28,160] ERROR
> > > [Controller-0-to-broker-0-send-thread],
> > > > > > Controller 0's connection to broker id:0,host:x.x.x.x,port:9092
> was
> > > > > > unsuccessful (kafka.controller.RequestSendThread)
> > > > > > java.net.ConnectException: Connection timed out
> > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > > at sun.nio.ch.Net.connect(Net.java:449)
> > > > > > at
> > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
> > > > > > at
> > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > > > > > at
> > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > > [2014-10-28 22:39:28,460] ERROR
> > > [Controller-0-to-broker-0-send-thread],
> > > > > > Controller 0 epoch 1 failed to send LeaderAndIsr request with
> > > > correlation
> > > > > > id 7 to broker id:0,host:x.x.x.x,port:9092. Reconnecting to
> broker.
> > > > > > (kafka.controller.RequestSendThread)
> > > > > > java.nio.channels.ClosedChannelException
> > > > > > at
> kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > > > > > at
> > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > >
> > > > > > On Tue, Oct 28, 2014 at 3:09 PM, Guozhang Wang <
> wangg...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> Sameer, the one you sent me is from the producer logs, not the
> > > broker
> > > > > >> logs.
> > > > > >> And /tmp/kafka-logs actually store the message log (i.e. the
> > "data")
> > > > > >> instead of the server logs. Their locations should be specified
> in
> > > > your
> > > > > >> log4j.properties file.
> > > > > >>
> > > > > >> On Tue, Oct 28, 2014 at 2:51 PM, Sameer Yami <sy...@ailive.net>
> > > > wrote:
> > > > > >>
> > > > > >> > Is this what you are looking for -
> > > > > >> >
> > > > > >> > 2014-10-28 21:46:24,146 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Connected to
> 172.31.16.31:9092
> > > for
> > > > > >> > producing
> > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > 172.31.16.31:9092
> > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 DEBUG
> > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata
> for
> > 1
> > > > > >> topic(s)
> > > > > >> > Set(Test)
> > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 WARN
> > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > > metadata
> > > > > >> > [{TopicMetadata for topic Test ->
> > > > > >> > No partition metadata for topic Test due to
> > > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> > class
> > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 DEBUG
> > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker
> partition
> > > info
> > > > > for
> > > > > >> > topic Test
> > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 INFO
> > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 9 for 1
> > > > topic(s)
> > > > > >> > Set(Test)
> > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 DEBUG
> > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > SO_TIMEOUT =
> > > > > 10000
> > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> SO_SNDBUF
> > =
> > > > > 102400
> > > > > >> > (requested 102400).
> > > > > >> > 2014-10-28 21:46:24,152 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Connected to
> 172.31.16.31:9092
> > > for
> > > > > >> > producing
> > > > > >> > 2014-10-28 21:46:24,154 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > 172.31.16.31:9092
> > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 DEBUG
> > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata
> for
> > 1
> > > > > >> topic(s)
> > > > > >> > Set(Test)
> > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 WARN
> > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > > metadata
> > > > > >> > [{TopicMetadata for topic Test ->
> > > > > >> > No partition metadata for topic Test due to
> > > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> > class
> > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 ERROR
> > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > > > > messages
> > > > > >> by
> > > > > >> > topic, partition due to: Failed to fetch topic metadata for
> > topic:
> > > > > Test
> > > > > >> > 2014-10-28 21:46:24,156 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000
> > ms
> > > > > before
> > > > > >> > retrying send. Remaining retries = 6
> > > > > >> > 2014-10-28 21:46:25,156 pool-13-thread-1 INFO
> > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 10 for 1
> > > > topic(s)
> > > > > >> > Set(Test)
> > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 DEBUG
> > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > SO_TIMEOUT =
> > > > > 10000
> > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> SO_SNDBUF
> > =
> > > > > 102400
> > > > > >> > (requested 102400).
> > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Connected to
> 172.31.16.31:9092
> > > for
> > > > > >> > producing
> > > > > >> > 2014-10-28 21:46:25,160 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > 172.31.16.31:9092
> > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata
> for
> > 1
> > > > > >> topic(s)
> > > > > >> > Set(Test)
> > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 WARN
> > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > > metadata
> > > > > >> > [{TopicMetadata for topic Test ->
> > > > > >> > No partition metadata for topic Test due to
> > > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> > class
> > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker
> partition
> > > info
> > > > > for
> > > > > >> > topic Test
> > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 INFO
> > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 11 for 1
> > > > topic(s)
> > > > > >> > Set(Test)
> > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 DEBUG
> > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > SO_TIMEOUT =
> > > > > 10000
> > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> SO_SNDBUF
> > =
> > > > > 102400
> > > > > >> > (requested 102400).
> > > > > >> > 2014-10-28 21:46:25,163 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Connected to
> 172.31.16.31:9092
> > > for
> > > > > >> > producing
> > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > 172.31.16.31:9092
> > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 DEBUG
> > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata
> for
> > 1
> > > > > >> topic(s)
> > > > > >> > Set(Test)
> > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 WARN
> > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > > metadata
> > > > > >> > [{TopicMetadata for topic Test ->
> > > > > >> > No partition metadata for topic Test due to
> > > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> > class
> > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 ERROR
> > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > > > > messages
> > > > > >> by
> > > > > >> > topic, partition due to: Failed to fetch topic metadata for
> > topic:
> > > > > Test
> > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000
> > ms
> > > > > before
> > > > > >> > retrying send. Remaining retries = 5
> > > > > >> >
> > > > > >> > As for /tmp/kafka-logs, I can see only these files -
> > > > > >> > recovery-point-offset-checkpoint
> replication-offset-checkpoint
> > > > > >> >
> > > > > >> > I do not see the usual Kafka log files here.
> > > > > >> >
> > > > > >> > thanks
> > > > > >> > Sameer
> > > > > >> >
> > > > > >> > On Tue, Oct 28, 2014 at 1:23 PM, Guozhang Wang <
> > > wangg...@gmail.com>
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > Sameer, could you check broker logs and see if it has
> similar
> > > > > >> > > LeaderNotAvailable exceptions?
> > > > > >> > >
> > > > > >> > > Guozhang
> > > > > >> > >
> > > > > >> > > On Tue, Oct 28, 2014 at 10:32 AM, Sameer Yami <
> > sy...@ailive.net
> > > >
> > > > > >> wrote:
> > > > > >> > >
> > > > > >> > > > Yes. I can telnet from outside using the public ip of the
> > > > instance
> > > > > >> and
> > > > > >> > > port
> > > > > >> > > > of 9092
> > > > > >> > > >
> > > > > >> > > > On Tue, Oct 28, 2014 at 10:08 AM, Roger Hoover <
> > > > > >> roger.hoo...@gmail.com
> > > > > >> > >
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Just a guess but could it be a firewall issue? Did you
> > > enable
> > > > > >> > > > connections
> > > > > >> > > > > to port 9092 from outside EC2 in a security group? Can
> > you
> > > > > >> telnet to
> > > > > >> > > > each
> > > > > >> > > > > broker IP and port?
> > > > > >> > > > >
> > > > > >> > > > > On Tue, Oct 28, 2014 at 10:01 AM, Sameer Yami <
> > > > sy...@ailive.net
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > There was a typo earlier.
> > > > > >> > > > > >
> > > > > >> > > > > > This is the output -
> > > > > >> > > > > >
> > > > > >> > > > > > Topic:Test PartitionCount:2 ReplicationFactor:1
> > > > Configs:
> > > > > >> > > > > > Topic: Test Partition: 0 Leader: 0
> > > > Replicas:
> > > > > 0
> > > > > >> > > > Isr: 0
> > > > > >> > > > > > Topic: Test Partition: 1 Leader: 0
> > > > Replicas:
> > > > > 0
> > > > > >> > > > Isr: 0
> > > > > >> > > > > >
> > > > > >> > > > > > thanks
> > > > > >> > > > > > Sameer
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > On Tue, Oct 28, 2014 at 9:40 AM, Guozhang Wang <
> > > > > >> wangg...@gmail.com
> > > > > >> > >
> > > > > >> > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Sameer,
> > > > > >> > > > > > >
> > > > > >> > > > > > > The topic name is not consistent from your describe
> > > > outputs.
> > > > > >> > > Normally
> > > > > >> > > > > > they
> > > > > >> > > > > > > should be either
> > > > > >> > > > > > >
> > > > > >> > > > > > > Topic:*Test* PartitionCount:2
> > ReplicationFactor:1
> > > > > >> > > Configs:
> > > > > >> > > > > > > Topic: *Test* Partition: 0 Leader: 0
> > > > > >> Replicas: 0
> > > > > >> > > > > > Isr: 0
> > > > > >> > > > > > > Topic: *Test* Partition: 1 Leader: 0
> > > > > >> Replicas: 0
> > > > > >> > > > > > Isr: 0
> > > > > >> > > > > > >
> > > > > >> > > > > > > Or
> > > > > >> > > > > > >
> > > > > >> > > > > > > Topic:*Heartbeat* PartitionCount:2
> > > > > ReplicationFactor:1
> > > > > >> > > > > Configs:
> > > > > >> > > > > > > Topic: *Heartbeat* Partition: 0
> Leader:
> > 0
> > > > > >> > > Replicas:
> > > > > >> > > > 0
> > > > > >> > > > > > > Isr: 0
> > > > > >> > > > > > > Topic: *Heartbeat* Partition: 1
> Leader:
> > 0
> > > > > >> > > Replicas:
> > > > > >> > > > 0
> > > > > >> > > > > > > Isr: 0
> > > > > >> > > > > > >
> > > > > >> > > > > > > Did you manually modify the topic name in ZK?
> > > > > >> > > > > > >
> > > > > >> > > > > > > Guozhang
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Tue, Oct 28, 2014 at 8:53 AM, Sameer Yami <
> > > > > >> sy...@ailive.net>
> > > > > >> > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Hi Guozhang,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > I was wondering if there is anything wrong in the
> > > above
> > > > > >> > > --describe
> > > > > >> > > > > > data.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > thanks
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Mon, Oct 27, 2014 at 2:56 PM, Sameer Yami <
> > > > > >> sy...@ailive.net
> > > > > >> > >
> > > > > >> > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > This is what I get -
> > > > > >> > > > > > > > > bin/kafka-topics.sh --zookeeper localhost:2181
> > > > > --describe
> > > > > >> > > > > > > > > Topic:Heartbeat PartitionCount:2
> > > > > >> ReplicationFactor:1
> > > > > >> > > > > > Configs:
> > > > > >> > > > > > > > > Topic: Test Partition: 0 Leader: 0
> > > > > >> > Replicas: 0
> > > > > >> > > > > > > Isr: 0
> > > > > >> > > > > > > > > Topic: Test Partition: 1 Leader: 0
> > > > > >> > Replicas: 0
> > > > > >> > > > > > > Isr: 0
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > The topics are being created at run-time.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > thanks
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Sun, Oct 26, 2014 at 4:49 PM, Guozhang Wang <
> > > > > >> > > > wangg...@gmail.com
> > > > > >> > > > > >
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >> Could you use the topic tools "--describe" to
> > check
> > > > if
> > > > > >> the
> > > > > >> > > topic
> > > > > >> > > > > > > Test's
> > > > > >> > > > > > > > >> partitions leaders are available?
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >>
> > > > > >> > >
> > http://kafka.apache.org/documentation.html#basic_ops_add_topic
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >> Guozhang
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >> On Fri, Oct 24, 2014 at 2:43 PM, Sameer Yami <
> > > > > >> > > sy...@ailive.net>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >> > That is not the case.
> > > > > >> > > > > > > > >> > This detailed error is below. Due to this
> > error ,
> > > > the
> > > > > >> > > producer
> > > > > >> > > > > is
> > > > > >> > > > > > > not
> > > > > >> > > > > > > > >> able
> > > > > >> > > > > > > > >> > to send messages
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main DEBUG
> > > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-52:
> > > > > >> > > > > > > > >> > Getting broker partition info for topic Test
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main INFO
> > > > > >> > > > > kafka.client.ClientUtils$-68:
> > > > > >> > > > > > > > >> Fetching
> > > > > >> > > > > > > > >> > metadata from broker
> > > > > id:0,host:54.215.200.61,port:9092
> > > > > >> > with
> > > > > >> > > > > > > > correlation
> > > > > >> > > > > > > > >> id
> > > > > >> > > > > > > > >> > 1 for 1 topic(s) Set(Test)
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,230 main DEBUG
> > > > > >> > > > > > > kafka.network.BlockingChannel-52:
> > > > > >> > > > > > > > >> > Created socket with SO_TIMEOUT = 10000
> > (requested
> > > > > >> 10000),
> > > > > >> > > > > > SO_RCVBUF
> > > > > >> > > > > > > =
> > > > > >> > > > > > > > >> > 187200 (requested -1), SO_SNDBUF = 102400
> > > > (requested
> > > > > >> > > 102400).
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,231 main INFO
> > > > > >> > > > > > kafka.producer.SyncProducer-68:
> > > > > >> > > > > > > > >> > Connected to 54.215.200.61:9092 for
> producing
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main INFO
> > > > > >> > > > > > kafka.producer.SyncProducer-68:
> > > > > >> > > > > > > > >> > Disconnecting from 54.215.200.61:9092
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main DEBUG
> > > > > >> > > > > kafka.client.ClientUtils$-52:
> > > > > >> > > > > > > > >> > Successfully fetched metadata for 1 topic(s)
> > > > > Set(Test)
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,257 main WARN
> > > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-83:
> > > > > >> > > > > > > > >> > Error while fetching metadata [{TopicMetadata
> > for
> > > > > topic
> > > > > >> > Test
> > > > > >> > > > ->
> > > > > >> > > > > > > > >> > No partition metadata for topic Test due to
> > > > > >> > > > > > > > >> > kafka.common.LeaderNotAvailableException}]
> for
> > > > topic
> > > > > >> > [Test]:
> > > > > >> > > > > class
> > > > > >> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,258 main ERROR
> > > > > >> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97:
> > > Failed
> > > > > to
> > > > > >> > > collate
> > > > > >> > > > > > > > messages
> > > > > >> > > > > > > > >> by
> > > > > >> > > > > > > > >> > topic, partition due to: Failed to fetch
> topic
> > > > > metadata
> > > > > >> > for
> > > > > >> > > > > topic:
> > > > > >> > > > > > > > Test
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >> > On Fri, Oct 24, 2014 at 1:11 PM, Guozhang
> Wang
> > <
> > > > > >> > > > > > wangg...@gmail.com>
> > > > > >> > > > > > > > >> wrote:
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >> > > Could you see if this is the case:
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
> > > > > >> > > > > > > > >> > > ?
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > > On Fri, Oct 24, 2014 at 11:35 AM, Sameer
> > Yami <
> > > > > >> > > > > sy...@ailive.net
> > > > > >> > > > > > >
> > > > > >> > > > > > > > >> wrote:
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > > > Hi,
> > > > > >> > > > > > > > >> > > >
> > > > > >> > > > > > > > >> > > > We have a Kafka / Zookeeper test setup
> on a
> > > > > single
> > > > > >> EC2
> > > > > >> > > > > > machine.
> > > > > >> > > > > > > > The
> > > > > >> > > > > > > > >> > > > host.name is the private ip address and
> > the
> > > > > >> > > > > > > advertised.host.name
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > >> > the
> > > > > >> > > > > > > > >> > > > public address in server.properties.
> > > > > >> > > > > > > > >> > > >
> > > > > >> > > > > > > > >> > > > We are trying to publish from outside EC2
> > > using
> > > > > the
> > > > > >> > > public
> > > > > >> > > > > ip
> > > > > >> > > > > > of
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > >> > EC2
> > > > > >> > > > > > > > >> > > > machine but get this error on the client
> > > side -
> > > > > >> > > > > > > > >> > > > No partition metadata for topic test due
> to
> > > > > >> > > > > > > > >> > > >
> kafka.common.LeaderNotAvailableException}]
> > > for
> > > > > >> topic
> > > > > >> > > > [test]:
> > > > > >> > > > > > > class
> > > > > >> > > > > > > > >> > > > kafka.common.LeaderNotAvailableException
> > > > > >> > > > > > > > >> > > >
> > > > > >> > > > > > > > >> > > > The zookeeper has the public ip stored in
> > > > > >> > > /brokers/ids/0.
> > > > > >> > > > > > > > >> > > > We have tried multiple combinations but
> it
> > > > works
> > > > > >> ONLY
> > > > > >> > > when
> > > > > >> > > > > the
> > > > > >> > > > > > > > >> > advertised
> > > > > >> > > > > > > > >> > > > host name is not used and the client is
> > > > internal
> > > > > to
> > > > > >> > EC2.
> > > > > >> > > > > > > > >> > > >
> > > > > >> > > > > > > > >> > > > Please let me know what the problem could
> > be.
> > > > > >> > > > > > > > >> > > >
> > > > > >> > > > > > > > >> > > > thanks
> > > > > >> > > > > > > > >> > > > Sameer
> > > > > >> > > > > > > > >> > > >
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > > --
> > > > > >> > > > > > > > >> > > -- Guozhang
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >> --
> > > > > >> > > > > > > > >> -- Guozhang
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > --
> > > > > >> > > > > > > -- Guozhang
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > --
> > > > > >> > > -- Guozhang
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> -- Guozhang
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
