Is it ok, if I send you directly?
On Tue, Nov 11, 2014 at 10:17 AM, Guozhang Wang <wangg...@gmail.com> wrote:
> Hi Sameer,
>
> I think apache mailing list has blocked your attachment. If it is too long
> to include in the email body could you paste it somewhere and give me the
> link?
>
> Guozhang
>
> On Tue, Nov 11, 2014 at 10:01 AM, Sameer Yami <sy...@ailive.net> wrote:
>
> > Hi Guozhang,
> >
> > I was wondering if you found anything wrong in the logs/
> >
> > thanks
> >
> >
> > On Fri, Nov 7, 2014 at 4:19 PM, Sameer Yami <sy...@ailive.net> wrote:
> >
> > > Hi Guozhang,
> > >
> > > Attached are the two logs with debug enabled.
> > >
> > > Thanks!
> > >
> > > On Fri, Nov 7, 2014 at 2:09 PM, Sameer Yami <sy...@ailive.net> wrote:
> > >
> > >> The version is kafka_2.10-0.8.1.1. It is not the latest trunk.
> > >> Will try enabling debug version.
> > >>
> > >> thanks
> > >>
> > >>
> > >> On Thu, Nov 6, 2014 at 9:37 PM, Guozhang Wang <wangg...@gmail.com>
> > wrote:
> > >>
> > >>> Sameer,
> > >>>
> > >>> The server logs do not contain any non-INFO logs, which is a bit
> wired.
> > >>> Did
> > >>> you deploy the current trunk of Kafka? Also could you enable DEBUG
> > level
> > >>> logging on Kafka brokers?
> > >>>
> > >>> Guozhang
> > >>>
> > >>> On Wed, Nov 5, 2014 at 3:50 PM, Sameer Yami <sy...@ailive.net>
> wrote:
> > >>>
> > >>> > The server.log was taken separately.
> > >>> > We ran the test again and the server and producer logs are below
> (to
> > >>> get
> > >>> > same timings).
> > >>> >
> > >>> >
> > >>> > Thanks!
> > >>> >
> > >>> >
> > >>> >
> > >>>
> >
> ------------------------------------------------------------------------------------
> > >>> >
> > >>> >
> > >>> >
> > >>> > Producer Logs -
> > >>> >
> > >>> >
> > >>> > 2014-11-05 23:38:58,693
> > >>> >
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680002 after 0ms
> > >>> > 2014-11-05 23:39:00,695
> > >>> >
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680002 after 0ms
> > >>> > 2014-11-05 23:39:02,696
> > >>> >
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680002 after 0ms
> > >>> > 2014-11-05 23:39:02,828 pool-13-thread-2 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Verifying properties
> > >>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property
> > auto.commit.interval.ms
> > >>> is
> > >>> > overridden to 1000
> > >>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property auto.offset.reset is
> > >>> > overridden to smallest
> > >>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property consumer.timeout.ms
> is
> > >>> > overridden to 10
> > >>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property group.id is
> overridden
> > >>> to
> > >>> > TestCheck
> > >>> > 2014-11-05 23:39:02,830 pool-13-thread-2 WARN
> > >>> > kafka.utils.VerifiableProperties-83: Property serializer.class is
> not
> > >>> valid
> > >>> > 2014-11-05 23:39:02,830 pool-13-thread-2 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property zookeeper.connect is
> > >>> > overridden to 172.31.25.198:2181
> > >>> > 2014-11-05 23:39:02,831 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Connecting to
> > >>> > zookeeper instance at 172.31.25.198:2181
> > >>> > 2014-11-05 23:39:02,831 pool-13-thread-2 DEBUG
> > >>> > org.I0Itec.zkclient.ZkConnection-63: Creating new ZookKeeper
> instance
> > >>> to
> > >>> > connect to 172.31.25.198:2181.
> > >>> > 2014-11-05 23:39:02,831 pool-13-thread-2 INFO
> > >>> > org.apache.zookeeper.ZooKeeper-379: Initiating client connection,
> > >>> > connectString=172.31.25.198:2181 sessionTimeout=6000
> > >>> > watcher=org.I0Itec.zkclient.ZkClient@3903b165
> > >>> > 2014-11-05 23:39:02,831 ZkClient-EventThread-29-172.31.25.198:2181
> > >>> INFO
> > >>> > org.I0Itec.zkclient.ZkEventThread-64: Starting ZkClient event
> thread.
> > >>> > 2014-11-05 23:39:02,831 pool-13-thread-1 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Verifying properties
> > >>> > 2014-11-05 23:39:02,836 pool-13-thread-2-SendThread() INFO
> > >>> > org.apache.zookeeper.ClientCnxn-1061: Opening socket connection to
> > >>> server /
> > >>> > 172.31.25.198:2181
> > >>> > 2014-11-05 23:39:02,836 pool-13-thread-1 WARN
> > >>> > kafka.utils.VerifiableProperties-83: Property batch.size is not
> valid
> > >>> > 2014-11-05 23:39:02,832 pool-13-thread-2 DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-878: Awaiting connection to Zookeeper
> > >>> server
> > >>> > 2014-11-05 23:39:02,836 pool-13-thread-1 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property
> > message.send.max.retries
> > >>> is
> > >>> > overridden to 10
> > >>> > 2014-11-05 23:39:02,836 pool-13-thread-2 DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-628: Waiting for keeper state
> > >>> SyncConnected
> > >>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property metadata.broker.list
> is
> > >>> > overridden to 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property retry.backoff.ms is
> > >>> > overridden to 1000
> > >>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property serializer.class is
> > >>> > overridden to kafka.serializer.StringEncoder
> > >>> > 2014-11-05 23:39:02,837
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > INFO org.apache.zookeeper.ClientCnxn-950: Socket connection
> > >>> established to
> > >>> > ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181,
> > >>> initiating
> > >>> > session
> > >>> > 2014-11-05 23:39:02,838
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-999: Session establishment
> > >>> request
> > >>> > sent on ip-172-31-25-198.us-west-1.compute.internal/
> > 172.31.25.198:2181
> > >>> > 2014-11-05 23:39:02,837 pool-13-thread-1 WARN
> > >>> > kafka.utils.VerifiableProperties-83: Property
> > zk.connectiontimeout.ms
> > >>> is
> > >>> > not valid
> > >>> > 2014-11-05 23:39:02,841
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > INFO org.apache.zookeeper.ClientCnxn-739: Session establishment
> > >>> complete on
> > >>> > server ip-172-31-25-198.us-west-1.compute.internal/
> > 172.31.25.198:2181,
> > >>> > sessionid = 0x1498251e8680003, negotiated timeout = 6000
> > >>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
> > >>> > state:SyncConnected type:None path:null
> > >>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread INFO
> > >>> > org.I0Itec.zkclient.ZkClient-449: zookeeper state changed
> > >>> (SyncConnected)
> > >>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-395: Leaving process event
> > >>> > 2014-11-05 23:39:02,841 pool-13-thread-2 DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-638: State is SyncConnected
> > >>> > 2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
> > >>> > kafka.utils.KafkaScheduler-52: Initializing task scheduler.
> > >>> > 2014-11-05 23:39:02,842 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting auto
> > >>> > committer every 1000 ms
> > >>> > 2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
> > >>> > kafka.utils.KafkaScheduler-52: Scheduling task
> > >>> kafka-consumer-autocommit
> > >>> > with initial delay 1000 ms and period 1000 ms.
> > >>> > 2014-11-05 23:39:02,843 pool-13-thread-2 DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], entering
> consume
> > >>> > 2014-11-05 23:39:02,843 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin
> > registering
> > >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
> > >>> > 2014-11-05 23:39:02,846
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 1,1 replyHeader:: 1,29,-101 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
> > >>> > response::
> > >>> > 2014-11-05 23:39:02,854
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 2,1 replyHeader:: 2,30,-101 request::
> > >>> > '/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
> > >>> > 2014-11-05 23:39:02,858
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 3,1 replyHeader:: 3,31,0 request::
> > >>> > '/consumers/TestCheck,,v{s{31,s{'world,'anyone}}},0 response::
> > >>> > '/consumers/TestCheck
> > >>> > 2014-11-05 23:39:02,861
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 4,1 replyHeader:: 4,32,0 request::
> > >>> > '/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
> > >>> > '/consumers/TestCheck/ids
> > >>> > 2014-11-05 23:39:02,866
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 5,1 replyHeader:: 5,33,0 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
> > >>> > response::
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
> > >>> > 2014-11-05 23:39:02,867 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end
> registering
> > >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
> > >>> > 2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Adding
> > >>> topicThreadId
> > >>> > (Test,TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0) and
> queue
> > >>> [] to
> > >>> > topicThreadIdAndQueues data structure
> > >>> > 2014-11-05 23:39:02,868
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting
> watcher
> > >>> > executor thread for consumer
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
> > >>> > 2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], adding topic
> > Test
> > >>> and
> > >>> > 1 streams to map.
> > >>> > 2014-11-05 23:39:02,870
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 6,3 replyHeader:: 6,33,0 request::
> > >>> > '/consumers/TestCheck/ids,T response::
> > >>> > s{32,32,1415230742859,1415230742859,0,1,0,0,0,1,33}
> > >>> > 2014-11-05 23:39:02,872
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 7,8 replyHeader:: 7,33,0 request::
> > >>> > '/consumers/TestCheck/ids,T response::
> > >>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> > >>> > 2014-11-05 23:39:02,873
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 8,3 replyHeader:: 8,33,-101 request::
> > >>> > '/brokers/topics/Test,T response::
> > >>> > 2014-11-05 23:39:02,873 pool-13-thread-2 DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-137: Subscribed data changes for
> > >>> > /brokers/topics/Test
> > >>> > 2014-11-05 23:39:02,873 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin
> > rebalancing
> > >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> > >>> > 2014-11-05 23:39:02,875 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.async.DefaultEventHandler-52: Handling 1 events
> > >>> > 2014-11-05 23:39:02,877
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 9,8 replyHeader:: 9,33,0 request::
> > >>> > '/brokers/ids,F response:: v{'0}
> > >>> > 2014-11-05 23:39:02,879
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 10,4 replyHeader:: 10,33,0 request::
> > >>> > '/brokers/ids/0,F response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> > >>> >
> > >>> > 2014-11-05 23:39:02,887
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 11,4 replyHeader:: 11,33,0 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> > >>> > response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> > >>> >
> > >>> > 2014-11-05 23:39:02,896
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 12,8 replyHeader:: 12,33,0 request::
> > >>> > '/consumers/TestCheck/ids,T response::
> > >>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> > >>> > 2014-11-05 23:39:02,898
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 13,4 replyHeader:: 13,33,0 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> > >>> > response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> > >>> >
> > >>> > 2014-11-05 23:39:02,908
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 14,8 replyHeader:: 14,33,0 request::
> > >>> > '/brokers/ids,F response:: v{'0}
> > >>> > 2014-11-05 23:39:02,910
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 15,4 replyHeader:: 15,33,0 request::
> > >>> > '/brokers/ids/0,F response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> > >>> >
> > >>> > 2014-11-05 23:39:02,917 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 0 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:02,917
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 16,4 replyHeader:: 16,33,-101 request::
> > >>> > '/brokers/topics/Test,T response::
> > >>> > 2014-11-05 23:39:02,920 pool-13-thread-2 DEBUG
> > >>> kafka.utils.ZkUtils$-52:
> > >>> > Partition map for /brokers/topics/Test is Map()
> > >>> > 2014-11-05 23:39:02,920 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ConsumerFetcherManager-68:
> > >>> > [ConsumerFetcherManager-1415230742842] Stopping leader finder
> thread
> > >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ConsumerFetcherManager-68:
> > >>> > [ConsumerFetcherManager-1415230742842] Stopping all fetchers
> > >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ConsumerFetcherManager-68:
> > >>> > [ConsumerFetcherManager-1415230742842] All connections stopped
> > >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all
> > >>> relevant
> > >>> > queues for this fetcher
> > >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 DEBUG
> > >>> > kafka.consumer.ConsumerIterator-52: Clearing the current data chunk
> > for
> > >>> > this consumer iterator
> > >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the
> data
> > >>> > chunks in all the consumer message iterators
> > >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all
> > >>> offsets
> > >>> > after clearing the fetcher queues
> > >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing
> > >>> partition
> > >>> > ownership
> > >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the
> > >>> following
> > >>> > partitions: ArrayBuffer() for topic Test with consumers:
> > >>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
> > >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 WARN
> > >>> > kafka.consumer.ZookeeperConsumerConnector-83:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], No broker
> > >>> partitions
> > >>> > consumed by consumer thread
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 for topic Test
> > >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the
> > cache
> > >>> > 2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per
> > >>> topic
> > >>> > cache Map(Test -> ArrayBuffer())
> > >>> > 2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per
> > >>> topic
> > >>> > cache Map(Test ->
> > >>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
> > >>> > 2014-11-05 23:39:02,923 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected
> > partitions :
> > >>> > 2014-11-05 23:39:02,927 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end
> rebalancing
> > >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> > >>> > 2014-11-05 23:39:02,928
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > >>> >
> > >>>
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > >>> > Starting
> > >>> > 2014-11-05 23:39:02,930 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:02,930 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:02,985
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-783: Got notification
> > >>> > sessionid:0x1498251e8680003
> > >>> > 2014-11-05 23:39:02,986
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-800: Got WatchedEvent
> > >>> > state:SyncConnected type:NodeCreated path:/brokers/topics/Test for
> > >>> > sessionid 0x1498251e8680003
> > >>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
> > >>> > state:SyncConnected type:NodeCreated path:/brokers/topics/Test
> > >>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> > >>> > org.I0Itec.zkclient.ZkEventThread-88: New event: ZkEvent[Data of
> > >>> > /brokers/topics/Test changed sent to
> > >>> >
> > >>> >
> > >>>
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
> > >>> > ]
> > >>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-395: Leaving process event
> > >>> > 2014-11-05 23:39:02,988 ZkClient-EventThread-29-172.31.25.198:2181
> > >>> DEBUG
> > >>> > org.I0Itec.zkclient.ZkEventThread-69: Delivering event #1
> > ZkEvent[Data
> > >>> of
> > >>> > /brokers/topics/Test changed sent to
> > >>> >
> > >>> >
> > >>>
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
> > >>> > ]
> > >>> > 2014-11-05 23:39:02,991
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 17,3 replyHeader:: 17,37,0 request::
> > >>> > '/brokers/topics/Test,T response::
> > >>> > s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
> > >>> > 2014-11-05 23:39:02,993
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 18,4 replyHeader:: 18,37,0 request::
> > >>> > '/brokers/topics/Test,T response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
> > >>> >
> > >>> > 2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Topic info for
> > >>> path
> > >>> > /brokers/topics/Test changed to
> > >>> > {"version":1,"partitions":{"1":[0],"0":[0]}}, triggering rebalance
> > >>> > 2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181
> > >>> DEBUG
> > >>> > org.I0Itec.zkclient.ZkEventThread-79: Delivering event #1 done
> > >>> > 2014-11-05 23:39:02,995
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin
> > rebalancing
> > >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> > >>> > 2014-11-05 23:39:02,996
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 19,8 replyHeader:: 19,37,0 request::
> > >>> > '/brokers/ids,F response:: v{'0}
> > >>> > 2014-11-05 23:39:02,998
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 20,4 replyHeader:: 20,37,0 request::
> > >>> > '/brokers/ids/0,F response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> > >>> >
> > >>> > 2014-11-05 23:39:03,008
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 21,4 replyHeader:: 21,37,0 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> > >>> > response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> > >>> >
> > >>> > 2014-11-05 23:39:03,014
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 22,8 replyHeader:: 22,37,0 request::
> > >>> > '/consumers/TestCheck/ids,T response::
> > >>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> > >>> > 2014-11-05 23:39:03,018
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 23,4 replyHeader:: 23,37,0 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> > >>> > response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> > >>> >
> > >>> > 2014-11-05 23:39:03,041
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 24,8 replyHeader:: 24,37,0 request::
> > >>> > '/brokers/ids,F response:: v{'0}
> > >>> > 2014-11-05 23:39:03,043
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 25,4 replyHeader:: 25,37,0 request::
> > >>> > '/brokers/ids/0,F response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> > >>> >
> > >>> > 2014-11-05 23:39:03,058
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 26,4 replyHeader:: 26,37,0 request::
> > >>> > '/brokers/topics/Test,T response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
> > >>> >
> > >>> > 2014-11-05 23:39:03,074
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.utils.ZkUtils$-52: Partition map for /brokers/topics/Test is
> > >>> Map(1 ->
> > >>> > List(0), 0 -> List(0))
> > >>> > 2014-11-05 23:39:03,075
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ConsumerFetcherManager-68:
> > >>> > [ConsumerFetcherManager-1415230742842] Stopping leader finder
> thread
> > >>> > 2014-11-05 23:39:03,082
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > >>> >
> > >>>
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > >>> > Shutting down
> > >>> > 2014-11-05 23:39:03,082 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:03,089
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > >>> >
> > >>>
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > >>> > Stopped
> > >>> > 2014-11-05 23:39:03,089
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > >>> >
> > >>>
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > >>> > Shutdown completed
> > >>> > 2014-11-05 23:39:03,089
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ConsumerFetcherManager-68:
> > >>> > [ConsumerFetcherManager-1415230742842] Stopping all fetchers
> > >>> > 2014-11-05 23:39:03,090
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ConsumerFetcherManager-68:
> > >>> > [ConsumerFetcherManager-1415230742842] All connections stopped
> > >>> > 2014-11-05 23:39:03,090
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all
> > >>> relevant
> > >>> > queues for this fetcher
> > >>> > 2014-11-05 23:39:03,090
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.ConsumerIterator-52: Clearing the current data chunk
> > for
> > >>> > this consumer iterator
> > >>> > 2014-11-05 23:39:03,091
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the
> data
> > >>> > chunks in all the consumer message iterators
> > >>> > 2014-11-05 23:39:03,091
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all
> > >>> offsets
> > >>> > after clearing the fetcher queues
> > >>> > 2014-11-05 23:39:03,091
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing
> > >>> partition
> > >>> > ownership
> > >>> > 2014-11-05 23:39:03,094 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:03,096
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the
> > >>> following
> > >>> > partitions: ArrayBuffer(0, 1) for topic Test with consumers:
> > >>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
> > >>> > 2014-11-05 23:39:03,098
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to
> > claim
> > >>> > partition 0
> > >>> > 2014-11-05 23:39:03,100
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 27,4 replyHeader:: 27,37,-101 request::
> > >>> > '/consumers/TestCheck/offsets/Test/0,F response::
> > >>> > 2014-11-05 23:39:03,116
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.PartitionTopicInfo-52: initial consumer offset of
> > >>> Test:0:
> > >>> > fetched offset = -1: consumed offset = -1 is -1
> > >>> > 2014-11-05 23:39:03,117
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.PartitionTopicInfo-52: initial fetch offset of
> Test:0:
> > >>> > fetched offset = -1: consumed offset = -1 is -1
> > >>> > 2014-11-05 23:39:03,118
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:0:
> fetched
> > >>> offset
> > >>> > = -1: consumed offset = -1 selected new offset -1
> > >>> > 2014-11-05 23:39:03,169
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to
> > claim
> > >>> > partition 1
> > >>> > 2014-11-05 23:39:03,171
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 28,4 replyHeader:: 28,37,-101 request::
> > >>> > '/consumers/TestCheck/offsets/Test/1,F response::
> > >>> > 2014-11-05 23:39:03,172
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.PartitionTopicInfo-52: initial consumer offset of
> > >>> Test:1:
> > >>> > fetched offset = -1: consumed offset = -1 is -1
> > >>> > 2014-11-05 23:39:03,172
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.PartitionTopicInfo-52: initial fetch offset of
> Test:1:
> > >>> > fetched offset = -1: consumed offset = -1 is -1
> > >>> > 2014-11-05 23:39:03,172
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:1:
> fetched
> > >>> offset
> > >>> > = -1: consumed offset = -1 selected new offset -1
> > >>> > 2014-11-05 23:39:03,175 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:03,175
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 29,1 replyHeader:: 29,38,-101 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> > >>> > response::
> > >>> > 2014-11-05 23:39:03,179
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 30,1 replyHeader:: 30,39,-101 request::
> > >>> > '/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0
> > >>> > response::
> > >>> > 2014-11-05 23:39:03,180 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:03,181 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 1 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:03,182 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:03,182 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:03,186
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 31,1 replyHeader:: 31,40,0 request::
> > >>> > '/consumers/TestCheck/owners,,v{s{31,s{'world,'anyone}}},0
> > response::
> > >>> > '/consumers/TestCheck/owners
> > >>> > 2014-11-05 23:39:03,189 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:03,190
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 32,1 replyHeader:: 32,41,0 request::
> > >>> > '/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0
> > >>> response::
> > >>> > '/consumers/TestCheck/owners/Test
> > >>> > 2014-11-05 23:39:03,190 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:03,191 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:03,192 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:03,193
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 33,1 replyHeader:: 33,42,0 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> > >>> > response:: '/consumers/TestCheck/owners/Test/1
> > >>> > 2014-11-05 23:39:03,193 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 10
> > >>> > 2014-11-05 23:39:03,193
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully
> > owned
> > >>> > partition 1 for topic Test
> > >>> > 2014-11-05 23:39:03,196
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 34,1 replyHeader:: 34,43,0 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/owners/Test/0,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> > >>> > response:: '/consumers/TestCheck/owners/Test/0
> > >>> > 2014-11-05 23:39:03,196
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully
> > owned
> > >>> > partition 0 for topic Test
> > >>> > 2014-11-05 23:39:03,196
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the
> > cache
> > >>> > 2014-11-05 23:39:03,197
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per
> > >>> topic
> > >>> > cache Map(Test -> ArrayBuffer(0, 1))
> > >>> > 2014-11-05 23:39:03,197
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per
> > >>> topic
> > >>> > cache Map(Test ->
> > >>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
> > >>> > 2014-11-05 23:39:03,197
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected
> > partitions :
> > >>> > Test:0: fetched offset = -1: consumed offset = -1,Test:1: fetched
> > >>> offset =
> > >>> > -1: consumed offset = -1
> > >>> > 2014-11-05 23:39:03,199
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > >>> >
> > >>>
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > >>> > Starting
> > >>> > 2014-11-05 23:39:03,199
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end
> rebalancing
> > >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> > >>> > 2014-11-05 23:39:03,202
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 35,8 replyHeader:: 35,43,0 request::
> > >>> > '/brokers/ids,F response:: v{'0}
> > >>> > 2014-11-05 23:39:03,204
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 36,4 replyHeader:: 36,43,0 request::
> > >>> > '/brokers/ids/0,F response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> > >>> >
> > >>> > 2014-11-05 23:39:03,214
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.utils.VerifiableProperties-68: Verifying properties
> > >>> > 2014-11-05 23:39:03,215
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.utils.VerifiableProperties-68: Property client.id is
> > >>> overridden
> > >>> > to TestCheck
> > >>> > 2014-11-05 23:39:03,215
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.utils.VerifiableProperties-68: Property
> > >>> metadata.broker.list is
> > >>> > overridden to x.x.x.x:9092
> > >>> > 2014-11-05 23:39:03,215
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.utils.VerifiableProperties-68: Property
> > request.timeout.ms
> > >>> is
> > >>> > overridden to 30000
> > >>> > 2014-11-05 23:39:03,217
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:x.x.x.x,port:9092 with correlation id 0 for 1 topic(s)
> > >>> Set(Test)
> > >>> > 2014-11-05 23:39:04,195 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 2 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:04,196 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:04,196 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:04,199 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:04,199 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:04,200 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:04,200 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:04,201 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 3 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:04,201 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:04,201 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:04,205 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:04,205 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:04,206 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:04,206 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:04,206 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 9
> > >>> > 2014-11-05 23:39:04,698
> > >>> >
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680002 after 0ms
> > >>> > 2014-11-05 23:39:05,205
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680003 after 0ms
> > >>> > 2014-11-05 23:39:05,207 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 4 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:05,208 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:05,208 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:05,211 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:05,211 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:05,212 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:05,212 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:05,212 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 5 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:05,213 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:05,213 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:05,217 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:05,217 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:05,217 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:05,217 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:05,218 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 8
> > >>> > 2014-11-05 23:39:06,218 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 6 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:06,219 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:06,219 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:06,224 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:06,224 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:06,225 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:06,225 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:06,226 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 7 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:06,226 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:06,226 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:06,230 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:06,230 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:06,231 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:06,231 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:06,231 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 7
> > >>> > 2014-11-05 23:39:06,699
> > >>> >
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680002 after 0ms
> > >>> > 2014-11-05 23:39:07,208
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680003 after 0ms
> > >>> > 2014-11-05 23:39:07,232 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 8 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:07,233 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:07,233 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:07,236 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:07,236 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:07,237 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:07,237 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:07,238 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 9 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:07,238 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:07,238 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:07,241 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:07,241 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:07,242 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:07,242 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:07,242 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 6
> > >>> > 2014-11-05 23:39:08,243 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 10 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:08,243 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:08,245 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:08,249 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:08,250 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:08,250 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:08,251 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 11 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:08,252 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:08,255 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:08,256 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:08,256 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:08,257 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:08,257 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 5
> > >>> > 2014-11-05 23:39:08,702
> > >>> >
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680002 after 0ms
> > >>> > 2014-11-05 23:39:09,211
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680003 after 0ms
> > >>> > 2014-11-05 23:39:09,257 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 12 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:09,258 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:09,258 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:09,261 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:09,262 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:09,263 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 13 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:09,264 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:09,264 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:09,268 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:09,268 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:09,269 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:09,269 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:09,269 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 4
> > >>> > 2014-11-05 23:39:10,270 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 14 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:10,271 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:10,271 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:10,275 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:10,275 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:10,275 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:10,276 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:10,276 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 15 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:10,277 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:10,277 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 3
> > >>> >
> > >>> >
> > >>> > Brokers Logs -
> > >>> >
> > >>> > [2014-11-05 23:39:02,983] INFO Topic creation
> > >>> > {"version":1,"partitions":{"1":[0],"0":[0]}}
> > (kafka.admin.AdminUtils$)
> > >>> > [2014-11-05 23:39:02,988] INFO [KafkaApi-0] Auto creation of topic
> > Test
> > >>> > with 2 partitions and replication factor 1 is successful!
> > >>> > (kafka.server.KafkaApis)
> > >>> > [2014-11-05 23:39:03,092] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:03,191] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:04,200] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:04,205] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:05,211] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:05,218] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:06,224] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:06,230] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:07,236] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:07,241] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Wed, Nov 5, 2014 at 3:05 PM, Guozhang Wang <wangg...@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> > > Sameer,
> > >>> > >
> > >>> > > Yes, this is the server log. But there seems no abnormal entries
> in
> > >>> it,
> > >>> > and
> > >>> > > it does not cover the same time range as the producer client
> > throwing
> > >>> > > LeaderNotAvailableException
> > >>> > > (it was 10/24, 14:30).
> > >>> > >
> > >>> > > The reason that I want to check the server log at that same
> reason
> > is
> > >>> > > that LeaderNotAvailableException
> > >>> > > is set by the brokers while handling requests, and its server log
> > >>> will
> > >>> > show
> > >>> > > what is the cause it set this error code.
> > >>> > >
> > >>> > > Guozhang
> > >>> > >
> > >>> > > On Tue, Nov 4, 2014 at 12:47 PM, Sameer Yami <sy...@ailive.net>
> > >>> wrote:
> > >>> > >
> > >>> > > > Hi Guozhang,
> > >>> > > >
> > >>> > > > This is the server.log -
> > >>> > > >
> > >>> > > > [2014-11-04 20:21:57,510] INFO Verifying properties
> > >>> > > > (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,545] INFO Property advertised.host.name
> is
> > >>> > > overridden
> > >>> > > > to x.x.x.x (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,545] INFO Property broker.id is
> overridden
> > >>> to 0
> > >>> > > > (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,545] INFO Property
> > controlled.shutdown.enable
> > >>> is
> > >>> > > > overridden to true (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,545] INFO Property host.name is
> overridden
> > to
> > >>> > > > 172.31.25.198 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,545] INFO Property log.cleaner.enable is
> > >>> > overridden
> > >>> > > to
> > >>> > > > false (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,546] INFO Property log.dirs is overridden
> to
> > >>> > > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,546] INFO Property
> > >>> > log.retention.check.interval.ms
> > >>> > > is
> > >>> > > > overridden to 60000 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,546] INFO Property log.retention.hours is
> > >>> > overridden
> > >>> > > > to 168 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,546] INFO Property log.segment.bytes is
> > >>> overridden
> > >>> > > to
> > >>> > > > 536870912 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,546] INFO Property num.io.threads is
> > >>> overridden
> > >>> > to 8
> > >>> > > > (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,547] INFO Property num.network.threads is
> > >>> > overridden
> > >>> > > > to 2 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,547] INFO Property num.partitions is
> > >>> overridden
> > >>> > to 2
> > >>> > > > (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,547] INFO Property port is overridden to
> > 9092
> > >>> > > > (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,547] INFO Property
> > >>> socket.receive.buffer.bytes is
> > >>> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,547] INFO Property
> socket.request.max.bytes
> > is
> > >>> > > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,547] INFO Property
> socket.send.buffer.bytes
> > is
> > >>> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,548] INFO Property zookeeper.connect is
> > >>> overridden
> > >>> > > to
> > >>> > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,548] INFO Property
> > >>> > zookeeper.connection.timeout.ms
> > >>> > > is
> > >>> > > > overridden to 1000000 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,563] INFO [Kafka Server 0], starting
> > >>> > > > (kafka.server.KafkaServer)
> > >>> > > > [2014-11-04 20:21:57,565] INFO [Kafka Server 0], Connecting to
> > >>> > zookeeper
> > >>> > > on
> > >>> > > > localhost:2181 (kafka.server.KafkaServer)
> > >>> > > > [2014-11-04 20:21:57,738] INFO Log directory '/tmp/kafka-logs'
> > not
> > >>> > found,
> > >>> > > > creating it. (kafka.log.LogManager)
> > >>> > > > [2014-11-04 20:21:57,748] INFO Starting log cleanup with a
> period
> > >>> of
> > >>> > > 60000
> > >>> > > > ms. (kafka.log.LogManager)
> > >>> > > > [2014-11-04 20:21:57,752] INFO Starting log flusher with a
> > default
> > >>> > period
> > >>> > > > of 9223372036854775807 ms. (kafka.log.LogManager)
> > >>> > > > [2014-11-04 20:21:57,783] INFO Awaiting socket connections on
> > >>> > > > ip-172-31-25-198.us-west-1.compute.internal:9092.
> > >>> > > (kafka.network.Acceptor)
> > >>> > > > [2014-11-04 20:21:57,784] INFO [Socket Server on Broker 0],
> > Started
> > >>> > > > (kafka.network.SocketServer)
> > >>> > > > [2014-11-04 20:21:57,852] INFO Will not load MX4J,
> mx4j-tools.jar
> > >>> is
> > >>> > not
> > >>> > > in
> > >>> > > > the classpath (kafka.utils.Mx4jLoader$)
> > >>> > > > [2014-11-04 20:21:57,884] INFO 0 successfully elected as leader
> > >>> > > > (kafka.server.ZookeeperLeaderElector)
> > >>> > > > [2014-11-04 20:21:57,967] INFO Registered broker 0 at path
> > >>> > /brokers/ids/0
> > >>> > > > with address x.x.x.x:9092. (kafka.utils.ZkUtils$)
> > >>> > > > [2014-11-04 20:21:57,981] INFO [Kafka Server 0], started
> > >>> > > > (kafka.server.KafkaServer)
> > >>> > > > [2014-11-04 20:21:58,050] INFO New leader is 0
> > >>> > > > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> > >>> > > > [2014-11-04 20:22:01,809] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:22:11,812] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:29,084] INFO Topic creation
> > >>> > > > {"version":1,"partitions":{"1":[0],"0":[0]}}
> > >>> (kafka.admin.AdminUtils$)
> > >>> > > > [2014-11-04 20:30:29,088] INFO [KafkaApi-0] Auto creation of
> > topic
> > >>> Test
> > >>> > > > with 2 partitions and replication factor 1 is successful!
> > >>> > > > (kafka.server.KafkaApis)
> > >>> > > > [2014-11-04 20:30:29,280] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:29,301] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:30,310] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:30,315] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:31,320] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:31,325] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:32,331] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:32,336] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:33,343] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:33,347] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:34,356] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:34,360] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:35,366] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:35,372] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:36,377] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:36,382] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:37,387] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:37,394] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:38,399] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:38,403] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:39,410] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:39,414] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:40,419] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:41,602] INFO Topic creation
> > >>> > > > {"version":1,"partitions":{"1":[0],"0":[0]}}
> > >>> (kafka.admin.AdminUtils$)
> > >>> > > >
> > >>> > > >
> > >>> > > >
> > >>> > > > thanks
> > >>> > > >
> > >>> > > >
> > >>> > > > On Fri, Oct 31, 2014 at 9:15 AM, Guozhang Wang <
> > wangg...@gmail.com
> > >>> >
> > >>> > > wrote:
> > >>> > > >
> > >>> > > > > The server.log should be the broker logs.
> > >>> > > > >
> > >>> > > > > Guozhang
> > >>> > > > >
> > >>> > > > > On Thu, Oct 30, 2014 at 11:32 AM, Sameer Yami <
> > sy...@ailive.net>
> > >>> > > wrote:
> > >>> > > > >
> > >>> > > > > > I do not see any other logs besides these.
> > >>> > > > > > I checked the log4j.properties and where the logs are
> getting
> > >>> > stored.
> > >>> > > > > > The only other logs (apart from the controller logs) in the
> > >>> > directory
> > >>> > > > are
> > >>> > > > > > zookeeper-gc.log,kafkaServer-gc.log, server.log and
> > >>> > state-change.log
> > >>> > > > > >
> > >>> > > > > > Thanks
> > >>> > > > > >
> > >>> > > > > > On Wed, Oct 29, 2014 at 4:13 PM, Guozhang Wang <
> > >>> wangg...@gmail.com
> > >>> > >
> > >>> > > > > wrote:
> > >>> > > > > >
> > >>> > > > > > > Sorry for getting late on this Sameer.
> > >>> > > > > > >
> > >>> > > > > > > The logs you showed above are from controller logs, do
> you
> > >>> see
> > >>> > > > brokers
> > >>> > > > > > logs
> > >>> > > > > > > besides that?
> > >>> > > > > > >
> > >>> > > > > > > Guozhang
> > >>> > > > > > >
> > >>> > > > > > > On Wed, Oct 29, 2014 at 3:40 PM, Sameer Yami <
> > >>> sy...@ailive.net>
> > >>> > > > wrote:
> > >>> > > > > > >
> > >>> > > > > > > > Hi Guozhang,
> > >>> > > > > > > >
> > >>> > > > > > > > Is there any clue in the above data as to what is
> wrong?
> > >>> > > > > > > >
> > >>> > > > > > > > thanks
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > > On Tue, Oct 28, 2014 at 3:45 PM, Sameer Yami <
> > >>> sy...@ailive.net
> > >>> > >
> > >>> > > > > wrote:
> > >>> > > > > > > >
> > >>> > > > > > > > > This is the one from the file specified in
> > >>> log4j.properties -
> > >>> > > > > > > > >
> > >>> > > > > > > > > [2014-10-28 22:38:36,082] DEBUG [TopicChangeListener
> on
> > >>> > > > Controller
> > >>> > > > > > 0]:
> > >>> > > > > > > > > Topic change listener fired for path /brokers/topics
> > with
> > >>> > > > children
> > >>> > > > > > Test
> > >>> > > > > > > > >
> > >>> (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > >>> > > > > > > > > [2014-10-28 22:38:36,094] INFO [TopicChangeListener
> on
> > >>> > > Controller
> > >>> > > > > 0]:
> > >>> > > > > > > New
> > >>> > > > > > > > > topics: [Set(Test)], deleted topics: [Set()], new
> > >>> partition
> > >>> > > > replica
> > >>> > > > > > > > > assignment [Map([Test,0] -> List(0), [Test,1] ->
> > >>> List(0))]
> > >>> > > > > > > > >
> > >>> (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > >>> > > > > > > > > [2014-10-28 22:38:36,094] INFO [Controller 0]: New
> > topic
> > >>> > > creation
> > >>> > > > > > > > callback
> > >>> > > > > > > > > for [Test,0],[Test,1]
> > (kafka.controller.KafkaController)
> > >>> > > > > > > > > [2014-10-28 22:38:36,095] INFO [Controller 0]: New
> > >>> partition
> > >>> > > > > creation
> > >>> > > > > > > > > callback for [Test,0],[Test,1]
> > >>> > > (kafka.controller.KafkaController)
> > >>> > > > > > > > > [2014-10-28 22:38:36,095] INFO [Partition state
> machine
> > >>> on
> > >>> > > > > Controller
> > >>> > > > > > > 0]:
> > >>> > > > > > > > > Invoking state change to NewPartition for partitions
> > >>> > > > > > [Test,0],[Test,1]
> > >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:38:36,109] INFO [Replica state machine
> > on
> > >>> > > > controller
> > >>> > > > > > 0]:
> > >>> > > > > > > > > Invoking state change to NewReplica for replicas
> > >>> > > > > > > > >
> > >>> > > > >
> > >>> [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > >>> > > > > > > > > (kafka.controller.ReplicaStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:38:36,111] INFO [Partition state
> machine
> > >>> on
> > >>> > > > > Controller
> > >>> > > > > > > 0]:
> > >>> > > > > > > > > Invoking state change to OnlinePartition for
> partitions
> > >>> > > > > > > [Test,0],[Test,1]
> > >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:38:36,111] DEBUG [Partition state
> > machine
> > >>> on
> > >>> > > > > > Controller
> > >>> > > > > > > > 0]:
> > >>> > > > > > > > > Live assigned replicas for partition [Test,0] are:
> > >>> [List(0)]
> > >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:38:36,112] DEBUG [Partition state
> > machine
> > >>> on
> > >>> > > > > > Controller
> > >>> > > > > > > > 0]:
> > >>> > > > > > > > > Initializing leader and isr for partition [Test,0] to
> > >>> > > > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:38:36,141] DEBUG [Partition state
> > machine
> > >>> on
> > >>> > > > > > Controller
> > >>> > > > > > > > 0]:
> > >>> > > > > > > > > Live assigned replicas for partition [Test,1] are:
> > >>> [List(0)]
> > >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:38:36,142] DEBUG [Partition state
> > machine
> > >>> on
> > >>> > > > > > Controller
> > >>> > > > > > > > 0]:
> > >>> > > > > > > > > Initializing leader and isr for partition [Test,1] to
> > >>> > > > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:38:36,150] INFO [Replica state machine
> > on
> > >>> > > > controller
> > >>> > > > > > 0]:
> > >>> > > > > > > > > Invoking state change to OnlineReplica for replicas
> > >>> > > > > > > > >
> > >>> > > > >
> > >>> [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > >>> > > > > > > > > (kafka.controller.ReplicaStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:39:28,160] ERROR
> > >>> > > > > > [Controller-0-to-broker-0-send-thread],
> > >>> > > > > > > > > Controller 0's connection to broker
> > >>> > id:0,host:x.x.x.x,port:9092
> > >>> > > > was
> > >>> > > > > > > > > unsuccessful (kafka.controller.RequestSendThread)
> > >>> > > > > > > > > java.net.ConnectException: Connection timed out
> > >>> > > > > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > >>> > > > > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > >>> > > > > > > > > at sun.nio.ch.Net.connect(Net.java:449)
> > >>> > > > > > > > > at
> > >>> > > > > >
> > >>> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
> > >>> > > > > > > > > at
> > >>> > > > > >
> > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > >>> > > > > > > > > at
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
> > >>> > > > > > > > > at
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
> > >>> > > > > > > > > at
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > >>> > > > > > > > > at
> > >>> > > > > >
> > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > >>> > > > > > > > > [2014-10-28 22:39:28,460] ERROR
> > >>> > > > > > [Controller-0-to-broker-0-send-thread],
> > >>> > > > > > > > > Controller 0 epoch 1 failed to send LeaderAndIsr
> > request
> > >>> with
> > >>> > > > > > > correlation
> > >>> > > > > > > > > id 7 to broker id:0,host:x.x.x.x,port:9092.
> > Reconnecting
> > >>> to
> > >>> > > > broker.
> > >>> > > > > > > > > (kafka.controller.RequestSendThread)
> > >>> > > > > > > > > java.nio.channels.ClosedChannelException
> > >>> > > > > > > > > at
> > >>> > > > kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
> > >>> > > > > > > > > at
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
> > >>> > > > > > > > > at
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > >>> > > > > > > > > at
> > >>> > > > > >
> > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > >>> > > > > > > > >
> > >>> > > > > > > > > On Tue, Oct 28, 2014 at 3:09 PM, Guozhang Wang <
> > >>> > > > wangg...@gmail.com
> > >>> > > > > >
> > >>> > > > > > > > wrote:
> > >>> > > > > > > > >
> > >>> > > > > > > > >> Sameer, the one you sent me is from the producer
> logs,
> > >>> not
> > >>> > the
> > >>> > > > > > broker
> > >>> > > > > > > > >> logs.
> > >>> > > > > > > > >> And /tmp/kafka-logs actually store the message log
> > >>> (i.e. the
> > >>> > > > > "data")
> > >>> > > > > > > > >> instead of the server logs. Their locations should
> be
> > >>> > > specified
> > >>> > > > in
> > >>> > > > > > > your
> > >>> > > > > > > > >> log4j.properties file.
> > >>> > > > > > > > >>
> > >>> > > > > > > > >> On Tue, Oct 28, 2014 at 2:51 PM, Sameer Yami <
> > >>> > > sy...@ailive.net>
> > >>> > > > > > > wrote:
> > >>> > > > > > > > >>
> > >>> > > > > > > > >> > Is this what you are looking for -
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,146 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > >>> > > > 172.31.16.31:9092
> > >>> > > > > > for
> > >>> > > > > > > > >> > producing
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > >>> > > > > > > 172.31.16.31:9092
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> > >>> > metadata
> > >>> > > > for
> > >>> > > > > 1
> > >>> > > > > > > > >> topic(s)
> > >>> > > > > > > > >> > Set(Test)
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 WARN
> > >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> > >>> > fetching
> > >>> > > > > > metadata
> > >>> > > > > > > > >> > [{TopicMetadata for topic Test ->
> > >>> > > > > > > > >> > No partition metadata for topic Test due to
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for
> topic
> > >>> > [Test]:
> > >>> > > > > class
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting
> > broker
> > >>> > > > partition
> > >>> > > > > > info
> > >>> > > > > > > > for
> > >>> > > > > > > > >> > topic Test
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata
> from
> > >>> > broker
> > >>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation
> > id 9
> > >>> > for 1
> > >>> > > > > > > topic(s)
> > >>> > > > > > > > >> > Set(Test)
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket
> > with
> > >>> > > > > SO_TIMEOUT =
> > >>> > > > > > > > 10000
> > >>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested
> > -1),
> > >>> > > > SO_SNDBUF
> > >>> > > > > =
> > >>> > > > > > > > 102400
> > >>> > > > > > > > >> > (requested 102400).
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,152 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > >>> > > > 172.31.16.31:9092
> > >>> > > > > > for
> > >>> > > > > > > > >> > producing
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,154 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > >>> > > > > > > 172.31.16.31:9092
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> > >>> > metadata
> > >>> > > > for
> > >>> > > > > 1
> > >>> > > > > > > > >> topic(s)
> > >>> > > > > > > > >> > Set(Test)
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 WARN
> > >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> > >>> > fetching
> > >>> > > > > > metadata
> > >>> > > > > > > > >> > [{TopicMetadata for topic Test ->
> > >>> > > > > > > > >> > No partition metadata for topic Test due to
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for
> topic
> > >>> > [Test]:
> > >>> > > > > class
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 ERROR
> > >>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97:
> Failed
> > to
> > >>> > > collate
> > >>> > > > > > > > messages
> > >>> > > > > > > > >> by
> > >>> > > > > > > > >> > topic, partition due to: Failed to fetch topic
> > >>> metadata
> > >>> > for
> > >>> > > > > topic:
> > >>> > > > > > > > Test
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,156 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back
> > off
> > >>> for
> > >>> > > 1000
> > >>> > > > > ms
> > >>> > > > > > > > before
> > >>> > > > > > > > >> > retrying send. Remaining retries = 6
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,156 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata
> from
> > >>> > broker
> > >>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation
> id
> > >>> 10
> > >>> > for
> > >>> > > 1
> > >>> > > > > > > topic(s)
> > >>> > > > > > > > >> > Set(Test)
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket
> > with
> > >>> > > > > SO_TIMEOUT =
> > >>> > > > > > > > 10000
> > >>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested
> > -1),
> > >>> > > > SO_SNDBUF
> > >>> > > > > =
> > >>> > > > > > > > 102400
> > >>> > > > > > > > >> > (requested 102400).
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > >>> > > > 172.31.16.31:9092
> > >>> > > > > > for
> > >>> > > > > > > > >> > producing
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,160 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > >>> > > > > > > 172.31.16.31:9092
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> > >>> > metadata
> > >>> > > > for
> > >>> > > > > 1
> > >>> > > > > > > > >> topic(s)
> > >>> > > > > > > > >> > Set(Test)
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 WARN
> > >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> > >>> > fetching
> > >>> > > > > > metadata
> > >>> > > > > > > > >> > [{TopicMetadata for topic Test ->
> > >>> > > > > > > > >> > No partition metadata for topic Test due to
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for
> topic
> > >>> > [Test]:
> > >>> > > > > class
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting
> > broker
> > >>> > > > partition
> > >>> > > > > > info
> > >>> > > > > > > > for
> > >>> > > > > > > > >> > topic Test
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata
> from
> > >>> > broker
> > >>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation
> id
> > >>> 11
> > >>> > for
> > >>> > > 1
> > >>> > > > > > > topic(s)
> > >>> > > > > > > > >> > Set(Test)
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket
> > with
> > >>> > > > > SO_TIMEOUT =
> > >>> > > > > > > > 10000
> > >>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested
> > -1),
> > >>> > > > SO_SNDBUF
> > >>> > > > > =
> > >>> > > > > > > > 102400
> > >>> > > > > > > > >> > (requested 102400).
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,163 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > >>> > > > 172.31.16.31:9092
> > >>> > > > > > for
> > >>> > > > > > > > >> > producing
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > >>> > > > > > > 172.31.16.31:9092
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> > >>> > metadata
> > >>> > > > for
> > >>> > > > > 1
> > >>> > > > > > > > >> topic(s)
> > >>> > > > > > > > >> > Set(Test)
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 WARN
> > >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> > >>> > fetching
> > >>> > > > > > metadata
> > >>> > > > > > > > >> > [{TopicMetadata for topic Test ->
> > >>> > > > > > > > >> > No partition metadata for topic Test due to
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for
> topic
> > >>> > [Test]:
> > >>> > > > > class
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 ERROR
> > >>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97:
> Failed
> > to
> > >>> > > collate
> > >>> > > > > > > > messages
> > >>> > > > > > > > >> by
> > >>> > > > > > > > >> > topic, partition due to: Failed to fetch topic
> > >>> metadata
> > >>> > for
> > >>> > > > > topic:
> > >>> > > > > > > > Test
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back
> > off
> > >>> for
> > >>> > > 1000
> > >>> > > > > ms
> > >>> > > > > > > > before
> > >>> > > > > > > > >> > retrying send. Remaining retries = 5
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > As for /tmp/kafka-logs, I can see only these
> files -
> > >>> > > > > > > > >> > recovery-point-offset-checkpoint
> > >>> > > > replication-offset-checkpoint
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > I do not see the usual Kafka log files here.
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > thanks
> > >>> > > > > > > > >> > Sameer
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > On Tue, Oct 28, 2014 at 1:23 PM, Guozhang Wang <
> > >>> > > > > > wangg...@gmail.com>
> > >>> > > > > > > > >> wrote:
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > > Sameer, could you check broker logs and see if
> it
> > >>> has
> > >>> > > > similar
> > >>> > > > > > > > >> > > LeaderNotAvailable exceptions?
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > Guozhang
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > On Tue, Oct 28, 2014 at 10:32 AM, Sameer Yami <
> > >>> > > > > sy...@ailive.net
> > >>> > > > > > >
> > >>> > > > > > > > >> wrote:
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > Yes. I can telnet from outside using the
> public
> > >>> ip of
> > >>> > > the
> > >>> > > > > > > instance
> > >>> > > > > > > > >> and
> > >>> > > > > > > > >> > > port
> > >>> > > > > > > > >> > > > of 9092
> > >>> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > On Tue, Oct 28, 2014 at 10:08 AM, Roger
> Hoover <
> > >>> > > > > > > > >> roger.hoo...@gmail.com
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > wrote:
> > >>> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > Just a guess but could it be a firewall
> issue?
> > >>> Did
> > >>> > > you
> > >>> > > > > > enable
> > >>> > > > > > > > >> > > > connections
> > >>> > > > > > > > >> > > > > to port 9092 from outside EC2 in a security
> > >>> group?
> > >>> > > Can
> > >>> > > > > you
> > >>> > > > > > > > >> telnet to
> > >>> > > > > > > > >> > > > each
> > >>> > > > > > > > >> > > > > broker IP and port?
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > > > On Tue, Oct 28, 2014 at 10:01 AM, Sameer
> Yami
> > <
> > >>> > > > > > > sy...@ailive.net
> > >>> > > > > > > > >
> > >>> > > > > > > > >> > > wrote:
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > > > > There was a typo earlier.
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > This is the output -
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > Topic:Test PartitionCount:2
> > >>> ReplicationFactor:1
> > >>> > > > > > > Configs:
> > >>> > > > > > > > >> > > > > > Topic: Test Partition: 0
> > >>> Leader: 0
> > >>> > > > > > > Replicas:
> > >>> > > > > > > > 0
> > >>> > > > > > > > >> > > > Isr: 0
> > >>> > > > > > > > >> > > > > > Topic: Test Partition: 1
> > >>> Leader: 0
> > >>> > > > > > > Replicas:
> > >>> > > > > > > > 0
> > >>> > > > > > > > >> > > > Isr: 0
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > thanks
> > >>> > > > > > > > >> > > > > > Sameer
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > On Tue, Oct 28, 2014 at 9:40 AM, Guozhang
> > >>> Wang <
> > >>> > > > > > > > >> wangg...@gmail.com
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > wrote:
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > > Sameer,
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > The topic name is not consistent from
> your
> > >>> > > describe
> > >>> > > > > > > outputs.
> > >>> > > > > > > > >> > > Normally
> > >>> > > > > > > > >> > > > > > they
> > >>> > > > > > > > >> > > > > > > should be either
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > Topic:*Test* PartitionCount:2
> > >>> > > > > ReplicationFactor:1
> > >>> > > > > > > > >> > > Configs:
> > >>> > > > > > > > >> > > > > > > Topic: *Test* Partition: 0
> > >>> > Leader: 0
> > >>> > > > > > > > >> Replicas: 0
> > >>> > > > > > > > >> > > > > > Isr: 0
> > >>> > > > > > > > >> > > > > > > Topic: *Test* Partition: 1
> > >>> > Leader: 0
> > >>> > > > > > > > >> Replicas: 0
> > >>> > > > > > > > >> > > > > > Isr: 0
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > Or
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > Topic:*Heartbeat* PartitionCount:2
> > >>> > > > > > > > ReplicationFactor:1
> > >>> > > > > > > > >> > > > > Configs:
> > >>> > > > > > > > >> > > > > > > Topic: *Heartbeat*
> Partition: 0
> > >>> > > > Leader:
> > >>> > > > > 0
> > >>> > > > > > > > >> > > Replicas:
> > >>> > > > > > > > >> > > > 0
> > >>> > > > > > > > >> > > > > > > Isr: 0
> > >>> > > > > > > > >> > > > > > > Topic: *Heartbeat*
> Partition: 1
> > >>> > > > Leader:
> > >>> > > > > 0
> > >>> > > > > > > > >> > > Replicas:
> > >>> > > > > > > > >> > > > 0
> > >>> > > > > > > > >> > > > > > > Isr: 0
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > Did you manually modify the topic name
> in
> > >>> ZK?
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > Guozhang
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > On Tue, Oct 28, 2014 at 8:53 AM, Sameer
> > >>> Yami <
> > >>> > > > > > > > >> sy...@ailive.net>
> > >>> > > > > > > > >> > > > wrote:
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > > Hi Guozhang,
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > > I was wondering if there is anything
> > >>> wrong in
> > >>> > > the
> > >>> > > > > > above
> > >>> > > > > > > > >> > > --describe
> > >>> > > > > > > > >> > > > > > data.
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > > thanks
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > > On Mon, Oct 27, 2014 at 2:56 PM,
> Sameer
> > >>> Yami <
> > >>> > > > > > > > >> sy...@ailive.net
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > wrote:
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > This is what I get -
> > >>> > > > > > > > >> > > > > > > > > bin/kafka-topics.sh --zookeeper
> > >>> > > localhost:2181
> > >>> > > > > > > > --describe
> > >>> > > > > > > > >> > > > > > > > > Topic:Heartbeat PartitionCount:2
> > >>> > > > > > > > >> ReplicationFactor:1
> > >>> > > > > > > > >> > > > > > Configs:
> > >>> > > > > > > > >> > > > > > > > > Topic: Test Partition: 0
> > >>> > > Leader: 0
> > >>> > > > > > > > >> > Replicas: 0
> > >>> > > > > > > > >> > > > > > > Isr: 0
> > >>> > > > > > > > >> > > > > > > > > Topic: Test Partition: 1
> > >>> > > Leader: 0
> > >>> > > > > > > > >> > Replicas: 0
> > >>> > > > > > > > >> > > > > > > Isr: 0
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > The topics are being created at
> > >>> run-time.
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > thanks
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > On Sun, Oct 26, 2014 at 4:49 PM,
> > >>> Guozhang
> > >>> > > Wang <
> > >>> > > > > > > > >> > > > wangg...@gmail.com
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > > > wrote:
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > >> Could you use the topic tools
> > >>> "--describe"
> > >>> > to
> > >>> > > > > check
> > >>> > > > > > > if
> > >>> > > > > > > > >> the
> > >>> > > > > > > > >> > > topic
> > >>> > > > > > > > >> > > > > > > Test's
> > >>> > > > > > > > >> > > > > > > > >> partitions leaders are available?
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > >
> > >>> > > > >
> http://kafka.apache.org/documentation.html#basic_ops_add_topic
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >> Guozhang
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >> On Fri, Oct 24, 2014 at 2:43 PM,
> > Sameer
> > >>> > Yami
> > >>> > > <
> > >>> > > > > > > > >> > > sy...@ailive.net>
> > >>> > > > > > > > >> > > > > > > wrote:
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >> > That is not the case.
> > >>> > > > > > > > >> > > > > > > > >> > This detailed error is below. Due
> > to
> > >>> this
> > >>> > > > > error ,
> > >>> > > > > > > the
> > >>> > > > > > > > >> > > producer
> > >>> > > > > > > > >> > > > > is
> > >>> > > > > > > > >> > > > > > > not
> > >>> > > > > > > > >> > > > > > > > >> able
> > >>> > > > > > > > >> > > > > > > > >> > to send messages
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main
> DEBUG
> > >>> > > > > > > > >> > > > > > > > >>
> > kafka.producer.BrokerPartitionInfo-52:
> > >>> > > > > > > > >> > > > > > > > >> > Getting broker partition info for
> > >>> topic
> > >>> > > Test
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main
> INFO
> > >>> > > > > > > > >> > > > > kafka.client.ClientUtils$-68:
> > >>> > > > > > > > >> > > > > > > > >> Fetching
> > >>> > > > > > > > >> > > > > > > > >> > metadata from broker
> > >>> > > > > > > > id:0,host:54.215.200.61,port:9092
> > >>> > > > > > > > >> > with
> > >>> > > > > > > > >> > > > > > > > correlation
> > >>> > > > > > > > >> > > > > > > > >> id
> > >>> > > > > > > > >> > > > > > > > >> > 1 for 1 topic(s) Set(Test)
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,230 main
> DEBUG
> > >>> > > > > > > > >> > > > > > > kafka.network.BlockingChannel-52:
> > >>> > > > > > > > >> > > > > > > > >> > Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > > > > (requested
> > >>> > > > > > > > >> 10000),
> > >>> > > > > > > > >> > > > > > SO_RCVBUF
> > >>> > > > > > > > >> > > > > > > =
> > >>> > > > > > > > >> > > > > > > > >> > 187200 (requested -1), SO_SNDBUF
> =
> > >>> 102400
> > >>> > > > > > > (requested
> > >>> > > > > > > > >> > > 102400).
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,231 main
> INFO
> > >>> > > > > > > > >> > > > > > kafka.producer.SyncProducer-68:
> > >>> > > > > > > > >> > > > > > > > >> > Connected to 54.215.200.61:9092
> > for
> > >>> > > > producing
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main
> INFO
> > >>> > > > > > > > >> > > > > > kafka.producer.SyncProducer-68:
> > >>> > > > > > > > >> > > > > > > > >> > Disconnecting from
> > >>> 54.215.200.61:9092
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main
> DEBUG
> > >>> > > > > > > > >> > > > > kafka.client.ClientUtils$-52:
> > >>> > > > > > > > >> > > > > > > > >> > Successfully fetched metadata
> for 1
> > >>> > > topic(s)
> > >>> > > > > > > > Set(Test)
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,257 main
> WARN
> > >>> > > > > > > > >> > > > > > > > >>
> > kafka.producer.BrokerPartitionInfo-83:
> > >>> > > > > > > > >> > > > > > > > >> > Error while fetching metadata
> > >>> > > [{TopicMetadata
> > >>> > > > > for
> > >>> > > > > > > > topic
> > >>> > > > > > > > >> > Test
> > >>> > > > > > > > >> > > > ->
> > >>> > > > > > > > >> > > > > > > > >> > No partition metadata for topic
> > Test
> > >>> due
> > >>> > to
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> > kafka.common.LeaderNotAvailableException}]
> > >>> > > > for
> > >>> > > > > > > topic
> > >>> > > > > > > > >> > [Test]:
> > >>> > > > > > > > >> > > > > class
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> kafka.common.LeaderNotAvailableException
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,258 main
> ERROR
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> > > kafka.producer.async.DefaultEventHandler-97:
> > >>> > > > > > Failed
> > >>> > > > > > > > to
> > >>> > > > > > > > >> > > collate
> > >>> > > > > > > > >> > > > > > > > messages
> > >>> > > > > > > > >> > > > > > > > >> by
> > >>> > > > > > > > >> > > > > > > > >> > topic, partition due to: Failed
> to
> > >>> fetch
> > >>> > > > topic
> > >>> > > > > > > > metadata
> > >>> > > > > > > > >> > for
> > >>> > > > > > > > >> > > > > topic:
> > >>> > > > > > > > >> > > > > > > > Test
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> > > > > > > > >> > > > > > > > >> > On Fri, Oct 24, 2014 at 1:11 PM,
> > >>> Guozhang
> > >>> > > > Wang
> > >>> > > > > <
> > >>> > > > > > > > >> > > > > > wangg...@gmail.com>
> > >>> > > > > > > > >> > > > > > > > >> wrote:
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> > > > > > > > >> > > > > > > > >> > > Could you see if this is the
> > case:
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > >
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >>
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
> > >>> > > > > > > > >> > > > > > > > >> > > ?
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> > > On Fri, Oct 24, 2014 at 11:35
> AM,
> > >>> > Sameer
> > >>> > > > > Yami <
> > >>> > > > > > > > >> > > > > sy...@ailive.net
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > > >> wrote:
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> > > > Hi,
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > > > > >> > > > We have a Kafka / Zookeeper
> > test
> > >>> > setup
> > >>> > > > on a
> > >>> > > > > > > > single
> > >>> > > > > > > > >> EC2
> > >>> > > > > > > > >> > > > > > machine.
> > >>> > > > > > > > >> > > > > > > > The
> > >>> > > > > > > > >> > > > > > > > >> > > > host.name is the private ip
> > >>> address
> > >>> > > and
> > >>> > > > > the
> > >>> > > > > > > > >> > > > > > > advertised.host.name
> > >>> > > > > > > > >> > > > > > > > is
> > >>> > > > > > > > >> > > > > > > > >> > the
> > >>> > > > > > > > >> > > > > > > > >> > > > public address in
> > >>> server.properties.
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > > > > >> > > > We are trying to publish from
> > >>> outside
> > >>> > > EC2
> > >>> > > > > > using
> > >>> > > > > > > > the
> > >>> > > > > > > > >> > > public
> > >>> > > > > > > > >> > > > > ip
> > >>> > > > > > > > >> > > > > > of
> > >>> > > > > > > > >> > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > >> > EC2
> > >>> > > > > > > > >> > > > > > > > >> > > > machine but get this error on
> > the
> > >>> > > client
> > >>> > > > > > side -
> > >>> > > > > > > > >> > > > > > > > >> > > > No partition metadata for
> topic
> > >>> test
> > >>> > > due
> > >>> > > > to
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > > kafka.common.LeaderNotAvailableException}]
> > >>> > > > > > for
> > >>> > > > > > > > >> topic
> > >>> > > > > > > > >> > > > [test]:
> > >>> > > > > > > > >> > > > > > > class
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > kafka.common.LeaderNotAvailableException
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > > > > >> > > > The zookeeper has the public
> ip
> > >>> > stored
> > >>> > > in
> > >>> > > > > > > > >> > > /brokers/ids/0.
> > >>> > > > > > > > >> > > > > > > > >> > > > We have tried multiple
> > >>> combinations
> > >>> > but
> > >>> > > > it
> > >>> > > > > > > works
> > >>> > > > > > > > >> ONLY
> > >>> > > > > > > > >> > > when
> > >>> > > > > > > > >> > > > > the
> > >>> > > > > > > > >> > > > > > > > >> > advertised
> > >>> > > > > > > > >> > > > > > > > >> > > > host name is not used and the
> > >>> client
> > >>> > is
> > >>> > > > > > > internal
> > >>> > > > > > > > to
> > >>> > > > > > > > >> > EC2.
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > > > > >> > > > Please let me know what the
> > >>> problem
> > >>> > > could
> > >>> > > > > be.
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > > > > >> > > > thanks
> > >>> > > > > > > > >> > > > > > > > >> > > > Sameer
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> > > --
> > >>> > > > > > > > >> > > > > > > > >> > > -- Guozhang
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >> --
> > >>> > > > > > > > >> > > > > > > > >> -- Guozhang
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > --
> > >>> > > > > > > > >> > > > > > > -- Guozhang
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > >
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > --
> > >>> > > > > > > > >> > > -- Guozhang
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >>
> > >>> > > > > > > > >>
> > >>> > > > > > > > >>
> > >>> > > > > > > > >> --
> > >>> > > > > > > > >> -- Guozhang
> > >>> > > > > > > > >>
> > >>> > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > > --
> > >>> > > > > > > -- Guozhang
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > --
> > >>> > > > > -- Guozhang
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > > --
> > >>> > > -- Guozhang
> > >>> > >
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> -- Guozhang
> > >>>
> > >>
> > >>
> > >
> >
>
>
>
> --
> -- Guozhang
>
