This might be a issue of KAFKA-992. The title only includes Server but this
bug actually applies to Consumers.
Which version are you using?
Guozhang
On Sun, Jan 12, 2014 at 8:00 AM, Seshadri, Balaji
<balaji.sesha...@dish.com>wrote:
> Please see log.
>
> seshbal:tm1mwdpl03-/apps/tc/tm1-deshdpconsumer101/apache-tomcat-7.0.42/logs
> >ls -1 catalina*|xargs -exec grep "Expired"
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,265} DEBUG
> [catalina-exec-3-EventThread] (ZkClient.java:351) - Received event:
> WatchedEvent state:Expired type:None path:null
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,265} DEBUG
> [catalina-exec-6-EventThread] (ZkClient.java:351) - Received event:
> WatchedEvent state:Expired type:None path:null
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,265} INFO
> [catalina-exec-6-EventThread] (ZkClient.java:449) - zookeeper state
> changed (Expired)
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,265} INFO
> [catalina-exec-3-EventThread] (ZkClient.java:449) - zookeeper state
> changed (Expired)
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,265} DEBUG
> [catalina-exec-3-EventThread] (ZkEventThread.java:88) - New event:
> ZkEvent[State changed to Expired sent to
> kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@152e95fa
> ]
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,265} DEBUG
> [catalina-exec-6-EventThread] (ZkEventThread.java:88) - New event:
> ZkEvent[State changed to Expired sent to
> kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@4090755c
> ]
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,265} DEBUG [
> ZkClient-EventThread-55-tvip-m1-mw-zookeeper.dish.com:2181]
> (ZkEventThread.java:69) - Delivering event #22 ZkEvent[State changed to
> Expired sent to
> kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@152e95fa
> ]
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,265} DEBUG [
> ZkClient-EventThread-90-tvip-m1-mw-zookeeper.dish.com:2181]
> (ZkEventThread.java:69) - Delivering event #23 ZkEvent[State changed to
> Expired sent to
> kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@4090755c
> ]
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,267} DEBUG
> [catalina-exec-10-EventThread] (ZkClient.java:351) - Received event:
> WatchedEvent state:Expired type:None path:null
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,267} INFO
> [catalina-exec-10-EventThread] (ZkClient.java:449) - zookeeper state
> changed (Expired)
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,267} DEBUG
> [catalina-exec-5-EventThread] (ZkClient.java:351) - Received event:
> WatchedEvent state:Expired type:None path:null
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,267} DEBUG
> [catalina-exec-10-EventThread] (ZkEventThread.java:88) - New event:
> ZkEvent[State changed to Expired sent to
> kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@1901a132
> ]
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,267} INFO
> [catalina-exec-5-EventThread] (ZkClient.java:449) - zookeeper state
> changed (Expired)
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,267} DEBUG [
> ZkClient-EventThread-162-tvip-m1-mw-zookeeper.dish.com:2181]
> (ZkEventThread.java:69) - Delivering event #24 ZkEvent[State changed to
> Expired sent to
> kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@1901a132
> ]
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,267} DEBUG
> [catalina-exec-5-EventThread] (ZkEventThread.java:88) - New event:
> ZkEvent[State changed to Expired sent to
> kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@10d1a0d2
> ]
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,267} DEBUG [
> ZkClient-EventThread-79-tvip-m1-mw-zookeeper.dish.com:2181]
> (ZkEventThread.java:69) - Delivering event #25 ZkEvent[State changed to
> Expired sent to
> kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@10d1a0d2
> ]
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,268} DEBUG
> [catalina-exec-8-EventThread] (ZkClient.java:351) - Received event:
> WatchedEvent state:Expired type:None path:null
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,268} INFO
> [catalina-exec-8-EventThread] (ZkClient.java:449) - zookeeper state
> changed (Expired)
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,269} DEBUG
> [catalina-exec-8-EventThread] (ZkEventThread.java:88) - New event:
> ZkEvent[State changed to Expired sent to
> kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@b977ea5]
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,269} DEBUG [
> ZkClient-EventThread-128-tvip-m1-mw-zookeeper.dish.com:2181]
> (ZkEventThread.java:69) - Delivering event #28 ZkEvent[State changed to
> Expired sent to
> kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@b977ea5]
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,272} DEBUG
> [catalina-exec-2-EventThread] (ZkClient.java:351) - Received event:
> WatchedEvent state:Expired type:None path:null
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,273} INFO
> [catalina-exec-2-EventThread] (ZkClient.java:449) - zookeeper state
> changed (Expired)
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,273} DEBUG
> [catalina-exec-2-EventThread] (ZkEventThread.java:88) - New event:
> ZkEvent[State changed to Expired sent to
> kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@376807ed
> ]
> catalina.out.2014-01-08_16-07-05:{2014-01-10 00:58:03,273} DEBUG [
> ZkClient-EventThread-41-tvip-m1-mw-zookeeper.dish.com:2181]
> (ZkEventThread.java:69) - Delivering event #32 ZkEvent[State changed to
> Expired sent to
> kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@376807ed
> ]
>
> -----Original Message-----
> From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com]
> Sent: Sunday, January 12, 2014 8:51 AM
> To: 'users@kafka.apache.org'
> Subject: RE: Looks like consumer fetchers get stopped we are not getting
> any data
>
> I do see ZkClient expired when this really happened.
>
>
> -----Original Message-----
> From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com]
> Sent: Saturday, January 11, 2014 10:31 PM
> To: users@kafka.apache.org
> Subject: RE: Looks like consumer fetchers get stopped we are not getting
> any data
>
> Currently its set to default,whats the timeout should we increase to.
> ________________________________________
> From: Guozhang Wang [wangg...@gmail.com]
> Sent: Saturday, January 11, 2014 3:53 PM
> To: users@kafka.apache.org
> Subject: Re: Looks like consumer fetchers get stopped we are not getting
> any data
>
> From the logs it seems the consumer's ZK registry log has lost, while
> KAFKA-693 is mainly due to server side issue. Could you check if there is
> a session timeout from the consumer on ZK log.
>
> Guozhang
>
>
> On Sat, Jan 11, 2014 at 2:33 PM, Seshadri, Balaji
> <balaji.sesha...@dish.com>wrote:
>
> > We found the below bug in Kafka JIRA.
> >
> > https://issues.apache.org/jira/browse/KAFKA-693
> >
> > Could you guys please let us know if this bug is causing the fetchers
> > getting stooped because of rebalance failure,we are using 0.8-beta and
> > this fix seems to be in 0.8 version.
> >
> > The issue happens only when we run it in tomcat,when we use JUnits
> > with same group.id to test we receive the messages.It would be great
> > if you could help us confirm if it is a bug.
> >
> > Also we would like to know if we need to upgrade the server/client to
> > 0.8 or just server side code.
> >
> >
> > ________________________________________
> > From: Rob Withers [robert.w.with...@gmail.com]
> > Sent: Friday, January 10, 2014 10:17 PM
> > To: users@kafka.apache.org
> > Subject: Re: Looks like consumer fetchers get stopped we are not
> > getting any data
> >
> > That was an interesting section too. Which GC settings would you
> suggest?
> >
> > Thank you,
> > - charlie
> >
> > > On Jan 10, 2014, at 10:11 PM, Jun Rao <jun...@gmail.com> wrote:
> > >
> > > Have you looked at our FAQ, especially
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretherem
> > anyrebalancesinmyconsumerlog
> > > ?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Fri, Jan 10, 2014 at 2:25 PM, Seshadri, Balaji
> > > <balaji.sesha...@dish.com>wrote:
> > >
> > >> Any clue would be helpful.
> > >>
> > >> -----Original Message-----
> > >> From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com]
> > >> Sent: Friday, January 10, 2014 12:46 PM
> > >> To: users@kafka.apache.org
> > >> Subject: RE: Looks like consumer fetchers get stopped we are not
> > >> getting any data
> > >>
> > >> Yes rebalance begins and exceptions occurs.
> > >>
> > >>
> > >> {2014-01-10 00:58:11,293} INFO
> > >>
> > [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660
> > b_watcher_executor]
> > >> (?:?) - [account-i
> > >> nfo-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b],
> > >> Cleared
> > the
> > >> data chunks in all the consumer message iterators
> > >> {2014-01-10 00:58:11,293} INFO
> > >>
> > [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660
> > b_watcher_executor]
> > >> (?:?) - [account-i
> > >> nfo-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b],
> > Committing
> > >> all offsets after clearing the fetcher queues
> > >> {2014-01-10 00:58:11,298} DEBUG [catalina-exec-12-SendThread(
> > >> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:759) - Got
> > >> ping response for sessionid: 0x1437b2879870005 af ter 0ms
> > >> {2014-01-10 00:58:11,313} INFO
> > >>
> > [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738_w
> > atcher_executor]
> > >> (?:?) -
> > >> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b673
> > >> 8],
> > >> begin rebalancing consumer
> > >> account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738
> > >> try
> > #1
> > >> {2014-01-10 00:58:11,314} DEBUG [catalina-exec-12-SendThread(
> > >> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) -
> > >> Reading reply sessionid:0x1437b2879870005, packet:: clientPath:null
> > serverPath:null
> > >> finished:false header:: 627,8 replyHeader:: 627,51539619966,0
> > request::
> > >> '/brokers/ids,F response:: v{'1}
> > >> {2014-01-10 00:58:11,315} DEBUG [catalina-exec-12-SendThread(
> > >> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) -
> > >> Reading reply sessionid:0x1437b2879870005, packet:: clientPath:null
> > serverPath:null
> > >> finished:false header:: 628,4 replyHeader:: 628,51539619966,0
> > request::
> > >> '/brokers/ids/1,F response::
> > >>
> > #7b2022686f7374223a22746d312d6b61666b6162726f6b6572313031222c20226a6d7
> > 85f706f7274223a393939392c2022706f7274223a393039322c202276657273696f6e2
> > 23a31207d,s{47244644685,47244644685,1388537628753,1388537628753,0,0,0,
> > 163056791896588316,74,0,47244644685}
> > >> {2014-01-10 00:58:11,316} DEBUG [catalina-exec-12-SendThread(
> > >> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) -
> > >> Reading reply sessionid:0x1437b2879870005, packet:: clientPath:null
> > serverPath:null
> > >> finished:false header:: 629,4 replyHeader:: 629,51539619966,-101
> > >> request::
> > >>
> > '/consumers/account-activated-hadoop-consumer/ids/account-activated-ha
> > doop-consumer_tm1mwdpl04-1389222557906-562b6738,F
> > >> response::
> > >> {2014-01-10 00:58:11,316} INFO
> > >>
> > [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738_w
> > atcher_executor]
> > >> (?:?) -
> > >> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b673
> > >> 8],
> > >> exception during rebalance
> > >> org.I0Itec.zkclient.exception.ZkNoNodeException:
> > >> org.apache.zookeeper.KeeperException$NoNodeException:
> > >> KeeperErrorCode = NoNode for
> > >>
> > /consumers/account-activated-hadoop-consumer/ids/account-activated-had
> > oop-consumer_tm1mwdpl04-1389222557906-562b6738
> > >> at
> > >> org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
> > >> at
> > >> org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
> > >> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
> > >> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
> > >> at kafka.utils.ZkUtils$.readData(Unknown Source)
> > >> at kafka.consumer.TopicCount$.constructTopicCount(Unknown
> Source)
> > >> at
> > >>
> > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$c
> > onsumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(Unk
> > nown
> > >> Source)
> > >> at
> > >>
> > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfu
> > n$syncedRebalance$1.apply$mcVI$sp(Unknown
> > >> Source)
> > >> at
> > scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:142)
> > >> at
> > >>
> > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedR
> > ebalance(Unknown
> > >> Source)
> > >> at
> > >>
> > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1
> > .run(Unknown
> > >> Source) Caused by:
> org.apache.zookeeper.KeeperException$NoNodeException:
> > >> KeeperErrorCode = NoNode for
> > >>
> > /consumers/account-activated-hadoop-consumer/ids/account-activated-had
> > oop-consumer_tm1mwdpl04-1389222557906-562b6738
> > >> at
> > >> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> > >> at
> > >> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
> > >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:956)
> > >> at
> > org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103)
> > >> :
> > >> -----Original Message-----
> > >> From: Guozhang Wang [mailto:wangg...@gmail.com]
> > >> Sent: Friday, January 10, 2014 12:30 PM
> > >> To: users@kafka.apache.org
> > >> Subject: Re: Looks like consumer fetchers get stopped we are not
> > >> getting any data
> > >>
> > >> Actually, the broken channel is broken by shutting down the
> > >> leader-finder-thread, which is shutdown either by a rebalance retry
> > >> or shutting down the consumer.
> > >>
> > >> Do you see "begin rebalance ..." before this log entry? And if yes,
> > search
> > >> to see if the rebalance keep failing.
> > >>
> > >> Guozhang
> > >>
> > >>
> > >> On Fri, Jan 10, 2014 at 11:23 AM, Guozhang Wang
> > >> <wangg...@gmail.com>
> > >> wrote:
> > >>
> > >>> From your logs the channel with the brokers are broken, are the
> > >>> brokers alive at that time?
> > >>>
> > >>> Guozhang
> > >>>
> > >>>
> > >>> On Fri, Jan 10, 2014 at 10:52 AM, Withers, Robert
> > >>> <robert.with...@dish.com
> > >>>> wrote:
> > >>>
> > >>>> The core problem is our consumers stop consuming and lag increases.
> > >>>> We found this blog:
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerse
> > emstohavestopped,why
> > >> ?.
> > >>>> This lists 3 possibilities.
> > >>>>
> > >>>> The blog also talks earlier about spurious rebalances, due to
> > >>>> improper GC settings, but we couldn't find what GC settings to use.
> > >>>> We are considering changing the zookeeper timeouts. We are a
> > >>>> little confused about the various issues, the sequence of issues
> > >>>> and what could cause the consumers to stop reading. If the
> > >>>> fetchers get shutdown, due to a ClosedByInterruptException in the
> "leader_finder"
> > >>>> thread, which tells the "executor_watcher" thread to shutdown the
> > >>>> fetchers, that would be another reason the consumers stop
> > >>>> processing
> > >> data. Is this possible?
> > >>>>
> > >>>> Thank you,
> > >>>> rob
> > >>>>
> > >>>> -----Original Message-----
> > >>>> From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com]
> > >>>> Sent: Friday, January 10, 2014 11:40 AM
> > >>>> To: users@kafka.apache.org
> > >>>> Subject: RE: Looks like consumer fetchers get stopped we are not
> > >>>> getting any data
> > >>>>
> > >>>> It would be helpful if you guys can shed some light why all
> > >>>> fetchers are getting stopped.
> > >>>>
> > >>>> -----Original Message-----
> > >>>> From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com]
> > >>>> Sent: Friday, January 10, 2014 11:28 AM
> > >>>> To: users@kafka.apache.org
> > >>>> Subject: RE: Looks like consumer fetchers get stopped we are not
> > >>>> getting any data
> > >>>>
> > >>>> We also got the below error when this happens.
> > >>>>
> > >>>> {2014-01-10 00:58:11,292} INFO
> > >>>>
> > >>>> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad
> > >>>> 5966
> > >>>> 0b_watcher_executor]
> > >>>> (?:?) -
> > >>>> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad
> > >>>> 5966
> > >>>> 0b],
> > >>>> exception during rebalance
> > >>>> org.I0Itec.zkclient.exception.ZkNoNodeException:
> > >>>> org.apache.zookeeper.KeeperException$NoNodeException:
> > >>>> KeeperErrorCode = NoNode for
> > >>
> > /consumers/account-info-updated-hadoop-consumer/ids/account-info-updat
> > ed-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b
> > >>>> at
> > >>>>
> org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
> > >>>> at
> > >>>> org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
> > >>>> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
> > >>>> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
> > >>>> at kafka.utils.ZkUtils$.readData(Unknown Source)
> > >>>> at kafka.consumer.TopicCount$.constructTopicCount(Unknown
> > >> Source)
> > >>>> at
> > >>>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.ka
> > >>>> fka$
> > >>>> consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalan
> > >>>> ce(U
> > >>>> nknown
> > >>>> Source)
> > >>>> at
> > >>>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$a
> > >>>> nonf un$syncedRebalance$1.apply$mcVI$sp(Unknown
> > >>>> Source)
> > >>>> at
> > >>>> scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:142)
> > >>>> at
> > >>>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.sy
> > >>>> nced
> > >>>> Rebalance(Unknown
> > >>>> Source)
> > >>>> at
> > >>>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$a
> > >>>> non$
> > >>>> 1.run(Unknown
> > >>>> Source) Caused by:
> > org.apache.zookeeper.KeeperException$NoNodeException:
> > >>>> KeeperErrorCode = NoNode for
> > >>
> > /consumers/account-info-updated-hadoop-consumer/ids/account-info-updat
> > ed-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b
> > >>>> at
> > >>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> > >>>> at
> > >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
> > >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:956)
> > >>>> at
> > >>>> org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103)
> > >>>> at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770)
> > >>>> at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766)
> > >>>> at
> > >>>> org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> > >>>> ... 9 more
> > >>>>
> > >>>> -----Original Message-----
> > >>>> From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com]
> > >>>> Sent: Friday, January 10, 2014 10:52 AM
> > >>>> To: users@kafka.apache.org
> > >>>> Subject: Looks like consumer fetchers get stopped we are not
> > >>>> getting any data
> > >>>>
> > >>>> Please let us know why we are not getting any data from Kafaka
> > >>>> after this log from Kafka,can you guys lets us know.
> > >>>>
> > >>>> What could be causing all fetchers associated to be stooped why
> > >>>> it is not doing retry.
> > >>>>
> > >>>> {2014-01-10 00:58:09,284} WARN
> > >>>>
> > >>>> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad
> > >>>> 5966
> > >>>> 0b-leader-finder-thread]
> > >>>> (?:?) - Fetching topic metadata with correlation id 3 for topics
> > >>>> [Set(account-info-updated)] from broker
> > >>>> [id:1,host:tm1-kafkabroker101,port:9092] failed
> > >>>> java.nio.channels.ClosedByInterruptException
> > >>>> at
> > >>
> > java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterru
> > ptibleChannel.java:202)
> > >>>> at
> > >> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:506)
> > >>>> at
> > java.nio.channels.SocketChannel.write(SocketChannel.java:493)
> > >>>> at kafka.network.BoundedByteBufferSend.writeTo(Unknown
> Source)
> > >>>> at kafka.network.Send$class.writeCompletely(Unknown Source)
> > >>>> at
> > >>>> kafka.network.BoundedByteBufferSend.writeCompletely(Unknown
> > >>>> Source)
> > >>>> at kafka.network.BlockingChannel.send(Unknown Source)
> > >>>> at kafka.producer.SyncProducer.liftedTree1$1(Unknown Source)
> > >>>> at
> > >>>> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(U
> > >>>> nkno
> > >>>> wn
> > >>>> Source)
> > >>>> at kafka.producer.SyncProducer.send(Unknown Source)
> > >>>> at kafka.client.ClientUtils$.fetchTopicMetadata(Unknown
> Source)
> > >>>> at kafka.client.ClientUtils$.fetchTopicMetadata(Unknown
> Source)
> > >>>> at
> > >>>> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(U
> > >>>> nkno
> > >>>> wn
> > >>>> Source)
> > >>>> at kafka.utils.ShutdownableThread.run(Unknown Source)
> > >>>> {2014-01-10 00:58:09,284} DEBUG
> > >>>> [account-access-hadoop-consumer_tm1mwdpl04-1389222551916-a0c87abc
> > >>>> _wat
> > >>>> cher_executor]
> > >>>> (?:?) - initial fetch offset of account-access:27: fetched offset
> > >>>> =
> > >> 9655:
> > >>>> consumed offset = 9655 is 9655
> > >>>> {2014-01-10 00:58:09,284} DEBUG
> > >>>> [bill-generated-hadoop-consumer_tm1mwdpl04-1389222547995-29a6dce9
> > >>>> _wat
> > >>>> cher_executor]
> > >>>> (?:?) - initial consumer offset of bill-generated:11: fetched
> > >>>> offset =
> > >> 152:
> > >>>> consumed offset = 152 is 152
> > >>>> {2014-01-10 00:58:09,284} DEBUG
> > >>>> [outbound-communications-hadoop-consumer_tm1mwdpl04-1389222550693
> > >>>> -8bc
> > >>>> 34b77_watcher_executor]
> > >>>> (?:?) -
> > >>>> [outbound-communications-hadoop-consumer_tm1mwdpl04-1389222550693
> > >>>> -8bc
> > >>>> 34b77],
> > >>>> outbound-communications:108: fetched offset = 1689: consumed
> > >>>> offset =
> > >>>> 1689 selected new offset 1689
> > >>>> {2014-01-10 00:58:09,284} DEBUG [catalina-exec-3-SendThread(
> > >>>> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) -
> > >>>> Reading reply sessionid:0x1434b49cf56383b, packet::
> > >>>> clientPath:null serverPath:null finished:false header:: 279,4
> replyHeader::
> > >> 279,51539617506,0 request::
> > >>>> '/consumers/outbound-call-attempted-hadoop-consumer/offsets/outbo
> > >>>> und-
> > >>>> call-attempted/14,F
> > >>>> response::
> > >>>> #30,s{39860186414,39860186414,1387517714994,1387517714994,0,0,0,0
> > >>>> ,1,0
> > >>>> ,39860186414}
> > >>>> {2014-01-10 00:58:09,285} INFO
> > >>>>
> > >>>> [outbound-communications-hadoop-consumer_tm1mwdpl04-1389222550693
> > >>>> -8bc
> > >>>> 34b77_watcher_executor]
> > >>>> (?:?) -
> > >>>> [outbound-communications-hadoop-consumer_tm1mwdpl04-1389222550693
> > >>>> -8bc
> > >>>> 34b77],
> > >>>> outbound-communications-hadoop-consumer_tm1mwdpl04-1389222550693-
> > >>>> 8bc3
> > >>>> 4b77-3
> > >>>> attempting to claim partition 109
> > >>>> {2014-01-10 00:58:09,284} DEBUG
> > >>>> [bill-generated-hadoop-consumer_tm1mwdpl04-1389222547995-29a6dce9
> > >>>> _wat
> > >>>> cher_executor]
> > >>>> (?:?) - initial fetch offset of bill-generated:11: fetched offset
> > >>>> =
> > 152:
> > >>>> consumed offset = 152 is 152
> > >>>> {2014-01-10 00:58:09,284} DEBUG [catalina-exec-12-SendThread(
> > >>>> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) -
> > >>>> Reading reply sessionid:0x1437b2879870005, packet::
> > >>>> clientPath:null serverPath:null finished:false header:: 619,1
> replyHeader::
> > >> 619,51539617508,0 request::
> > >>>> '/consumers/account-activated-hadoop-consumer/owners/account-acti
> > >>>> vate
> > >>>> d/68,#6163636f756e742d6163746976617465642d6861646f6f702d636f6e737
> > >>>> 56d6
> > >>>> 5725f746d316d7764706c30342d313338393232323535373930362d3536326236
> > >>>> 3733
> > >>>> 382d30,v{s{31,s{'world,'anyone}}},1
> > >>>> response::
> > >>>> '/consumers/account-activated-hadoop-consumer/owners/account-acti
> > >>>> vate
> > >>>> d/68
> > >>>> {2014-01-10 00:58:09,284} DEBUG
> > >>>> [account-access-hadoop-consumer_tm1mwdpl04-1389222551916-a0c87abc
> > >>>> _wat
> > >>>> cher_executor]
> > >>>> (?:?) -
> > >>>> [account-access-hadoop-consumer_tm1mwdpl04-1389222551916-a0c87abc
> > >>>> ],
> > >>>> account-access:27: fetched offset = 9655: consumed offset = 9655
> > >>>> selected new offset 9655
> > >>>> {2014-01-10 00:58:09,284} INFO
> > >>>>
> > >>>> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad
> > >>>> 5966
> > >>>> 0b_watcher_executor]
> > >>>> (?:?) -
> > >>>> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad
> > >>>> 5966
> > >>>> 0b-leader-finder-thread],
> > >>>> Shutdown completed
> > >>>> {2014-01-10 00:58:09,284} INFO
> > >>>>
> > >>>> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad
> > >>>> 5966
> > >>>> 0b-leader-finder-thread]
> > >>>> (?:?) -
> > >>>> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad
> > >>>> 5966
> > >>>> 0b-leader-finder-thread],
> > >>>> Stopped
> > >>>> {2014-01-10 00:58:09,285} INFO
> > >>>>
> > >>>> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad
> > >>>> 5966
> > >>>> 0b_watcher_executor]
> > >>>> (?:?) - [ConsumerFetcherManager-1389222553163] Stopping all
> > >>>> fetchers
> > >>>
> > >>>
> > >>> --
> > >>> -- Guozhang
> > >>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >>
> >
>
>
>
> --
> -- Guozhang
>
--
-- Guozhang
