Hi Guozhang, sorry for asking a little unrelated problem here.
We found a consumer stopping fetching data when doing an network upgrade, if the consumer have connection problem with one broker (but OK with zookeeper and other brokers) the fetcherrunnable will stop, but there is no chance to restart the thread again ( since there is no zookeeper timeout so no reblance will be triggered) kafka version is 0.7.2 and we are using high level consumer. I did a simulation test using iptables and the result is same( no chance to restart fetcherrunnable). I have read the code, seems the exception will not be handled outside. Am I wrong about this? THANKS. On Sat, May 17, 2014 at 12:19 AM, Guozhang Wang <wangg...@gmail.com> wrote: > Hi Yonghui, > > Could you check if consumer2's fetcher thread is still alive? Also we have > an entry in FAQ wiki page about "consumer stopped consuming", current > Apache has some issues with the wiki page, but you may want to check it out > once Apache page resumes. > > Guozhang > > > On Wed, May 14, 2014 at 8:59 PM, Yonghui Zhao <zhaoyong...@gmail.com> > wrote: > > > Thanks GuoZhang. > > > > After last accident, we stop all consumers and then restart all consumers > > one by one then it is ok. > > > > 2 brokers, 10 partitions / broker, 3 consumers, each create 10 stream > > > > So consumer1 consumes 10 partitions, consumer2 consumers another 10 > > partitions, consumer3 is idle. > > > > Today we find some exceptions in consumer2, after these exceptions > > consumer2 doesn't work, no message is consumed. > > > > But in zk I found the owner ship doesn't change, consumer1 own 10 > > partitions consumer 2 own another 10 partitions/ > > > > How can we avoid this happen again? > > > > > > From the log we see the error sequence: > > > > > > > > > > *exception during commitOffsets, Reconnect in multifetch due to socket > > error, rebalance 2 times, error in FetcherRunnable'Seems the last > error > > in FetcherRunnable is deadly, after this error no message is consumed, > but > > zk ownership doesn't release.*Here is all the kafka related log in that > > time > > *:* > > > > [WARN 2014-05-13 16:19:05.020] > > kafka.utils.Logging$class.warn(Logging.scala:79) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception > > during commitOffsets] > > at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103) > > at > > > > > kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251) > > at > > > > > kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248) > > at > > > > > kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248) > > at > > > > > kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246) > > at > > > > > kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246) > > at > > > > > kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232) > > at > > > > > kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126) > > at kafka.utils.Utils$$anon$2.run(Utils.scala:58) > > [INFO 2014-05-13 16:19:08.991] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c ZK expired; > > release old broker parition ownership; re-register consumer > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c] > > [INFO 2014-05-13 16:19:08.991] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin > > registering consumer > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in ZK] > > [WARN 2014-05-13 16:19:09.001] > > kafka.utils.Logging$class.warn(Logging.scala:79) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception > > during commitOffsets] > > at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103) > > at > > > > > kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251) > > at > > > > > kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248) > > at > > > > > kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248) > > at > > > > > kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246) > > at > > > > > kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246) > > at > > > > > kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232) > > at > > > > > kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126) > > at kafka.utils.Utils$$anon$2.run(Utils.scala:58) > > [INFO 2014-05-13 16:19:09.002] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end > registering > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in > ZK] > > [INFO 2014-05-13 16:19:09.003] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin > > rebalancing consumer > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0] > > [INFO 2014-05-13 16:19:09.063] > > kafka.utils.Logging$class.info(Logging.scala:69) > > [Reconnect in multifetch due to socket error: ] > > at kafka.utils.Utils$.read(Utils.scala:538) > > at > > > > > kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) > > at kafka.network.Receive$class.readCompletely(Transmission.scala:55) > > at > > > > > kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) > > at > kafka.consumer.SimpleConsumer.getResponse(SimpleConsumer.scala:177) > > at > > kafka.consumer.SimpleConsumer.liftedTree2$1(SimpleConsumer.scala:117) > > at kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:115) > > at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60) > > [INFO 2014-05-13 16:19:09.068] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted] > > [INFO 2014-05-13 16:19:09.069] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12] > > [INFO 2014-05-13 16:19:09.071] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Cleared all relevant queues for this fetcher] > > [INFO 2014-05-13 16:19:09.071] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.071] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.071] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.072] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.072] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.072] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.072] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.072] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.072] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.072] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.072] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Cleared the data chunks in all the consumer message iterators] > > [INFO 2014-05-13 16:19:09.073] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing > all > > offsets after clearing the fetcher queues] > > [INFO 2014-05-13 16:19:09.073] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing > > partition ownership] > > [INFO 2014-05-13 16:19:09.073] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing > the > > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8, > > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei > > with consumers: > > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)] > > [INFO 2014-05-13 16:19:09.073] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting > to > > claim partition 1-8] > > [INFO 2014-05-13 16:19:09.077] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting > to > > claim partition 1-4] > > [INFO 2014-05-13 16:19:09.081] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting > to > > claim partition 1-3] > > [INFO 2014-05-13 16:19:09.085] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting > to > > claim partition 1-2] > > [INFO 2014-05-13 16:19:09.089] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting > to > > claim partition 1-1] > > [INFO 2014-05-13 16:19:09.093] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting > to > > claim partition 1-5] > > [INFO 2014-05-13 16:19:09.097] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting > to > > claim partition 1-6] > > [INFO 2014-05-13 16:19:09.101] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting > to > > claim partition 1-9] > > [INFO 2014-05-13 16:19:09.105] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting > to > > claim partition 1-7] > > [INFO 2014-05-13 16:19:09.109] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting > to > > claim partition 1-0] > > [INFO 2014-05-13 16:19:09.118] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully > > owned partition 1-2 for topic sensei] > > [INFO 2014-05-13 16:19:09.122] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully > > owned partition 1-3 for topic sensei] > > [INFO 2014-05-13 16:19:09.127] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully > > owned partition 1-5 for topic sensei] > > [INFO 2014-05-13 16:19:09.132] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully > > owned partition 1-0 for topic sensei] > > [INFO 2014-05-13 16:19:09.137] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully > > owned partition 1-9 for topic sensei] > > [INFO 2014-05-13 16:19:09.153] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully > > owned partition 1-6 for topic sensei] > > [INFO 2014-05-13 16:19:09.157] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully > > owned partition 1-7 for topic sensei] > > [INFO 2014-05-13 16:19:09.161] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully > > owned partition 1-1 for topic sensei] > > [INFO 2014-05-13 16:19:09.166] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully > > owned partition 1-4 for topic sensei] > > [INFO 2014-05-13 16:19:09.170] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully > > owned partition 1-8 for topic sensei] > > [INFO 2014-05-13 16:19:09.170] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the > > cache] > > [INFO 2014-05-13 16:19:09.171] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected > > partitions : sensei:1-0: fetched offset = 1555136587: consumed offset = > > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset = > > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset = > > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset = > > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset = > > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset = > > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset = > > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset = > > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset = > > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset = > > 1558487487] > > [INFO 2014-05-13 16:19:09.171] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end > rebalancing > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try > #0] > > [INFO 2014-05-13 16:19:09.172] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.172] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.172] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 > from > > 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.172] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.172] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.173] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.173] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.173] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.173] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.173] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.188] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin > > rebalancing consumer > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0] > > [INFO 2014-05-13 16:19:09.240] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted] > > [INFO 2014-05-13 16:19:09.240] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12] > > [INFO 2014-05-13 16:19:09.241] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Cleared all relevant queues for this fetcher] > > [INFO 2014-05-13 16:19:09.241] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.241] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.241] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.241] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.242] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.242] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.242] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.242] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.242] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.242] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.242] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Cleared the data chunks in all the consumer message iterators] > > [INFO 2014-05-13 16:19:09.242] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing > all > > offsets after clearing the fetcher queues] > > [INFO 2014-05-13 16:19:09.285] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing > > partition ownership] > > [INFO 2014-05-13 16:19:09.333] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing > the > > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8, > > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei > > with consumers: > > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)] > > [INFO 2014-05-13 16:19:09.333] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting > to > > claim partition 1-8] > > [INFO 2014-05-13 16:19:09.337] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting > to > > claim partition 1-4] > > [INFO 2014-05-13 16:19:09.341] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting > to > > claim partition 1-3] > > [INFO 2014-05-13 16:19:09.345] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting > to > > claim partition 1-2] > > [INFO 2014-05-13 16:19:09.349] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting > to > > claim partition 1-1] > > [INFO 2014-05-13 16:19:09.354] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting > to > > claim partition 1-5] > > [INFO 2014-05-13 16:19:09.358] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting > to > > claim partition 1-6] > > [INFO 2014-05-13 16:19:09.362] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting > to > > claim partition 1-9] > > [INFO 2014-05-13 16:19:09.366] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting > to > > claim partition 1-7] > > [INFO 2014-05-13 16:19:09.370] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting > to > > claim partition 1-0] > > [INFO 2014-05-13 16:19:09.379] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully > > owned partition 1-2 for topic sensei] > > [INFO 2014-05-13 16:19:09.390] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [/consumers/RecommendEvent/owners/sensei/1-3 exists with value > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 during > > connection loss; this is ok] > > [INFO 2014-05-13 16:19:09.390] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully > > owned partition 1-3 for topic sensei] > > [INFO 2014-05-13 16:19:09.395] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully > > owned partition 1-5 for topic sensei] > > [INFO 2014-05-13 16:19:09.400] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully > > owned partition 1-0 for topic sensei] > > [INFO 2014-05-13 16:19:09.405] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully > > owned partition 1-9 for topic sensei] > > [INFO 2014-05-13 16:19:09.409] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully > > owned partition 1-6 for topic sensei] > > [INFO 2014-05-13 16:19:09.417] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [/consumers/RecommendEvent/owners/sensei/1-7 exists with value > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 during > > connection loss; this is ok] > > [INFO 2014-05-13 16:19:09.417] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully > > owned partition 1-7 for topic sensei] > > [INFO 2014-05-13 16:19:09.422] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully > > owned partition 1-1 for topic sensei] > > [INFO 2014-05-13 16:19:09.430] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [/consumers/RecommendEvent/owners/sensei/1-4 exists with value > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 during > > connection loss; this is ok] > > [INFO 2014-05-13 16:19:09.430] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully > > owned partition 1-4 for topic sensei] > > [INFO 2014-05-13 16:19:09.435] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully > > owned partition 1-8 for topic sensei] > > [INFO 2014-05-13 16:19:09.435] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the > > cache] > > [INFO 2014-05-13 16:19:09.435] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected > > partitions : sensei:1-0: fetched offset = 1555136587: consumed offset = > > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset = > > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset = > > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset = > > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset = > > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset = > > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset = > > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset = > > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset = > > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset = > > 1558487487] > > [INFO 2014-05-13 16:19:09.436] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end > rebalancing > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try > #0] > > [INFO 2014-05-13 16:19:09.438] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.438] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin > > rebalancing consumer > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0] > > [INFO 2014-05-13 16:19:09.438] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.438] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 > from > > 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.438] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.438] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.438] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.438] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.439] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.439] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.439] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.492] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted] > > [INFO 2014-05-13 16:19:09.492] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12] > > [INFO 2014-05-13 16:19:09.492] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Cleared all relevant queues for this fetcher] > > [INFO 2014-05-13 16:19:09.492] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.492] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.493] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.493] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.493] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.493] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.493] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.493] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.494] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.494] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Clearing the current data chunk for this consumer iterator] > > [INFO 2014-05-13 16:19:09.494] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [Cleared the data chunks in all the consumer message iterators] > > [INFO 2014-05-13 16:19:09.494] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing > all > > offsets after clearing the fetcher queues] > > [INFO 2014-05-13 16:19:09.539] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing > > partition ownership] > > [INFO 2014-05-13 16:19:09.582] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing > the > > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8, > > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei > > with consumers: > > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8, > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8, > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8, > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)] > > [INFO 2014-05-13 16:19:09.582] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting > to > > claim partition 1-8] > > [INFO 2014-05-13 16:19:09.586] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting > to > > claim partition 1-4] > > [INFO 2014-05-13 16:19:09.590] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting > to > > claim partition 1-3] > > [INFO 2014-05-13 16:19:09.594] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting > to > > claim partition 1-2] > > [INFO 2014-05-13 16:19:09.598] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting > to > > claim partition 1-1] > > [INFO 2014-05-13 16:19:09.602] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting > to > > claim partition 1-5] > > [INFO 2014-05-13 16:19:09.606] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting > to > > claim partition 1-6] > > [INFO 2014-05-13 16:19:09.610] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting > to > > claim partition 1-9] > > [INFO 2014-05-13 16:19:09.614] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting > to > > claim partition 1-7] > > [INFO 2014-05-13 16:19:09.618] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting > to > > claim partition 1-0] > > [INFO 2014-05-13 16:19:09.633] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully > > owned partition 1-2 for topic sensei] > > [INFO 2014-05-13 16:19:09.638] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully > > owned partition 1-3 for topic sensei] > > [INFO 2014-05-13 16:19:09.646] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully > > owned partition 1-5 for topic sensei] > > [INFO 2014-05-13 16:19:09.656] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully > > owned partition 1-0 for topic sensei] > > [INFO 2014-05-13 16:19:09.660] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully > > owned partition 1-9 for topic sensei] > > [INFO 2014-05-13 16:19:09.665] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully > > owned partition 1-6 for topic sensei] > > [INFO 2014-05-13 16:19:09.669] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully > > owned partition 1-7 for topic sensei] > > [INFO 2014-05-13 16:19:09.674] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully > > owned partition 1-1 for topic sensei] > > [INFO 2014-05-13 16:19:09.679] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully > > owned partition 1-4 for topic sensei] > > [INFO 2014-05-13 16:19:09.684] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully > > owned partition 1-8 for topic sensei] > > [INFO 2014-05-13 16:19:09.684] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the > > cache] > > [INFO 2014-05-13 16:19:09.684] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected > > partitions : sensei:1-0: fetched offset = 1555136587: consumed offset = > > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset = > > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset = > > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset = > > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset = > > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset = > > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset = > > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset = > > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset = > > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset = > > 1558487487] > > [INFO 2014-05-13 16:19:09.684] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end > rebalancing > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try > #0] > > [INFO 2014-05-13 16:19:09.685] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.685] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.685] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 > from > > 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.685] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.686] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.686] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.686] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.686] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.686] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587 > > from 10.101.10.12:9092] > > [INFO 2014-05-13 16:19:09.686] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487 > > from 10.101.10.12:9092] > > [ERROR 2014-05-13 16:19:18.687] > > kafka.utils.Logging$class.error(Logging.scala:89) [error in > FetcherRunnable > > ] > > at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:49) > > at > > > kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:186) > > at kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:113) > > at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60) > > [INFO 2014-05-13 16:19:18.687] > > kafka.utils.Logging$class.info(Logging.scala:61) > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12] > > > > > > > > 2014-05-12 23:22 GMT+08:00 Guozhang Wang <wangg...@gmail.com>: > > > > > Hello Yonghui, > > > > > > In 0.7 the consumer rebalance logic is distributed and in some corner > > cases > > > such as soft-failure-caused-consecutive rebalances some consumer may > > > consider the rebalance as complete while others are still trying the > > > rebalance process. You can check the GC logs on your consumer to verify > > if > > > that is the case: > > > > > > https://issues.apache.org/jira/browse/KAFKA-242 > > > > > > If you bounce the consumers to trigger another rebalance, this issue > > would > > > likely to be resolved. > > > > > > To solve this issue in 0.9 we are moving the group management like load > > > rebalance from the ZK-based distributed logic into a centralized > > > coordiantor. Details can be found here: > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design > > > > > > Guozhang > > > > > > > > > On Mon, May 12, 2014 at 12:48 AM, Yonghui Zhao <zhaoyong...@gmail.com > > > >wrote: > > > > > > > Hi, > > > > > > > > We are using kafka 0.7. > > > > > > > > 2 brokers, each broker has 10 partitions for one topic > > > > 3 consumers in one consumer group, each consumer create 10 streams. > > > > > > > > > > > > Today, when we want to rollout new service. > > > > After we restart one consumer we find exceptions and warning. > > > > > > > > kafka.common.ConsumerRebalanceFailedException: > > > > RecommendEvent_sd-sns-relation01.bj-1399630465426-53d3aefc can't > > > rebalance > > > > after 4 retries > > > > > > > > > > > > [INFO 2014-05-12 15:17:47.364] > > > > kafka.utils.Logging$class.info(Logging.scala:61) > > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-2 data: > > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-2 stored > > data: > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1] > > > > [INFO 2014-05-12 15:17:47.366] > > > > kafka.utils.Logging$class.info(Logging.scala:61) > > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting > for > > > the > > > > partition ownership to be deleted: 1-2] > > > > [INFO 2014-05-12 15:17:47.375] > > > > kafka.utils.Logging$class.info(Logging.scala:61) > > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-3 data: > > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-3 stored > > data: > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1] > > > > [INFO 2014-05-12 15:17:47.375] > > > > kafka.utils.Logging$class.info(Logging.scala:61) > > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting > for > > > the > > > > partition ownership to be deleted: 1-3] > > > > [INFO 2014-05-12 15:17:47.385] > > > > kafka.utils.Logging$class.info(Logging.scala:61) > > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-5 data: > > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-5 stored > > data: > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2] > > > > [INFO 2014-05-12 15:17:47.386] > > > > kafka.utils.Logging$class.info(Logging.scala:61) > > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting > for > > > the > > > > partition ownership to be deleted: 1-5] > > > > > > > > > > > > > > > > And I opened zk viewer. > > > > > > > > In zk, we found 2 consumers in ConsumerGroup/ids: > > > > > > > > RecommendEvent_sd-sns-relation02.bj-1399635256619-5d8123c6 > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3 > > > > > > > > > > > > And in owners/topic/ we found all partitions are assigned to > > > > sd-sns-relation03.bj: > > > > > > > > Here is the owner info: > > > > 1:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0 > > > > 1:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0 > > > > 1:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1 > > > > 1:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1 > > > > 1:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2 > > > > 1:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2 > > > > 1:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3 > > > > 1:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3 > > > > 1:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4 > > > > 1:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4 > > > > > > > > 2:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0 > > > > 2:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1 > > > > 2:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2 > > > > 2:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3 > > > > 2:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4 > > > > 2:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-5 > > > > 2:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-6 > > > > 2:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-7 > > > > 2:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-8 > > > > 2:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-9 > > > > > > > > > > > > So all partitions are assigned to sd-sns-relation03.bj, but from > logs > > > and > > > > counter, we are sure sd-sns-relation02.bj has input too. > > > > > > > > > > > > My question is: > > > > > > > > 1. why rebalance failed? > > > > 2. why owner info is wrong? btw: zkclient is 0.2 > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > -- > -- Guozhang > -- *Best Regards 向河林*
