Sorry, I misspoke, I did manage to find the logs with failed rebalances on host62:
worker-6727.log:2014-02-02 01:46:14 k.c.ZookeeperConsumerConnector [INFO] [trackingGroup_prod-host62-1391197017388-c3ce6986], trackingGroup_prod-host62-1391197017388-c3ce6986-0 attempting to claim partition 108 worker-6727.log:2014-02-02 01:46:14 k.c.ZookeeperConsumerConnector [INFO] [trackingGroup_prod-host62-1391197017388-c3ce6986], trackingGroup_prod-host62-1391197017388-c3ce6986-0 attempting to claim partition 109 worker-6727.log:2014-02-02 01:46:14 k.u.ZkUtils$ [INFO] conflict in /consumers/trackingGroup/owners/Events2/108 data: trackingGroup_prod-host62-1391197017388-c3ce6986-0 stored data: trackingGroup_prod-host62-1391197017407-f3e67af0-0 worker-6727.log:2014-02-02 01:46:14 k.c.ZookeeperConsumerConnector [INFO] [trackingGroup_prod-host62-1391197017388-c3ce6986], waiting for the partition ownership to be deleted: 108 worker-6727.log:2014-02-02 01:46:14 k.u.ZkUtils$ [INFO] conflict in /consumers/trackingGroup/owners/Events2/109 data: trackingGroup_prod-host62-1391197017388-c3ce6986-0 stored data: trackingGroup_prod-host62-1391197017407-f3e67af0-0 worker-6727.log:2014-02-02 01:46:14 k.c.ZookeeperConsumerConnector [INFO] [trackingGroup_prod-host62-1391197017388-c3ce6986], waiting for the partition ownership to be deleted: 109 worker-6727.log:2014-02-02 01:46:14 k.c.ZookeeperConsumerConnector [INFO] [trackingGroup_prod-host62-1391197017388-c3ce6986], end rebalancing consumer trackingGroup_prod-host62-1391197017388-c3ce6986 try #356 worker-6727.log:2014-02-02 01:46:14 k.c.ZookeeperConsumerConnector [INFO] [trackingGroup_prod-host62-1391197017388-c3ce6986], Rebalancing attempt failed. Clearing the cache before the next rebalancing operation is triggered They look to have the same view of the topology though: Rebalance output for host 61: http://pastebin.com/MP2nfExR Rabalance output for host 62: http://pastebin.com/0jhmM4L2 Could there be a stale Zookeeper connection holding on to an ephemeral node? On Mon, Feb 3, 2014 at 10:49 AM, Drew Goya <d...@gradientx.com> wrote: > That is another strange thing, this was the only consumer doing > rebalances, host62 was happily consuming from its partitions. > > > On Sat, Feb 1, 2014 at 9:15 PM, Jun Rao <jun...@gmail.com> wrote: > >> What is host062 doing when the conflict occurred? Is it doing rebalance >> too? If so, does it see the same set of partitions and consumers as >> host061 >> does? >> >> Thanks, >> >> Jun >> >> >> On Sat, Feb 1, 2014 at 5:53 PM, Drew Goya <d...@gradientx.com> wrote: >> >> > Hey all, this issue has recently popped up again. I've got a member of >> a >> > consumer group stuck in a rebalance loop. It attempts to claim >> partitions >> > but those are always conflicted: >> > >> > 2014-02-02 01:46:22 k.c.ZookeeperConsumerConnector [INFO] >> > [trackingGroup_host061.pe1i.gradientx.com-1391301920791-99d22986], >> > trackingGroup_host061.pe1i.gradientx.com-1391301920791-99d22986-0 >> > attempting to claim partition 94 >> > 2014-02-02 01:46:22 k.c.ZookeeperConsumerConnector [INFO] >> > [trackingGroup_host061.pe1i.gradientx.com-1391301920791-99d22986], >> > trackingGroup_host061.pe1i.gradientx.com-1391301920791-99d22986-0 >> > attempting to claim partition 95 >> > 2014-02-02 01:46:22 k.u.ZkUtils$ [INFO] conflict in >> > /consumers/trackingGroup/owners/Events2/94 data: >> > trackingGroup_host061.pe1i.gradientx.com-1391301920791-99d22986-0 stored >> > data: trackingGroup_host062.pe1i.gradientx.com-1391197016949-ba8c6bb5-0 >> > 2014-02-02 01:46:22 k.c.ZookeeperConsumerConnector [INFO] >> > [trackingGroup_host061.pe1i.gradientx.com-1391301920791-99d22986], >> waiting >> > for the partition ownership to be deleted: 94 >> > 2014-02-02 01:46:22 k.u.ZkUtils$ [INFO] conflict in >> > /consumers/trackingGroup/owners/Events2/95 data: >> > trackingGroup_host061.pe1i.gradientx.com-1391301920791-99d22986-0 stored >> > data: trackingGroup_host062.pe1i.gradientx.com-1391197016949-ba8c6bb5-0 >> > 2014-02-02 01:46:22 k.c.ZookeeperConsumerConnector [INFO] >> > [trackingGroup_host061.pe1i.gradientx.com-1391301920791-99d22986], >> waiting >> > for the partition ownership to be deleted: 95 >> > 2014-02-02 01:46:22 k.c.ZookeeperConsumerConnector [INFO] >> > [trackingGroup_host061.pe1i.gradientx.com-1391301920791-99d22986], end >> > rebalancing consumer >> > trackingGroup_host061.pe1i.gradientx.com-1391301920791-99d22986 try #352 >> > 2014-02-02 01:46:22 k.c.ZookeeperConsumerConnector [INFO] >> > [trackingGroup_host061.pe1i.gradientx.com-1391301920791-99d22986], >> > Rebalancing attempt failed. Clearing the cache before the next >> rebalancing >> > operation is triggered >> > >> > Our zookeeper cluster is currently running 3.3.6. >> > >> > >> > >> > >> > On Mon, Dec 23, 2013 at 12:30 PM, Jason Rosenberg <j...@squareup.com> >> > wrote: >> > >> > > We recently upgraded to 3.4.5, so far without incident. But I'd be >> > > interested to know whether we confirm that there are known problems >> with >> > > this! >> > > >> > > Jason >> > > >> > > >> > > On Mon, Dec 23, 2013 at 2:04 PM, Drew Goya <d...@gradientx.com> >> wrote: >> > > >> > > > Thanks, I migrated our ZK cluster over to 3.3 this weekend. >> Hopefully >> > > that >> > > > does it! >> > > > >> > > > >> > > > On Fri, Dec 20, 2013 at 9:09 AM, Jun Rao <jun...@gmail.com> wrote: >> > > > >> > > > > Hmm, not sure how stable 3.4.4. We have been using 3.3.4 and >> haven't >> > > seen >> > > > > issues with ZK as long as there aren't many ZK session >> expirations. >> > > > > >> > > > > Thanks, >> > > > > >> > > > > Jun >> > > > > >> > > > > >> > > > > On Thu, Dec 19, 2013 at 9:41 PM, Drew Goya <d...@gradientx.com> >> > wrote: >> > > > > >> > > > > > Our cluster is currently running 3.4.4. >> > > > > > >> > > > > > I see Kafka is currently using the 3.3.4 client, is there a >> > potential >> > > > > > conflict there? >> > > > > > >> > > > > > >> > > > > > On Wed, Dec 18, 2013 at 9:12 PM, Jun Rao <jun...@gmail.com> >> wrote: >> > > > > > >> > > > > > > The issue is that consumer 007 didn't see consumer 006 during >> > > > > > rebalancing. >> > > > > > > So, it made a decision in conflict with consumer 006. Consumer >> > 007 >> > > > > should >> > > > > > > have another ZK watcher fired to trigger another rebalance >> when >> > if >> > > it >> > > > > > will >> > > > > > > see consumer 006. Which version of ZK are you using? >> > > > > > > >> > > > > > > Thanks, >> > > > > > > >> > > > > > > Jun >> > > > > > > >> > > > > > > >> > > > > > > On Wed, Dec 18, 2013 at 9:38 AM, Drew Goya < >> d...@gradientx.com> >> > > > wrote: >> > > > > > > >> > > > > > > > Thanks for the help with this Jun, really appreciate it! >> So I >> > > > found >> > > > > > this >> > > > > > > > in the logs for consumer 007 about an hour previous. >> Besides >> > > that >> > > > no >> > > > > > > real >> > > > > > > > activity. >> > > > > > > > >> > > > > > > > It looks like 007 rebalanced and successfully claimed >> partition >> > > > > 24-27. >> > > > > > > > Shortly after that its zookeeper client timed out and >> > > reconnected. >> > > > > It >> > > > > > > > didn't rebalance again after this. >> > > > > > > > >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > begin >> > > > > > > > rebalancing consumer >> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 >> try >> > #0 >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherManager [INFO] >> > > > > > > > [ConsumerFetcherManager-1387249529483] Stopping leader >> finder >> > > > thread >> > > > > > > > 2013-12-17 15:51:06 >> ConsumerFetcherManager$LeaderFinderThread >> > > > [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread], >> > > > > > > > Shutting down >> > > > > > > > 2013-12-17 15:51:06 >> ConsumerFetcherManager$LeaderFinderThread >> > > > [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread], >> > > > > > > > Stopped >> > > > > > > > 2013-12-17 15:51:06 >> ConsumerFetcherManager$LeaderFinderThread >> > > > [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread], >> > > > > > > > Shutdown completed >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherManager [INFO] >> > > > > > > > [ConsumerFetcherManager-1387249529483] Stopping all fetchers >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13], >> > > > > > > > Shutting down >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13], >> > > > > > > > Stopped >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13], >> > > > > > > > Shutdown completed >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11], >> > > > > > > > Shutting down >> > > > > > > > 2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to >> > socket >> > > > > > error: >> > > > > > > > null >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11], >> > > > > > > > Stopped >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11], >> > > > > > > > Shutdown completed >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10], >> > > > > > > > Shutting down >> > > > > > > > 2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to >> > socket >> > > > > > error: >> > > > > > > > null >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10], >> > > > > > > > Stopped >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10], >> > > > > > > > Shutdown completed >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12], >> > > > > > > > Shutting down >> > > > > > > > 2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to >> > socket >> > > > > > error: >> > > > > > > > null >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12], >> > > > > > > > Stopped >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12], >> > > > > > > > Shutdown completed >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherManager [INFO] >> > > > > > > > [ConsumerFetcherManager-1387249529483] All connections >> stopped >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > Cleared >> > > > > > all >> > > > > > > > relevant queues for this fetcher >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > Cleared >> > > > > > the >> > > > > > > > data chunks in all the consumer message iterators >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > > > Committing >> > > > > > > > all offsets after clearing the fetcher queues >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > > Releasing >> > > > > > > > partition ownership >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > > Consumer >> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 >> > > > > rebalancing >> > > > > > > the >> > > > > > > > following partitions: List(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, >> > 11, >> > > > 12, >> > > > > > 13, >> > > > > > > > 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, >> 29, >> > > 30, >> > > > > 31, >> > > > > > > 32, >> > > > > > > > 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, >> 48, >> > > 49, >> > > > > 50, >> > > > > > > 51, >> > > > > > > > 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, >> 67, >> > > 68, >> > > > > 69, >> > > > > > > 70, >> > > > > > > > 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, >> 86, >> > > 87, >> > > > > 88, >> > > > > > > 89, >> > > > > > > > 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, >> > 104, >> > > > 105, >> > > > > > > 106, >> > > > > > > > 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, >> > 119, >> > > > 120, >> > > > > > > 121, >> > > > > > > > 122, 123, 124, 125, 126, 127) for topic Events2 with >> consumers: >> > > > > > > > >> > > List(trackingGroup_prod-storm-sup-trk001-1387249529775-2a8484f1-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk001-1387249529775-2a8484f1-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk002-1387249530831-97c586ab-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk002-1387249530831-97c586ab-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk003-1387249529739-f2de3dd9-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk003-1387249529739-f2de3dd9-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk004-1387249530445-8f57ec5c-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk004-1387249530445-8f57ec5c-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk005-1387249530451-d59c669a-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk005-1387249530451-d59c669a-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk005-1387249530452-2b244683-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk005-1387249530452-2b244683-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk008-1387249526700-11ba655b-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk008-1387249526700-11ba655b-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk009-1387249530020-cb36831c-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk009-1387249530020-cb36831c-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk010-1387249529975-d43aff06-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk010-1387249529975-d43aff06-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk011-1387249527684-479a04f9-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk011-1387249527684-479a04f9-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk012-1387249530208-155ecd68-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk012-1387249530208-155ecd68-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk013-1387249530700-b323ee53-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk013-1387249530700-b323ee53-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk014-1387249529916-e32e6363-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk014-1387249529916-e32e6363-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk015-1387249529709-d655ccd4-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk015-1387249529709-d655ccd4-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk016-1387249531064-bc8f8f3e-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk016-1387249531064-bc8f8f3e-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk017-1387249530635-35f505b7-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk017-1387249530635-35f505b7-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk018-1387249530621-84327f5f-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk018-1387249530621-84327f5f-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk019-1387249530418-80afccf9-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk019-1387249530418-80afccf9-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk020-1387249530930-906e99e1-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk020-1387249530930-906e99e1-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk021-1387249529761-705a5bca-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk021-1387249529761-705a5bca-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk022-1387249530347-3d40b4f9-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk022-1387249530347-3d40b4f9-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk023-1387249529067-957d280b-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk023-1387249529067-957d280b-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk024-1387249530625-f8118f02-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk024-1387249530625-f8118f02-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk025-1387249530213-cccfffc8-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk025-1387249530213-cccfffc8-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk046-1387249527798-2164c569-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk046-1387249527798-2164c569-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk047-1387249530559-6b49ce74-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk047-1387249530559-6b49ce74-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk048-1387249529976-aba1e428-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk048-1387249529976-aba1e428-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk050-1387249530465-dc203a62-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk050-1387249530465-dc203a62-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk051-1387249530406-46f7a649-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk051-1387249530406-46f7a649-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk052-1387249530423-e06e4210-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk052-1387249530423-e06e4210-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk054-1387249530369-68e494e6-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk054-1387249530369-68e494e6-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk055-1387249529961-bec0abbc-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk055-1387249529961-bec0abbc-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk056-1387249531590-957c0b49-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk056-1387249531590-957c0b49-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk057-1387249530341-d8476874-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk057-1387249530341-d8476874-1, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk058-1387249530730-20554b4d-0, >> > > > > > > > >> trackingGroup_prod-storm-sup-trk058-1387249530730-20554b4d-1) >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0 >> > > > > attempting >> > > > > > > to >> > > > > > > > claim partition 24 >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0 >> > > > > attempting >> > > > > > > to >> > > > > > > > claim partition 25 >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1 >> > > > > attempting >> > > > > > > to >> > > > > > > > claim partition 26 >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1 >> > > > > attempting >> > > > > > > to >> > > > > > > > claim partition 27 >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0 >> > > > > > successfully >> > > > > > > > owned partition 25 for topic Events2 >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1 >> > > > > > successfully >> > > > > > > > owned partition 26 for topic Events2 >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0 >> > > > > > successfully >> > > > > > > > owned partition 24 for topic Events2 >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1 >> > > > > > successfully >> > > > > > > > owned partition 27 for topic Events2 >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > > Updating >> > > > > > > the >> > > > > > > > cache >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > > > > Consumer >> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 >> > > selected >> > > > > > > > partitions : Events2:24: fetched offset = 969809475: >> consumed >> > > > offset >> > > > > = >> > > > > > > > 969809475,Events2:25: fetched offset = 983792923: consumed >> > > offset = >> > > > > > > > 983792923,Events2:26: fetched offset = 90409778: consumed >> > offset >> > > = >> > > > > > > > 90409778,Events2:27: fetched offset = 979456347: consumed >> > offset >> > > = >> > > > > > > > 979456347 >> > > > > > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] >> > > > > > > > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], >> > end >> > > > > > > > rebalancing consumer >> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 >> try >> > #0 >> > > > > > > > 2013-12-17 15:51:06 >> ConsumerFetcherManager$LeaderFinderThread >> > > > [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread], >> > > > > > > > Starting >> > > > > > > > 2013-12-17 15:51:06 VerifiableProperties [INFO] Verifying >> > > > properties >> > > > > > > > 2013-12-17 15:51:06 VerifiableProperties [INFO] Property >> > > > client.idis >> > > > > > > > overridden to trackingGroup >> > > > > > > > 2013-12-17 15:51:06 VerifiableProperties [INFO] Property >> > > > > > > > metadata.broker.list is overridden to >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> prod-kafka-broker001:9092,prod-kafka-broker010:9092,prod-kafka-broker011:9092,prod-kafka-broker012:9092,prod-kafka-broker013:9092,prod-kafka-broker014:9092,prod-kafka-broker015:9092,prod-kafka-broker002:9092,prod-kafka-broker003:9092,prod-kafka-broker004:9092,prod-kafka-broker005:9092,prod-kafka-broker006:9092,prod-kafka-broker007:9092,prod-kafka-broker008:9092,prod-kafka-broker009:9092 >> > > > > > > > 2013-12-17 15:51:06 VerifiableProperties [INFO] Property >> > > > > > > > request.timeout.msis overridden to 30000 >> > > > > > > > 2013-12-17 15:51:06 ClientUtils$ [INFO] Fetching metadata >> from >> > > > broker >> > > > > > > > id:15,host:prod-kafka-broker015,port:9092 with correlation >> id >> > 23 >> > > > for >> > > > > 1 >> > > > > > > > topic(s) Set(Events2) >> > > > > > > > 2013-12-17 15:51:06 SyncProducer [INFO] Connected to >> > > > > > > > prod-kafka-broker015:9092 for producing >> > > > > > > > 2013-12-17 15:51:06 SyncProducer [INFO] Disconnecting from >> > > > > > > > prod-kafka-broker015:9092 >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-3], >> > > > > > > > Starting >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-4], >> > > > > > > > Starting >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-9], >> > > > > > > > Starting >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-2], >> > > > > > > > Starting >> > > > > > > > 2013-12-17 15:51:06 ConsumerFetcherManager [INFO] >> > > > > > > > [ConsumerFetcherManager-1387249529483] Added fetcher for >> > > partitions >> > > > > > > > ArrayBuffer([[Events2,25], initOffset 983792923 to broker >> > > > > > > > id:3,host:prod-kafka-broker003,port:9092] , [[Events2,26], >> > > > initOffset >> > > > > > > > 90409778 to broker >> id:4,host:prod-kafka-broker004,port:9092] , >> > > > > > > > [[Events2,24], initOffset 969809475 to broker >> > > > > > > > id:2,host:prod-kafka-broker002,port:9092] , [[Events2,27], >> > > > initOffset >> > > > > > > > 979456347 to broker >> id:9,host:prod-kafka-broker009,port:9092] ) >> > > > > > > > 2013-12-17 15:51:15 executor [ERROR] >> > > > > > > > 2013-12-17 15:51:15 executor [ERROR] >> > > > > > > > 2013-12-17 15:51:15 ClientCnxn [INFO] Client session timed >> out, >> > > > have >> > > > > > not >> > > > > > > > heard from server in 5015ms for sessionid 0x342e4febc180841, >> > > > closing >> > > > > > > socket >> > > > > > > > connection and attempting reconnect >> > > > > > > > 2013-12-17 15:51:15 ZkClient [INFO] zookeeper state changed >> > > > > > > (Disconnected) >> > > > > > > > 2013-12-17 15:51:16 ClientCnxn [INFO] Opening socket >> connection >> > > to >> > > > > > server >> > > > > > > > prod-zookeeper-kafka002/10.4.34.186:2181 >> > > > > > > > 2013-12-17 15:51:16 ClientCnxn [INFO] Socket connection >> > > established >> > > > > to >> > > > > > > > prod-zookeeper-kafka002/10.4.34.186:2181, initiating >> session >> > > > > > > > 2013-12-17 15:51:16 ClientCnxn [INFO] Session establishment >> > > > complete >> > > > > on >> > > > > > > > server prod-zookeeper-kafka002/10.4.34.186:2181, sessionid >> = >> > > > > > > > 0x342e4febc180841, negotiated timeout = 6000 >> > > > > > > > 2013-12-17 15:51:16 ZkClient [INFO] zookeeper state changed >> > > > > > > (SyncConnected) >> > > > > > > > 2013-12-17 15:51:16 executor [ERROR] >> > > > > > > > 2013-12-17 15:51:20 executor [ERROR] >> > > > > > > > 2013-12-17 15:51:24 executor [ERROR] >> > > > > > > > >> > > > > > > > >> > > > > > > > On Tue, Dec 17, 2013 at 9:24 PM, Jun Rao <jun...@gmail.com> >> > > wrote: >> > > > > > > > >> > > > > > > > > What's consumer trackingGroup_prod-storm-sup-trk007 doing >> at >> > > the >> > > > > same >> > > > > > > > time? >> > > > > > > > > It's the one that caused the conflict in ZK. >> > > > > > > > > >> > > > > > > > > Thanks, >> > > > > > > > > >> > > > > > > > > Jun >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > On Tue, Dec 17, 2013 at 9:19 PM, Drew Goya < >> > d...@gradientx.com >> > > > >> > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > I explored that possibility but I'm not seeing any ZK >> > session >> > > > > > > > expirations >> > > > > > > > > > in the logs and it doesn't look like these rebalances >> > > complete. >> > > > > > > > > > >> > > > > > > > > > They fail due to conflicts in the zookeeper data >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > On Tue, Dec 17, 2013 at 8:53 PM, Jun Rao < >> jun...@gmail.com >> > > >> > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Have you looked at >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog >> > > > > > > > > > > ? >> > > > > > > > > > > >> > > > > > > > > > > Thanks, >> > > > > > > > > > > >> > > > > > > > > > > Jun >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > On Tue, Dec 17, 2013 at 9:24 AM, Drew Goya < >> > > > d...@gradientx.com >> > > > > > >> > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Hey all, >> > > > > > > > > > > > >> > > > > > > > > > > > I've recently been having problems with consumer >> groups >> > > > > > > > rebalancing. >> > > > > > > > > > I'm >> > > > > > > > > > > > using several high level consumers which all belong >> to >> > > the >> > > > > same >> > > > > > > > > group. >> > > > > > > > > > > > Occasionally one or two of them will get stuck in a >> > > > > rebalance >> > > > > > > > loop. >> > > > > > > > > > > They >> > > > > > > > > > > > attempt to rebalance, but the partitions they try to >> > > claim >> > > > > are >> > > > > > > > owned. >> > > > > > > > > > > > Anyone run into this? Ideas? >> > > > > > > > > > > > >> > > > > > > > > > > > I see errors in my zookeeper logs like: >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:12:31,171 [myid:001] - INFO >> > > > > [ProcessThread(sid:1 >> > > > > > > > > > > > cport:-1)::PrepRequestProcessor@627] - Got >> user-level >> > > > > > > > > KeeperException >> > > > > > > > > > > when >> > > > > > > > > > > > processing sessionid:0x342e4febc180852 type:create >> > > > > cxid:0x1a9a >> > > > > > > > > > > > zxid:0x501390d4b txntype:-1 reqpath:n/a Error >> > > > > > > > > > > > >> Path:/kafka/consumers/trackingGroup/owners/Events2/25 >> > > > > > > > > > > Error:KeeperErrorCode >> > > > > > > > > > > > = NodeExists for >> > > > > > /kafka/consumers/trackingGroup/owners/Events2/25 >> > > > > > > > > > > > >> > > > > > > > > > > > And errors in my kafka logs like: >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:32 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > begin >> > > > > > > > > > > > rebalancing consumer >> > > > > > > > > > > > >> > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306 >> > > > > try >> > > > > > #8 >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO] >> > > > > > > > > > > > [ConsumerFetcherManager-1387249530381] Stopping >> leader >> > > > finder >> > > > > > > > thread >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO] >> > > > > > > > > > > > [ConsumerFetcherManager-1387249530381] Stopping all >> > > > fetchers >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO] >> > > > > > > > > > > > [ConsumerFetcherManager-1387249530381] All >> connections >> > > > > stopped >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > Cleared >> > > > > > > > > > all >> > > > > > > > > > > > relevant queues for this fetcher >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > Cleared >> > > > > > > > > > the >> > > > > > > > > > > > data chunks in all the consumer message iterators >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > > > Committing >> > > > > > > > > > > > all offsets after clearing the fetcher queues >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > > Releasing >> > > > > > > > > > > > partition ownership >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > > Consumer >> > > > > > > > > > > > >> > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306 >> > > > > > > > > rebalancing >> > > > > > > > > > > the >> > > > > > > > > > > > following partitions: List(0, 1, 2, 3, 4, 5, 6, 7, >> 8, >> > 9, >> > > > 10, >> > > > > > 11, >> > > > > > > > 12, >> > > > > > > > > > 13, >> > > > > > > > > > > > 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, >> 27, >> > > 28, >> > > > > 29, >> > > > > > > 30, >> > > > > > > > > 31, >> > > > > > > > > > > 32, >> > > > > > > > > > > > 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, >> 46, >> > > 47, >> > > > > 48, >> > > > > > > 49, >> > > > > > > > > 50, >> > > > > > > > > > > 51, >> > > > > > > > > > > > 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, >> 65, >> > > 66, >> > > > > 67, >> > > > > > > 68, >> > > > > > > > > 69, >> > > > > > > > > > > 70, >> > > > > > > > > > > > 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, >> 84, >> > > 85, >> > > > > 86, >> > > > > > > 87, >> > > > > > > > > 88, >> > > > > > > > > > > 89, >> > > > > > > > > > > > 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, >> 102, >> > > 103, >> > > > > > 104, >> > > > > > > > 105, >> > > > > > > > > > > 106, >> > > > > > > > > > > > 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, >> 117, >> > > 118, >> > > > > > 119, >> > > > > > > > 120, >> > > > > > > > > > > 121, >> > > > > > > > > > > > 122, 123, 124, 125, 126, 127) for topic Events2 with >> > > > > consumers: >> > > > > > > > > > > > >> > > > > > > >> > List(trackingGroup_prod-storm-sup-trk001-1387249529775-2a8484f1-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk001-1387249529775-2a8484f1-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk002-1387249530831-97c586ab-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk002-1387249530831-97c586ab-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk003-1387249529739-f2de3dd9-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk003-1387249529739-f2de3dd9-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk004-1387249530445-8f57ec5c-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk004-1387249530445-8f57ec5c-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk005-1387249530451-d59c669a-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk005-1387249530451-d59c669a-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk005-1387249530452-2b244683-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk005-1387249530452-2b244683-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk008-1387249526700-11ba655b-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk008-1387249526700-11ba655b-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk009-1387249530020-cb36831c-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk009-1387249530020-cb36831c-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk010-1387249529975-d43aff06-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk010-1387249529975-d43aff06-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk011-1387249527684-479a04f9-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk011-1387249527684-479a04f9-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk012-1387249530208-155ecd68-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk012-1387249530208-155ecd68-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk013-1387249530700-b323ee53-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk013-1387249530700-b323ee53-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk014-1387249529916-e32e6363-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk014-1387249529916-e32e6363-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk015-1387249529709-d655ccd4-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk015-1387249529709-d655ccd4-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk016-1387249531064-bc8f8f3e-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk016-1387249531064-bc8f8f3e-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk017-1387249530635-35f505b7-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk017-1387249530635-35f505b7-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk018-1387249530621-84327f5f-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk018-1387249530621-84327f5f-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk019-1387249530418-80afccf9-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk019-1387249530418-80afccf9-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk020-1387249530930-906e99e1-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk020-1387249530930-906e99e1-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk021-1387249529761-705a5bca-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk021-1387249529761-705a5bca-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk022-1387249530347-3d40b4f9-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk022-1387249530347-3d40b4f9-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk023-1387249529067-957d280b-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk023-1387249529067-957d280b-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk024-1387249530625-f8118f02-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk024-1387249530625-f8118f02-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk025-1387249530213-cccfffc8-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk025-1387249530213-cccfffc8-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk046-1387249527798-2164c569-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk046-1387249527798-2164c569-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk047-1387249530559-6b49ce74-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk047-1387249530559-6b49ce74-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk048-1387249529976-aba1e428-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk048-1387249529976-aba1e428-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk050-1387249530465-dc203a62-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk050-1387249530465-dc203a62-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk051-1387249530406-46f7a649-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk051-1387249530406-46f7a649-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk052-1387249530423-e06e4210-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk052-1387249530423-e06e4210-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk054-1387249530369-68e494e6-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk054-1387249530369-68e494e6-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk055-1387249529961-bec0abbc-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk055-1387249529961-bec0abbc-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk056-1387249531590-957c0b49-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk056-1387249531590-957c0b49-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk057-1387249530341-d8476874-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk057-1387249530341-d8476874-1, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk058-1387249530730-20554b4d-0, >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk058-1387249530730-20554b4d-1) >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0 >> > > > > > > > > attempting >> > > > > > > > > > > to >> > > > > > > > > > > > claim partition 24 >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0 >> > > > > > > > > attempting >> > > > > > > > > > > to >> > > > > > > > > > > > claim partition 25 >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1 >> > > > > > > > > attempting >> > > > > > > > > > > to >> > > > > > > > > > > > claim partition 26 >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1 >> > > > > > > > > attempting >> > > > > > > > > > > to >> > > > > > > > > > > > claim partition 27 >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZkUtils$ [INFO] conflict in >> > > > > > > > > > > > /consumers/trackingGroup/owners/Events2/25 data: >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0 >> > > > > > > stored >> > > > > > > > > > data: >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0 >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > waiting >> > > > > > > > > > for >> > > > > > > > > > > > the partition ownership to be deleted: 25 >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZkUtils$ [INFO] conflict in >> > > > > > > > > > > > /consumers/trackingGroup/owners/Events2/26 data: >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1 >> > > > > > > stored >> > > > > > > > > > data: >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1 >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > waiting >> > > > > > > > > > for >> > > > > > > > > > > > the partition ownership to be deleted: 26 >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZkUtils$ [INFO] conflict in >> > > > > > > > > > > > /consumers/trackingGroup/owners/Events2/24 data: >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0 >> > > > > > > stored >> > > > > > > > > > data: >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0 >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > waiting >> > > > > > > > > > for >> > > > > > > > > > > > the partition ownership to be deleted: 24 >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZkUtils$ [INFO] conflict in >> > > > > > > > > > > > /consumers/trackingGroup/owners/Events2/27 data: >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1 >> > > > > > > stored >> > > > > > > > > > data: >> > > > > > > > > > > > >> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1 >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > waiting >> > > > > > > > > > for >> > > > > > > > > > > > the partition ownership to be deleted: 27 >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > end >> > > > > > > > > > > > rebalancing consumer >> > > > > > > > > > > > >> > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306 >> > > > > try >> > > > > > #8 >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > > > Rebalancing >> > > > > > > > > > > > attempt failed. Clearing the cache before the next >> > > > > rebalancing >> > > > > > > > > > operation >> > > > > > > > > > > is >> > > > > > > > > > > > triggered >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO] >> > > > > > > > > > > > [ConsumerFetcherManager-1387249530381] Stopping >> leader >> > > > finder >> > > > > > > > thread >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO] >> > > > > > > > > > > > [ConsumerFetcherManager-1387249530381] Stopping all >> > > > fetchers >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO] >> > > > > > > > > > > > [ConsumerFetcherManager-1387249530381] All >> connections >> > > > > stopped >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > Cleared >> > > > > > > > > > all >> > > > > > > > > > > > relevant queues for this fetcher >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > Cleared >> > > > > > > > > > the >> > > > > > > > > > > > data chunks in all the consumer message iterators >> > > > > > > > > > > > >> > > > > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector >> [INFO] >> > > > > > > > > > > > >> > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], >> > > > > > > > > > Committing >> > > > > > > > > > > > all offsets after clearing the fetcher queues >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >
