Thanks for the help with this Jun, really appreciate it! So I found this in the logs for consumer 007 about an hour previous. Besides that no real activity.
It looks like 007 rebalanced and successfully claimed partition 24-27. Shortly after that its zookeeper client timed out and reconnected. It didn't rebalance again after this. 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], begin rebalancing consumer trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 try #0 2013-12-17 15:51:06 ConsumerFetcherManager [INFO] [ConsumerFetcherManager-1387249529483] Stopping leader finder thread 2013-12-17 15:51:06 ConsumerFetcherManager$LeaderFinderThread [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread], Shutting down 2013-12-17 15:51:06 ConsumerFetcherManager$LeaderFinderThread [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread], Stopped 2013-12-17 15:51:06 ConsumerFetcherManager$LeaderFinderThread [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread], Shutdown completed 2013-12-17 15:51:06 ConsumerFetcherManager [INFO] [ConsumerFetcherManager-1387249529483] Stopping all fetchers 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13], Shutting down 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13], Stopped 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13], Shutdown completed 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11], Shutting down 2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to socket error: null 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11], Stopped 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11], Shutdown completed 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10], Shutting down 2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to socket error: null 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10], Stopped 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10], Shutdown completed 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12], Shutting down 2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to socket error: null 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12], Stopped 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12], Shutdown completed 2013-12-17 15:51:06 ConsumerFetcherManager [INFO] [ConsumerFetcherManager-1387249529483] All connections stopped 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Cleared all relevant queues for this fetcher 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Cleared the data chunks in all the consumer message iterators 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Committing all offsets after clearing the fetcher queues 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Releasing partition ownership 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Consumer trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 rebalancing the following partitions: List(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127) for topic Events2 with consumers: List(trackingGroup_prod-storm-sup-trk001-1387249529775-2a8484f1-0, trackingGroup_prod-storm-sup-trk001-1387249529775-2a8484f1-1, trackingGroup_prod-storm-sup-trk002-1387249530831-97c586ab-0, trackingGroup_prod-storm-sup-trk002-1387249530831-97c586ab-1, trackingGroup_prod-storm-sup-trk003-1387249529739-f2de3dd9-0, trackingGroup_prod-storm-sup-trk003-1387249529739-f2de3dd9-1, trackingGroup_prod-storm-sup-trk004-1387249530445-8f57ec5c-0, trackingGroup_prod-storm-sup-trk004-1387249530445-8f57ec5c-1, trackingGroup_prod-storm-sup-trk005-1387249530451-d59c669a-0, trackingGroup_prod-storm-sup-trk005-1387249530451-d59c669a-1, trackingGroup_prod-storm-sup-trk005-1387249530452-2b244683-0, trackingGroup_prod-storm-sup-trk005-1387249530452-2b244683-1, trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0, trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1, trackingGroup_prod-storm-sup-trk008-1387249526700-11ba655b-0, trackingGroup_prod-storm-sup-trk008-1387249526700-11ba655b-1, trackingGroup_prod-storm-sup-trk009-1387249530020-cb36831c-0, trackingGroup_prod-storm-sup-trk009-1387249530020-cb36831c-1, trackingGroup_prod-storm-sup-trk010-1387249529975-d43aff06-0, trackingGroup_prod-storm-sup-trk010-1387249529975-d43aff06-1, trackingGroup_prod-storm-sup-trk011-1387249527684-479a04f9-0, trackingGroup_prod-storm-sup-trk011-1387249527684-479a04f9-1, trackingGroup_prod-storm-sup-trk012-1387249530208-155ecd68-0, trackingGroup_prod-storm-sup-trk012-1387249530208-155ecd68-1, trackingGroup_prod-storm-sup-trk013-1387249530700-b323ee53-0, trackingGroup_prod-storm-sup-trk013-1387249530700-b323ee53-1, trackingGroup_prod-storm-sup-trk014-1387249529916-e32e6363-0, trackingGroup_prod-storm-sup-trk014-1387249529916-e32e6363-1, trackingGroup_prod-storm-sup-trk015-1387249529709-d655ccd4-0, trackingGroup_prod-storm-sup-trk015-1387249529709-d655ccd4-1, trackingGroup_prod-storm-sup-trk016-1387249531064-bc8f8f3e-0, trackingGroup_prod-storm-sup-trk016-1387249531064-bc8f8f3e-1, trackingGroup_prod-storm-sup-trk017-1387249530635-35f505b7-0, trackingGroup_prod-storm-sup-trk017-1387249530635-35f505b7-1, trackingGroup_prod-storm-sup-trk018-1387249530621-84327f5f-0, trackingGroup_prod-storm-sup-trk018-1387249530621-84327f5f-1, trackingGroup_prod-storm-sup-trk019-1387249530418-80afccf9-0, trackingGroup_prod-storm-sup-trk019-1387249530418-80afccf9-1, trackingGroup_prod-storm-sup-trk020-1387249530930-906e99e1-0, trackingGroup_prod-storm-sup-trk020-1387249530930-906e99e1-1, trackingGroup_prod-storm-sup-trk021-1387249529761-705a5bca-0, trackingGroup_prod-storm-sup-trk021-1387249529761-705a5bca-1, trackingGroup_prod-storm-sup-trk022-1387249530347-3d40b4f9-0, trackingGroup_prod-storm-sup-trk022-1387249530347-3d40b4f9-1, trackingGroup_prod-storm-sup-trk023-1387249529067-957d280b-0, trackingGroup_prod-storm-sup-trk023-1387249529067-957d280b-1, trackingGroup_prod-storm-sup-trk024-1387249530625-f8118f02-0, trackingGroup_prod-storm-sup-trk024-1387249530625-f8118f02-1, trackingGroup_prod-storm-sup-trk025-1387249530213-cccfffc8-0, trackingGroup_prod-storm-sup-trk025-1387249530213-cccfffc8-1, trackingGroup_prod-storm-sup-trk046-1387249527798-2164c569-0, trackingGroup_prod-storm-sup-trk046-1387249527798-2164c569-1, trackingGroup_prod-storm-sup-trk047-1387249530559-6b49ce74-0, trackingGroup_prod-storm-sup-trk047-1387249530559-6b49ce74-1, trackingGroup_prod-storm-sup-trk048-1387249529976-aba1e428-0, trackingGroup_prod-storm-sup-trk048-1387249529976-aba1e428-1, trackingGroup_prod-storm-sup-trk050-1387249530465-dc203a62-0, trackingGroup_prod-storm-sup-trk050-1387249530465-dc203a62-1, trackingGroup_prod-storm-sup-trk051-1387249530406-46f7a649-0, trackingGroup_prod-storm-sup-trk051-1387249530406-46f7a649-1, trackingGroup_prod-storm-sup-trk052-1387249530423-e06e4210-0, trackingGroup_prod-storm-sup-trk052-1387249530423-e06e4210-1, trackingGroup_prod-storm-sup-trk054-1387249530369-68e494e6-0, trackingGroup_prod-storm-sup-trk054-1387249530369-68e494e6-1, trackingGroup_prod-storm-sup-trk055-1387249529961-bec0abbc-0, trackingGroup_prod-storm-sup-trk055-1387249529961-bec0abbc-1, trackingGroup_prod-storm-sup-trk056-1387249531590-957c0b49-0, trackingGroup_prod-storm-sup-trk056-1387249531590-957c0b49-1, trackingGroup_prod-storm-sup-trk057-1387249530341-d8476874-0, trackingGroup_prod-storm-sup-trk057-1387249530341-d8476874-1, trackingGroup_prod-storm-sup-trk058-1387249530730-20554b4d-0, trackingGroup_prod-storm-sup-trk058-1387249530730-20554b4d-1) 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0 attempting to claim partition 24 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0 attempting to claim partition 25 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1 attempting to claim partition 26 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1 attempting to claim partition 27 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0 successfully owned partition 25 for topic Events2 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1 successfully owned partition 26 for topic Events2 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0 successfully owned partition 24 for topic Events2 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1 successfully owned partition 27 for topic Events2 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Updating the cache 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Consumer trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 selected partitions : Events2:24: fetched offset = 969809475: consumed offset = 969809475,Events2:25: fetched offset = 983792923: consumed offset = 983792923,Events2:26: fetched offset = 90409778: consumed offset = 90409778,Events2:27: fetched offset = 979456347: consumed offset = 979456347 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], end rebalancing consumer trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 try #0 2013-12-17 15:51:06 ConsumerFetcherManager$LeaderFinderThread [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread], Starting 2013-12-17 15:51:06 VerifiableProperties [INFO] Verifying properties 2013-12-17 15:51:06 VerifiableProperties [INFO] Property client.id is overridden to trackingGroup 2013-12-17 15:51:06 VerifiableProperties [INFO] Property metadata.broker.list is overridden to prod-kafka-broker001:9092,prod-kafka-broker010:9092,prod-kafka-broker011:9092,prod-kafka-broker012:9092,prod-kafka-broker013:9092,prod-kafka-broker014:9092,prod-kafka-broker015:9092,prod-kafka-broker002:9092,prod-kafka-broker003:9092,prod-kafka-broker004:9092,prod-kafka-broker005:9092,prod-kafka-broker006:9092,prod-kafka-broker007:9092,prod-kafka-broker008:9092,prod-kafka-broker009:9092 2013-12-17 15:51:06 VerifiableProperties [INFO] Property request.timeout.msis overridden to 30000 2013-12-17 15:51:06 ClientUtils$ [INFO] Fetching metadata from broker id:15,host:prod-kafka-broker015,port:9092 with correlation id 23 for 1 topic(s) Set(Events2) 2013-12-17 15:51:06 SyncProducer [INFO] Connected to prod-kafka-broker015:9092 for producing 2013-12-17 15:51:06 SyncProducer [INFO] Disconnecting from prod-kafka-broker015:9092 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-3], Starting 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-4], Starting 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-9], Starting 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-2], Starting 2013-12-17 15:51:06 ConsumerFetcherManager [INFO] [ConsumerFetcherManager-1387249529483] Added fetcher for partitions ArrayBuffer([[Events2,25], initOffset 983792923 to broker id:3,host:prod-kafka-broker003,port:9092] , [[Events2,26], initOffset 90409778 to broker id:4,host:prod-kafka-broker004,port:9092] , [[Events2,24], initOffset 969809475 to broker id:2,host:prod-kafka-broker002,port:9092] , [[Events2,27], initOffset 979456347 to broker id:9,host:prod-kafka-broker009,port:9092] ) 2013-12-17 15:51:15 executor [ERROR] 2013-12-17 15:51:15 executor [ERROR] 2013-12-17 15:51:15 ClientCnxn [INFO] Client session timed out, have not heard from server in 5015ms for sessionid 0x342e4febc180841, closing socket connection and attempting reconnect 2013-12-17 15:51:15 ZkClient [INFO] zookeeper state changed (Disconnected) 2013-12-17 15:51:16 ClientCnxn [INFO] Opening socket connection to server prod-zookeeper-kafka002/10.4.34.186:2181 2013-12-17 15:51:16 ClientCnxn [INFO] Socket connection established to prod-zookeeper-kafka002/10.4.34.186:2181, initiating session 2013-12-17 15:51:16 ClientCnxn [INFO] Session establishment complete on server prod-zookeeper-kafka002/10.4.34.186:2181, sessionid = 0x342e4febc180841, negotiated timeout = 6000 2013-12-17 15:51:16 ZkClient [INFO] zookeeper state changed (SyncConnected) 2013-12-17 15:51:16 executor [ERROR] 2013-12-17 15:51:20 executor [ERROR] 2013-12-17 15:51:24 executor [ERROR] On Tue, Dec 17, 2013 at 9:24 PM, Jun Rao <jun...@gmail.com> wrote: > What's consumer trackingGroup_prod-storm-sup-trk007 doing at the same time? > It's the one that caused the conflict in ZK. > > Thanks, > > Jun > > > On Tue, Dec 17, 2013 at 9:19 PM, Drew Goya <d...@gradientx.com> wrote: > > > I explored that possibility but I'm not seeing any ZK session expirations > > in the logs and it doesn't look like these rebalances complete. > > > > They fail due to conflicts in the zookeeper data > > > > > > On Tue, Dec 17, 2013 at 8:53 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > Have you looked at > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog > > > ? > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Tue, Dec 17, 2013 at 9:24 AM, Drew Goya <d...@gradientx.com> wrote: > > > > > > > Hey all, > > > > > > > > I've recently been having problems with consumer groups rebalancing. > > I'm > > > > using several high level consumers which all belong to the same > group. > > > > Occasionally one or two of them will get stuck in a rebalance loop. > > > They > > > > attempt to rebalance, but the partitions they try to claim are owned. > > > > Anyone run into this? Ideas? > > > > > > > > I see errors in my zookeeper logs like: > > > > > > > > 2013-12-17 17:12:31,171 [myid:001] - INFO [ProcessThread(sid:1 > > > > cport:-1)::PrepRequestProcessor@627] - Got user-level > KeeperException > > > when > > > > processing sessionid:0x342e4febc180852 type:create cxid:0x1a9a > > > > zxid:0x501390d4b txntype:-1 reqpath:n/a Error > > > > Path:/kafka/consumers/trackingGroup/owners/Events2/25 > > > Error:KeeperErrorCode > > > > = NodeExists for /kafka/consumers/trackingGroup/owners/Events2/25 > > > > > > > > And errors in my kafka logs like: > > > > > > > > 2013-12-17 17:20:32 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], begin > > > > rebalancing consumer > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306 try #8 > > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO] > > > > [ConsumerFetcherManager-1387249530381] Stopping leader finder thread > > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO] > > > > [ConsumerFetcherManager-1387249530381] Stopping all fetchers > > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO] > > > > [ConsumerFetcherManager-1387249530381] All connections stopped > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], Cleared > > all > > > > relevant queues for this fetcher > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], Cleared > > the > > > > data chunks in all the consumer message iterators > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], > > Committing > > > > all offsets after clearing the fetcher queues > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], > Releasing > > > > partition ownership > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], > Consumer > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306 > rebalancing > > > the > > > > following partitions: List(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, > > 13, > > > > 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, > 31, > > > 32, > > > > 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, > 50, > > > 51, > > > > 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, > 69, > > > 70, > > > > 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, > 88, > > > 89, > > > > 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, > > > 106, > > > > 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, > > > 121, > > > > 122, 123, 124, 125, 126, 127) for topic Events2 with consumers: > > > > List(trackingGroup_prod-storm-sup-trk001-1387249529775-2a8484f1-0, > > > > trackingGroup_prod-storm-sup-trk001-1387249529775-2a8484f1-1, > > > > trackingGroup_prod-storm-sup-trk002-1387249530831-97c586ab-0, > > > > trackingGroup_prod-storm-sup-trk002-1387249530831-97c586ab-1, > > > > trackingGroup_prod-storm-sup-trk003-1387249529739-f2de3dd9-0, > > > > trackingGroup_prod-storm-sup-trk003-1387249529739-f2de3dd9-1, > > > > trackingGroup_prod-storm-sup-trk004-1387249530445-8f57ec5c-0, > > > > trackingGroup_prod-storm-sup-trk004-1387249530445-8f57ec5c-1, > > > > trackingGroup_prod-storm-sup-trk005-1387249530451-d59c669a-0, > > > > trackingGroup_prod-storm-sup-trk005-1387249530451-d59c669a-1, > > > > trackingGroup_prod-storm-sup-trk005-1387249530452-2b244683-0, > > > > trackingGroup_prod-storm-sup-trk005-1387249530452-2b244683-1, > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0, > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1, > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0, > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1, > > > > trackingGroup_prod-storm-sup-trk008-1387249526700-11ba655b-0, > > > > trackingGroup_prod-storm-sup-trk008-1387249526700-11ba655b-1, > > > > trackingGroup_prod-storm-sup-trk009-1387249530020-cb36831c-0, > > > > trackingGroup_prod-storm-sup-trk009-1387249530020-cb36831c-1, > > > > trackingGroup_prod-storm-sup-trk010-1387249529975-d43aff06-0, > > > > trackingGroup_prod-storm-sup-trk010-1387249529975-d43aff06-1, > > > > trackingGroup_prod-storm-sup-trk011-1387249527684-479a04f9-0, > > > > trackingGroup_prod-storm-sup-trk011-1387249527684-479a04f9-1, > > > > trackingGroup_prod-storm-sup-trk012-1387249530208-155ecd68-0, > > > > trackingGroup_prod-storm-sup-trk012-1387249530208-155ecd68-1, > > > > trackingGroup_prod-storm-sup-trk013-1387249530700-b323ee53-0, > > > > trackingGroup_prod-storm-sup-trk013-1387249530700-b323ee53-1, > > > > trackingGroup_prod-storm-sup-trk014-1387249529916-e32e6363-0, > > > > trackingGroup_prod-storm-sup-trk014-1387249529916-e32e6363-1, > > > > trackingGroup_prod-storm-sup-trk015-1387249529709-d655ccd4-0, > > > > trackingGroup_prod-storm-sup-trk015-1387249529709-d655ccd4-1, > > > > trackingGroup_prod-storm-sup-trk016-1387249531064-bc8f8f3e-0, > > > > trackingGroup_prod-storm-sup-trk016-1387249531064-bc8f8f3e-1, > > > > trackingGroup_prod-storm-sup-trk017-1387249530635-35f505b7-0, > > > > trackingGroup_prod-storm-sup-trk017-1387249530635-35f505b7-1, > > > > trackingGroup_prod-storm-sup-trk018-1387249530621-84327f5f-0, > > > > trackingGroup_prod-storm-sup-trk018-1387249530621-84327f5f-1, > > > > trackingGroup_prod-storm-sup-trk019-1387249530418-80afccf9-0, > > > > trackingGroup_prod-storm-sup-trk019-1387249530418-80afccf9-1, > > > > trackingGroup_prod-storm-sup-trk020-1387249530930-906e99e1-0, > > > > trackingGroup_prod-storm-sup-trk020-1387249530930-906e99e1-1, > > > > trackingGroup_prod-storm-sup-trk021-1387249529761-705a5bca-0, > > > > trackingGroup_prod-storm-sup-trk021-1387249529761-705a5bca-1, > > > > trackingGroup_prod-storm-sup-trk022-1387249530347-3d40b4f9-0, > > > > trackingGroup_prod-storm-sup-trk022-1387249530347-3d40b4f9-1, > > > > trackingGroup_prod-storm-sup-trk023-1387249529067-957d280b-0, > > > > trackingGroup_prod-storm-sup-trk023-1387249529067-957d280b-1, > > > > trackingGroup_prod-storm-sup-trk024-1387249530625-f8118f02-0, > > > > trackingGroup_prod-storm-sup-trk024-1387249530625-f8118f02-1, > > > > trackingGroup_prod-storm-sup-trk025-1387249530213-cccfffc8-0, > > > > trackingGroup_prod-storm-sup-trk025-1387249530213-cccfffc8-1, > > > > trackingGroup_prod-storm-sup-trk046-1387249527798-2164c569-0, > > > > trackingGroup_prod-storm-sup-trk046-1387249527798-2164c569-1, > > > > trackingGroup_prod-storm-sup-trk047-1387249530559-6b49ce74-0, > > > > trackingGroup_prod-storm-sup-trk047-1387249530559-6b49ce74-1, > > > > trackingGroup_prod-storm-sup-trk048-1387249529976-aba1e428-0, > > > > trackingGroup_prod-storm-sup-trk048-1387249529976-aba1e428-1, > > > > trackingGroup_prod-storm-sup-trk050-1387249530465-dc203a62-0, > > > > trackingGroup_prod-storm-sup-trk050-1387249530465-dc203a62-1, > > > > trackingGroup_prod-storm-sup-trk051-1387249530406-46f7a649-0, > > > > trackingGroup_prod-storm-sup-trk051-1387249530406-46f7a649-1, > > > > trackingGroup_prod-storm-sup-trk052-1387249530423-e06e4210-0, > > > > trackingGroup_prod-storm-sup-trk052-1387249530423-e06e4210-1, > > > > trackingGroup_prod-storm-sup-trk054-1387249530369-68e494e6-0, > > > > trackingGroup_prod-storm-sup-trk054-1387249530369-68e494e6-1, > > > > trackingGroup_prod-storm-sup-trk055-1387249529961-bec0abbc-0, > > > > trackingGroup_prod-storm-sup-trk055-1387249529961-bec0abbc-1, > > > > trackingGroup_prod-storm-sup-trk056-1387249531590-957c0b49-0, > > > > trackingGroup_prod-storm-sup-trk056-1387249531590-957c0b49-1, > > > > trackingGroup_prod-storm-sup-trk057-1387249530341-d8476874-0, > > > > trackingGroup_prod-storm-sup-trk057-1387249530341-d8476874-1, > > > > trackingGroup_prod-storm-sup-trk058-1387249530730-20554b4d-0, > > > > trackingGroup_prod-storm-sup-trk058-1387249530730-20554b4d-1) > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0 > attempting > > > to > > > > claim partition 24 > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0 > attempting > > > to > > > > claim partition 25 > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1 > attempting > > > to > > > > claim partition 26 > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1 > attempting > > > to > > > > claim partition 27 > > > > > > > > 2013-12-17 17:20:33 ZkUtils$ [INFO] conflict in > > > > /consumers/trackingGroup/owners/Events2/25 data: > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0 stored > > data: > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0 > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], waiting > > for > > > > the partition ownership to be deleted: 25 > > > > > > > > 2013-12-17 17:20:33 ZkUtils$ [INFO] conflict in > > > > /consumers/trackingGroup/owners/Events2/26 data: > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1 stored > > data: > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1 > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], waiting > > for > > > > the partition ownership to be deleted: 26 > > > > > > > > 2013-12-17 17:20:33 ZkUtils$ [INFO] conflict in > > > > /consumers/trackingGroup/owners/Events2/24 data: > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0 stored > > data: > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0 > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], waiting > > for > > > > the partition ownership to be deleted: 24 > > > > > > > > 2013-12-17 17:20:33 ZkUtils$ [INFO] conflict in > > > > /consumers/trackingGroup/owners/Events2/27 data: > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1 stored > > data: > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1 > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], waiting > > for > > > > the partition ownership to be deleted: 27 > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], end > > > > rebalancing consumer > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306 try #8 > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], > > Rebalancing > > > > attempt failed. Clearing the cache before the next rebalancing > > operation > > > is > > > > triggered > > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO] > > > > [ConsumerFetcherManager-1387249530381] Stopping leader finder thread > > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO] > > > > [ConsumerFetcherManager-1387249530381] Stopping all fetchers > > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO] > > > > [ConsumerFetcherManager-1387249530381] All connections stopped > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], Cleared > > all > > > > relevant queues for this fetcher > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], Cleared > > the > > > > data chunks in all the consumer message iterators > > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO] > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306], > > Committing > > > > all offsets after clearing the fetcher queues > > > > > > > > > >
