Hi, Came across a rebalancing issue in using KafkaStreams. I have two machines, Machine1 and Machine2, machine1 is consuming all partitions and machine2 is completely free and not processing any partitions. If I shutdown machine1, then machine2 will take over and would start consuming all partitions.
But, in this scenario, its not using the complete cluster. *Machine 1* 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread [StreamThread-27] Creating active task 1_2 with assigned partitions [LIC3-43-lic3-deb-ci2-43-repartition-2, LIC3-43-lic3-cnt-ci-43-repartition-2] 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread [StreamThread-20] State transition from PARTITIONS_REVOKED to ASSIGNING_PARTITIONS. 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread [StreamThread-31] State transition from PARTITIONS_REVOKED to ASSIGNING_PARTITIONS. 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread [StreamThread-13] State transition from PARTITIONS_REVOKED to ASSIGNING_PARTITIONS. 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread [StreamThread-16] State transition from ASSIGNING_PARTITIONS to RUNNING. 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread [StreamThread-18] Creating active task 0_4 with assigned partitions [testS5-4] 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread [StreamThread-13] Creating active task 0_2 with assigned partitions [testS5-2] 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread [StreamThread-31] Creating active task 2_0 with assigned partitions [LIC3-43-lic3-cnt-li-43-repartition-0, LIC3-43-lic3-cnt-li-43_1-repartition-0] 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread [StreamThread-20] Creating active task 0_9 with assigned partitions [testS5-9] 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread [StreamThread-29] Creating active task 2_5 with assigned partitions [LIC3-43-lic3-cnt-li-43-repartition-5, LIC3-43-lic3-cnt-li-43_1-repartition-5] *Machine2* 21:54:08 INFO StreamThread:248 - stream-thread [StreamThread-8] at state RUNNING: partitions [] revoked at the beginning of consumer rebalance. 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group LIC3-43 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread [StreamThread-5] Updating suspended tasks to contain active tasks [] 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread [StreamThread-5] Removing all active tasks [] 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread [StreamThread-5] Removing all standby tasks [] 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread [StreamThread-18] Updating suspended tasks to contain active tasks [] 2017-06-23 21:54:08 INFO StreamThread:163 - stream-thread [StreamThread-8] State transition from RUNNING to PARTITIONS_REVOKED. 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread [StreamThread-18] Removing all active tasks [] 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group LIC3-43 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread [StreamThread-18] Removing all standby tasks [] 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread [StreamThread-1] Updating suspended tasks to contain active tasks [] 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread [StreamThread-15] Updating suspended tasks to contain active tasks [] 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread [StreamThread-15] Removing all active tasks [] 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group LIC3-43 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread [StreamThread-30] Updating suspended tasks to contain active tasks [] 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread [StreamThread-14] Updating suspended tasks to contain active tasks [] 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread [StreamThread-15] Removing all standby tasks [] 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread [StreamThread-1] Removing all active tasks [] 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread [StreamThread-1] Removing all standby tasks [] 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group LIC3-43 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group LIC3-43 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread [StreamThread-14] Removing all active tasks [] 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread [StreamThread-30] Removing all active tasks [] 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread [StreamThread-30] Removing all standby tasks [] 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group LIC3-43 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread [StreamThread-14] Removing all standby tasks [] 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread [StreamThread-20] Updating suspended tasks to contain active tasks [] 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread [StreamThread-20] Removing all active tasks [] 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread [StreamThread-6] Updating suspended tasks to contain active tasks [] 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread [StreamThread-9] Updating suspended tasks to contain active tasks [] 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group LIC3-43 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread [StreamThread-9] Removing all active tasks [] 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread [StreamThread-9] Removing all standby tasks [] 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group LIC3-43 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from REBALANCING to REBALANCING. 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread [StreamThread-4] Updating suspended tasks to contain active tasks [] 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread [StreamThread-31] Updating suspended tasks to contain active tasks [] 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread [StreamThread-31] Removing all active tasks [] 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread [StreamThread-31] Removing all standby tasks [] Regards, -Sameer.
