Hi,

Came across a rebalancing issue in using KafkaStreams. I have two machines,
Machine1 and Machine2, machine1 is consuming all partitions and machine2 is
completely free and not processing any partitions. If I shutdown machine1,
then machine2 will take over and would start consuming all partitions.

But, in this scenario, its not using the complete cluster.

*Machine 1*
2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO  StreamThread:858 - stream-thread
[StreamThread-27] Creating active task 1_2 with assigned partitions
[LIC3-43-lic3-deb-ci2-43-repartition-2,
LIC3-43-lic3-cnt-ci-43-repartition-2]
2017-06-23 21:54:07 INFO  StreamThread:163 - stream-thread
[StreamThread-20] State transition from PARTITIONS_REVOKED to
ASSIGNING_PARTITIONS.
2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO  StreamThread:163 - stream-thread
[StreamThread-31] State transition from PARTITIONS_REVOKED to
ASSIGNING_PARTITIONS.
2017-06-23 21:54:07 INFO  StreamThread:163 - stream-thread
[StreamThread-13] State transition from PARTITIONS_REVOKED to
ASSIGNING_PARTITIONS.
2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO  StreamThread:163 - stream-thread
[StreamThread-16] State transition from ASSIGNING_PARTITIONS to RUNNING.
2017-06-23 21:54:07 INFO  StreamThread:858 - stream-thread
[StreamThread-18] Creating active task 0_4 with assigned partitions
[testS5-4]
2017-06-23 21:54:07 INFO  StreamThread:858 - stream-thread
[StreamThread-13] Creating active task 0_2 with assigned partitions
[testS5-2]
2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO  StreamThread:858 - stream-thread
[StreamThread-31] Creating active task 2_0 with assigned partitions
[LIC3-43-lic3-cnt-li-43-repartition-0,
LIC3-43-lic3-cnt-li-43_1-repartition-0]
2017-06-23 21:54:07 INFO  StreamThread:858 - stream-thread
[StreamThread-20] Creating active task 0_9 with assigned partitions
[testS5-9]
2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO  StreamThread:858 - stream-thread
[StreamThread-29] Creating active task 2_5 with assigned partitions
[LIC3-43-lic3-cnt-li-43-repartition-5,
LIC3-43-lic3-cnt-li-43_1-repartition-5]


*Machine2*

 21:54:08 INFO  StreamThread:248 - stream-thread [StreamThread-8] at state
RUNNING: partitions [] revoked at the beginning of consumer rebalance.
2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
[StreamThread-5] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
[StreamThread-5] Removing all active tasks []
2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
[StreamThread-5] Removing all standby tasks []
2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
[StreamThread-18] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO  StreamThread:163 - stream-thread [StreamThread-8]
State transition from RUNNING to PARTITIONS_REVOKED.
2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
[StreamThread-18] Removing all active tasks []
2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
[StreamThread-18] Removing all standby tasks []
2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
[StreamThread-1] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
[StreamThread-15] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
[StreamThread-15] Removing all active tasks []
2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
[StreamThread-30] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
[StreamThread-14] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
[StreamThread-15] Removing all standby tasks []
2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
[StreamThread-1] Removing all active tasks []
2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
[StreamThread-1] Removing all standby tasks []
2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
[StreamThread-14] Removing all active tasks []
2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
[StreamThread-30] Removing all active tasks []
2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
[StreamThread-30] Removing all standby tasks []
2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
[StreamThread-14] Removing all standby tasks []
2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
[StreamThread-20] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
[StreamThread-20] Removing all active tasks []
2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
[StreamThread-6] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
[StreamThread-9] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
[StreamThread-9] Removing all active tasks []
2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
[StreamThread-9] Removing all standby tasks []
2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
[StreamThread-4] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
[StreamThread-31] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
[StreamThread-31] Removing all active tasks []
2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
[StreamThread-31] Removing all standby tasks []


Regards,
-Sameer.

Reply via email to