Hi Eno, Yes, I was concerned that machine2 ispite of being free is not utilized at all.
Partitions would be 60 and streams would have 63 threads. So, I understand now why threads were not started on machine2. For others, 1 partition would be consumed by 1 thread. So, thus the mismatch. Thanks, -Sameer. On Fri, Jun 23, 2017 at 11:02 PM, Eno Thereska <eno.there...@gmail.com> wrote: > Hi Sameer, > > Could you elaborate on your question? Are you concerned that machine2 does > not have any tasks in the beginning? > > Could you share your streams configuration? In particular how many threads > does each stream instance have? Also how many topics and partitions do you > have? > > > Thanks, > Eno > > > On 23 Jun 2017, at 17:31, Sameer Kumar <sam.kum.w...@gmail.com> wrote: > > > > Hi, > > > > Came across a rebalancing issue in using KafkaStreams. I have two > machines, > > Machine1 and Machine2, machine1 is consuming all partitions and machine2 > is > > completely free and not processing any partitions. If I shutdown > machine1, > > then machine2 will take over and would start consuming all partitions. > > > > But, in this scenario, its not using the complete cluster. > > > > *Machine 1* > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread > > [StreamThread-27] Creating active task 1_2 with assigned partitions > > [LIC3-43-lic3-deb-ci2-43-repartition-2, > > LIC3-43-lic3-cnt-ci-43-repartition-2] > > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread > > [StreamThread-20] State transition from PARTITIONS_REVOKED to > > ASSIGNING_PARTITIONS. > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread > > [StreamThread-31] State transition from PARTITIONS_REVOKED to > > ASSIGNING_PARTITIONS. > > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread > > [StreamThread-13] State transition from PARTITIONS_REVOKED to > > ASSIGNING_PARTITIONS. > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread > > [StreamThread-16] State transition from ASSIGNING_PARTITIONS to RUNNING. > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread > > [StreamThread-18] Creating active task 0_4 with assigned partitions > > [testS5-4] > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread > > [StreamThread-13] Creating active task 0_2 with assigned partitions > > [testS5-2] > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread > > [StreamThread-31] Creating active task 2_0 with assigned partitions > > [LIC3-43-lic3-cnt-li-43-repartition-0, > > LIC3-43-lic3-cnt-li-43_1-repartition-0] > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread > > [StreamThread-20] Creating active task 0_9 with assigned partitions > > [testS5-9] > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread > > [StreamThread-29] Creating active task 2_5 with assigned partitions > > [LIC3-43-lic3-cnt-li-43-repartition-5, > > LIC3-43-lic3-cnt-li-43_1-repartition-5] > > > > > > *Machine2* > > > > 21:54:08 INFO StreamThread:248 - stream-thread [StreamThread-8] at state > > RUNNING: partitions [] revoked at the beginning of consumer rebalance. > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > LIC3-43 > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > [StreamThread-5] Updating suspended tasks to contain active tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > [StreamThread-5] Removing all active tasks [] > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > [StreamThread-5] Removing all standby tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > [StreamThread-18] Updating suspended tasks to contain active tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:163 - stream-thread > [StreamThread-8] > > State transition from RUNNING to PARTITIONS_REVOKED. > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > [StreamThread-18] Removing all active tasks [] > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > LIC3-43 > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > [StreamThread-18] Removing all standby tasks [] > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > [StreamThread-1] Updating suspended tasks to contain active tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > [StreamThread-15] Updating suspended tasks to contain active tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > [StreamThread-15] Removing all active tasks [] > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > LIC3-43 > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > [StreamThread-30] Updating suspended tasks to contain active tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > [StreamThread-14] Updating suspended tasks to contain active tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > [StreamThread-15] Removing all standby tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > [StreamThread-1] Removing all active tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > [StreamThread-1] Removing all standby tasks [] > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > LIC3-43 > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > LIC3-43 > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > [StreamThread-14] Removing all active tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > [StreamThread-30] Removing all active tasks [] > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > [StreamThread-30] Removing all standby tasks [] > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > LIC3-43 > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > [StreamThread-14] Removing all standby tasks [] > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > [StreamThread-20] Updating suspended tasks to contain active tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > [StreamThread-20] Removing all active tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > [StreamThread-6] Updating suspended tasks to contain active tasks [] > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > [StreamThread-9] Updating suspended tasks to contain active tasks [] > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > LIC3-43 > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > [StreamThread-9] Removing all active tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > [StreamThread-9] Removing all standby tasks [] > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > LIC3-43 > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > REBALANCING to REBALANCING. > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > [StreamThread-4] Updating suspended tasks to contain active tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > [StreamThread-31] Updating suspended tasks to contain active tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > [StreamThread-31] Removing all active tasks [] > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > [StreamThread-31] Removing all standby tasks [] > > > > > > Regards, > > -Sameer. > >
