Hi Sameer, >From your shared logs it seems the number of "threads" is actually larger than the number of topic partitions (note each machine seem to have lots of threads). More specifically, I have only saw max. of 6 partitions for any topics, e.g.:
LIC3-43-lic3-deb-ci2-43-repartition-0 LIC3-43-lic3-deb-ci2-43-repartition-1 LIC3-43-lic3-deb-ci2-43-repartition-2 ... LIC3-43-lic3-deb-ci2-43-repartition-5 If that is the case, and assume you only have one sub-topology, then only 6 threads out of the TOTAL number of threads across your machines will ever get any data: this is because within a sub-topology, the number of tasks (i.e. the unit of parallelism that can be distributed across threads) will but the MAX(#.partitions of input topics). Guozhang On Fri, Jun 23, 2017 at 7:32 PM, Sameer Kumar <sam.kum.w...@gmail.com> wrote: > Hi Eno, > > Yes, I was concerned that machine2 ispite of being free is not utilized at > all. > > Partitions would be 60 and streams would have 63 threads. So, I understand > now why threads were not started on machine2. > > For others, 1 partition would be consumed by 1 thread. So, thus the > mismatch. > > Thanks, > -Sameer. > > On Fri, Jun 23, 2017 at 11:02 PM, Eno Thereska <eno.there...@gmail.com> > wrote: > > > Hi Sameer, > > > > Could you elaborate on your question? Are you concerned that machine2 > does > > not have any tasks in the beginning? > > > > Could you share your streams configuration? In particular how many > threads > > does each stream instance have? Also how many topics and partitions do > you > > have? > > > > > > Thanks, > > Eno > > > > > On 23 Jun 2017, at 17:31, Sameer Kumar <sam.kum.w...@gmail.com> wrote: > > > > > > Hi, > > > > > > Came across a rebalancing issue in using KafkaStreams. I have two > > machines, > > > Machine1 and Machine2, machine1 is consuming all partitions and > machine2 > > is > > > completely free and not processing any partitions. If I shutdown > > machine1, > > > then machine2 will take over and would start consuming all partitions. > > > > > > But, in this scenario, its not using the complete cluster. > > > > > > *Machine 1* > > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread > > > [StreamThread-27] Creating active task 1_2 with assigned partitions > > > [LIC3-43-lic3-deb-ci2-43-repartition-2, > > > LIC3-43-lic3-cnt-ci-43-repartition-2] > > > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread > > > [StreamThread-20] State transition from PARTITIONS_REVOKED to > > > ASSIGNING_PARTITIONS. > > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread > > > [StreamThread-31] State transition from PARTITIONS_REVOKED to > > > ASSIGNING_PARTITIONS. > > > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread > > > [StreamThread-13] State transition from PARTITIONS_REVOKED to > > > ASSIGNING_PARTITIONS. > > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread > > > [StreamThread-16] State transition from ASSIGNING_PARTITIONS to > RUNNING. > > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread > > > [StreamThread-18] Creating active task 0_4 with assigned partitions > > > [testS5-4] > > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread > > > [StreamThread-13] Creating active task 0_2 with assigned partitions > > > [testS5-2] > > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread > > > [StreamThread-31] Creating active task 2_0 with assigned partitions > > > [LIC3-43-lic3-cnt-li-43-repartition-0, > > > LIC3-43-lic3-cnt-li-43_1-repartition-0] > > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread > > > [StreamThread-20] Creating active task 0_9 with assigned partitions > > > [testS5-9] > > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread > > > [StreamThread-29] Creating active task 2_5 with assigned partitions > > > [LIC3-43-lic3-cnt-li-43-repartition-5, > > > LIC3-43-lic3-cnt-li-43_1-repartition-5] > > > > > > > > > *Machine2* > > > > > > 21:54:08 INFO StreamThread:248 - stream-thread [StreamThread-8] at > state > > > RUNNING: partitions [] revoked at the beginning of consumer rebalance. > > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > > LIC3-43 > > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > > [StreamThread-5] Updating suspended tasks to contain active tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > > [StreamThread-5] Removing all active tasks [] > > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > > [StreamThread-5] Removing all standby tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > > [StreamThread-18] Updating suspended tasks to contain active tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:163 - stream-thread > > [StreamThread-8] > > > State transition from RUNNING to PARTITIONS_REVOKED. > > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > > [StreamThread-18] Removing all active tasks [] > > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > > LIC3-43 > > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > > [StreamThread-18] Removing all standby tasks [] > > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > > [StreamThread-1] Updating suspended tasks to contain active tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > > [StreamThread-15] Updating suspended tasks to contain active tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > > [StreamThread-15] Removing all active tasks [] > > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > > LIC3-43 > > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > > [StreamThread-30] Updating suspended tasks to contain active tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > > [StreamThread-14] Updating suspended tasks to contain active tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > > [StreamThread-15] Removing all standby tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > > [StreamThread-1] Removing all active tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > > [StreamThread-1] Removing all standby tasks [] > > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > > LIC3-43 > > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > > LIC3-43 > > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > > [StreamThread-14] Removing all active tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > > [StreamThread-30] Removing all active tasks [] > > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > > [StreamThread-30] Removing all standby tasks [] > > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > > LIC3-43 > > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > > [StreamThread-14] Removing all standby tasks [] > > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > > [StreamThread-20] Updating suspended tasks to contain active tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > > [StreamThread-20] Removing all active tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > > [StreamThread-6] Updating suspended tasks to contain active tasks [] > > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > > [StreamThread-9] Updating suspended tasks to contain active tasks [] > > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > > LIC3-43 > > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > > [StreamThread-9] Removing all active tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > > [StreamThread-9] Removing all standby tasks [] > > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group > > > LIC3-43 > > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client > > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from > > > REBALANCING to REBALANCING. > > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > > [StreamThread-4] Updating suspended tasks to contain active tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread > > > [StreamThread-31] Updating suspended tasks to contain active tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread > > > [StreamThread-31] Removing all active tasks [] > > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread > > > [StreamThread-31] Removing all standby tasks [] > > > > > > > > > Regards, > > > -Sameer. > > > > > -- -- Guozhang
