Hi Eno,

Yes, I was concerned that machine2 ispite of being free is not utilized at
all.

Partitions would be 60 and streams would have 63 threads. So, I understand
now why threads were not started on machine2.

For others, 1 partition would be consumed by 1 thread. So, thus the
mismatch.

Thanks,
-Sameer.

On Fri, Jun 23, 2017 at 11:02 PM, Eno Thereska <eno.there...@gmail.com>
wrote:

> Hi Sameer,
>
> Could you elaborate on your question? Are you concerned that machine2 does
> not have any tasks in the beginning?
>
> Could you share your streams configuration? In particular how many threads
> does each stream instance have? Also how many topics and partitions do you
> have?
>
>
> Thanks,
> Eno
>
> > On 23 Jun 2017, at 17:31, Sameer Kumar <sam.kum.w...@gmail.com> wrote:
> >
> > Hi,
> >
> > Came across a rebalancing issue in using KafkaStreams. I have two
> machines,
> > Machine1 and Machine2, machine1 is consuming all partitions and machine2
> is
> > completely free and not processing any partitions. If I shutdown
> machine1,
> > then machine2 will take over and would start consuming all partitions.
> >
> > But, in this scenario, its not using the complete cluster.
> >
> > *Machine 1*
> > 2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO  StreamThread:858 - stream-thread
> > [StreamThread-27] Creating active task 1_2 with assigned partitions
> > [LIC3-43-lic3-deb-ci2-43-repartition-2,
> > LIC3-43-lic3-cnt-ci-43-repartition-2]
> > 2017-06-23 21:54:07 INFO  StreamThread:163 - stream-thread
> > [StreamThread-20] State transition from PARTITIONS_REVOKED to
> > ASSIGNING_PARTITIONS.
> > 2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO  StreamThread:163 - stream-thread
> > [StreamThread-31] State transition from PARTITIONS_REVOKED to
> > ASSIGNING_PARTITIONS.
> > 2017-06-23 21:54:07 INFO  StreamThread:163 - stream-thread
> > [StreamThread-13] State transition from PARTITIONS_REVOKED to
> > ASSIGNING_PARTITIONS.
> > 2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO  StreamThread:163 - stream-thread
> > [StreamThread-16] State transition from ASSIGNING_PARTITIONS to RUNNING.
> > 2017-06-23 21:54:07 INFO  StreamThread:858 - stream-thread
> > [StreamThread-18] Creating active task 0_4 with assigned partitions
> > [testS5-4]
> > 2017-06-23 21:54:07 INFO  StreamThread:858 - stream-thread
> > [StreamThread-13] Creating active task 0_2 with assigned partitions
> > [testS5-2]
> > 2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO  StreamThread:858 - stream-thread
> > [StreamThread-31] Creating active task 2_0 with assigned partitions
> > [LIC3-43-lic3-cnt-li-43-repartition-0,
> > LIC3-43-lic3-cnt-li-43_1-repartition-0]
> > 2017-06-23 21:54:07 INFO  StreamThread:858 - stream-thread
> > [StreamThread-20] Creating active task 0_9 with assigned partitions
> > [testS5-9]
> > 2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO  StreamThread:858 - stream-thread
> > [StreamThread-29] Creating active task 2_5 with assigned partitions
> > [LIC3-43-lic3-cnt-li-43-repartition-5,
> > LIC3-43-lic3-cnt-li-43_1-repartition-5]
> >
> >
> > *Machine2*
> >
> > 21:54:08 INFO  StreamThread:248 - stream-thread [StreamThread-8] at state
> > RUNNING: partitions [] revoked at the beginning of consumer rebalance.
> > 2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
> > [StreamThread-5] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
> > [StreamThread-5] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
> > [StreamThread-5] Removing all standby tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
> > [StreamThread-18] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:163 - stream-thread
> [StreamThread-8]
> > State transition from RUNNING to PARTITIONS_REVOKED.
> > 2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
> > [StreamThread-18] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
> > [StreamThread-18] Removing all standby tasks []
> > 2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
> > [StreamThread-1] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
> > [StreamThread-15] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
> > [StreamThread-15] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
> > [StreamThread-30] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
> > [StreamThread-14] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
> > [StreamThread-15] Removing all standby tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
> > [StreamThread-1] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
> > [StreamThread-1] Removing all standby tasks []
> > 2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
> > [StreamThread-14] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
> > [StreamThread-30] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
> > [StreamThread-30] Removing all standby tasks []
> > 2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
> > [StreamThread-14] Removing all standby tasks []
> > 2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
> > [StreamThread-20] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
> > [StreamThread-20] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
> > [StreamThread-6] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
> > [StreamThread-9] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
> > [StreamThread-9] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
> > [StreamThread-9] Removing all standby tasks []
> > 2017-06-23 21:54:08 INFO  AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO  KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
> > [StreamThread-4] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1042 - stream-thread
> > [StreamThread-31] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1049 - stream-thread
> > [StreamThread-31] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO  StreamThread:1064 - stream-thread
> > [StreamThread-31] Removing all standby tasks []
> >
> >
> > Regards,
> > -Sameer.
>
>

Reply via email to