We have a Kafka Streams (2.4) app consisting of 5 instances. It reads from a Kafka topic with 20 partitions (5 brokers).
We notice that the partition assignment does not always lead to well distributed load over the different threads. We notice this at startup as well as after a recovery of a failed thread. 1. At startup, some instances get a significantly lower load and sometimes even no load. It seems like instances that come up slightly later get no partitions assigned (because of sticky assignment?). 2. When one thread (container) dies and comes back it often does not receive any or very few partitions to work on. We assume this has to do with the sticky assignment. Is there any way we can make this distribution more equal? I was also wondering whether Kafka Streams takes into account colocation of Kafka brokers with stream processing threads when assigning partitions. Do partitions on brokers get assigned to the streams thread that is colocated with it on the same machine?
