Hi, I have a microservice hosting kafka streams application. I have 3 instances of microservice hosted in 3 pods, each is having 2 kafka stream threads, thus total 6 stream threads as part of consumer group. There are 3 source topics: A, B, C each having 12, 6, 6 partitions respectively i.e. total 24 source topic partitions.
Now, the issue I am facing is the distribution of source topic partitions among stream threads. Considering that 6 streams threads and overall 24 topic partitions, each stream thread is assigned 4 partitions, so no issue there. However the main issue arises is that of partitions assigned to stream thread from each topic. So, sometimes, I am getting 4 partitions from Topic A assigned to stream thread 1, while other stream thread will get partitions from Topic B and C only. What I am expecting is, partitions from each topic will get evenly distributed among 6 threads, so each stream thread should get 2 partitions of topic A, and 1 partition of topic B and C each, thus making 4 partitions of input topics. This is required for proper load balancing since topic A carries a lot more event traffic than topic B and C due to which partitions of each topic should be distributed evenly across all stream threads. I am using kafka stream 2.5.1 Is this the default behavior of kafka streams 2.5.1 client? Is there any configuration to change this behaviour? Is there any change in logic in later versions of streams that would guarantee the distribution the way I described above?
