@Matthias Can you help with this since I remember having conversation with you in the past on this topic wherein it was mentioned that partition assignment to stream task logic might change in future releases
On Mon, Jun 13, 2022, 11:05 Pushkar Deole <pdeole2...@gmail.com> wrote: > Hi, > > I have a microservice hosting kafka streams application. I have 3 > instances of microservice hosted in 3 pods, each is having 2 kafka stream > threads, thus total 6 stream threads as part of consumer group. > There are 3 source topics: A, B, C each having 12, 6, 6 partitions > respectively i.e. total 24 source topic partitions. > > Now, the issue I am facing is the distribution of source topic partitions > among stream threads. Considering that 6 streams threads and overall 24 > topic partitions, each stream thread is assigned 4 partitions, so no issue > there. However the main issue arises is that of partitions assigned to > stream thread from each topic. So, sometimes, I am getting 4 partitions > from Topic A assigned to stream thread 1, while other stream thread will > get partitions from Topic B and C only. > > What I am expecting is, partitions from each topic will get evenly > distributed among 6 threads, so each stream thread should get 2 partitions > of topic A, and 1 partition of topic B and C each, thus making 4 partitions > of input topics. This is required for proper load balancing since topic A > carries a lot more event traffic than topic B and C due to which partitions > of each topic should be distributed evenly across all stream threads. > > I am using kafka stream 2.5.1 > Is this the default behavior of kafka streams 2.5.1 client? Is there any > configuration to change this behaviour? > Is there any change in logic in later versions of streams that would > guarantee the distribution the way I described above? >
