Hey, I have a typical scenario of a kafka-streams application in a production environment.
We have a kafka-cluster with multiple topics. Messages from one topic is being consumed by a the kafka-streams application. The topic, currently, has 9 partitions. We have configured consumer thread count to 14. We are running 2 instances of this stream application on 2 different machines, thereby consisting of 28 threads across both machines. The group id for the consumers are the same. But, what I observe is that all partitions are being assigned to threads on a single machine. Now, I do understand that if the task on the active machine fails, then the threads in the other machine would take over. My question is that is there a way that kafka-streams can auto-balance across instances of the same stream application ? If yes, how do I go about doing that ? Please let me know. Thanks, Best, Karthik Prasad Senior Software Engineer Sony Interactive Entertainment
