Hi there, I have a kafka application that uses kafka consumer low-level api to help us process data from a single partition concurrently. Our use case is to send out 800k messages per sec. We are able to do that with 4 boxes using 10k threads and each request taking 50ms in a thread. (1000/50*10000*4)
I understand that kafka in general uses partitions as its parallelism model. It is my understanding that if I want the exact same behavior with kafka streams, I'd need to create 40k partitions for this topic. Is that right? What is the overhead on creating thousands of partitions? If we end up wanting to send out millions of messages per second, is increasing the partitions the only way? Best, Praveen
