Hi Akshay, In regards to your 3rd question (and indirectly to your 2nd question), instead of having different consumer groups, why not just multiple consumers in the same group? That would ensure that each consumer only reads from one partition in the topic. You can even assign the partition if you want. That might be better than having different groups read the same topic if they only care about one partition. Keep in mind if you are going to have distinct data in each partition, you will want to either specify the partition you publish to or use a key.
I would also curious on your first question. Thanks, Subhash On Fri, Jan 18, 2019 at 11:58 AM Akshay Dhokale <akshaydhokale...@gmail.com> wrote: > Hello, > > > > We are planning to use Apache Kafka in our system to replace local MSMQs > (.net message queues). > > > > *Single Environment* > > Currently, we have about 100 message queues per environment. > > In Kafka, we plan to use approximately 20 topics to accommodate the 100 > message queues. > > We will have 100 consumer groups (1 per each message queue) that read from > these topics. > > > > For example, existing queues A, B, C, D will be replaced by a single Kafka > Topic. > > There will be 4 consumer groups that read from this topic. > > Each consumer group will receive all messages in the topic but will only > process the messages for its assigned queue. It will discard the other > messages. > > > > *One Data Center* > > In our current deployment architecture, we host multiple environments in a > single data center. (About 100 environments in one data center) > > We plan to use one (or a few) Kafka cluster for one data center. > > > > With this architecture, one Kafka cluster will have about 2K topics (20 per > environment X 100 environments). > > And there will be about 10K consumer groups. > > > > Can you kindly provide your recommendations and best practices for this > use-case and scale particularly around the below > > 1. Are there any limits on the number of topics? Is it okay to have 2K > topics in a Kafka cluster? > 2. Are there any limits on the number of consumer groups? Is it okay to > have 10K consumer groups in a cluster? > 3. The consumer groups will be discarding a lot of messages meant for > other message queues. Are there any recommended approach for this > scenario > or ways to avoid it? > > > > Thank You, > > Akshay Dhokale >
