Also attempted another pattern where 1. Created a topic T with 'n' partitions. 2. Created a consumer group process with 'n + 1' threads subscribing from topic 'T' with a groupID 'y' 3. Added another consumer group process with 'n + 1' threads subscribing from same topic 'T' with same groupID 'z' (Note that 2 and 3 subscribe from same topic but different groups)
Can a single topic with multiple partitions abetted with multiple consumer groups increase parallelism is consumption? On Wed, Sep 11, 2013 at 4:48 PM, prashant amar <amasin...@gmail.com> wrote: > A Design Question that needs verification: > > 1. Created a topic T with 'n' partitions. > 2. Created a consumer group process with 'n + 1' threads subscribing from > topic 'T' with a groupID 'y' > 3. Added another consumer group process with 'n + 1' threads subscribing > from same topic 'T' with same groupID 'y' > > On doing so, I noticed that the previous consumer group stops consuming > and the new consumer beings to consume > > I was attempting to model on demand parallelization in an event where an > consumer group cannot keep up with the events produced. Rather than > increase the threadpool capacity in the same process, it would make sense > to distribute the load across multiple processes. > > Advice please? > > Regards > Amardeep >
