Re: [External] Allow parallel processing

2019-11-18 Thread Eric Azama
I second Dave's suggestion. With regards to the consumers round-robining between topics, they usually round-robin in batches. So you'll probably see a consumer work on a large batch of records from TopicA before moving on to TopicB. Depending on the behavior of the producers this might appear the

Re: [External] Allow parallel processing

2019-11-18 Thread Tauzell, Dave
I would go with #1: 1. It will be easier to add new "batch producers" since you won't need to worry about re-partitioning 2. You have more control over the parallelism since you can have different numbers of partitions for each topic 3. You can easily split out your consumer into N consumers if