Ricardo, The parallelism of each logical consumer (consumer group) is the number of partitions. So with four partitions it could make sense to have one logical consumer (application) have two processes on different machines each with two threads, or one process with four. While with two logical consumers (two different applications) you would want each to have 4 threads (4*2 = 8 threads total).
There are also considerations depending on which consumer code you are using (which I'm decidedly not someone with good information on) Christian On Wed, Jan 28, 2015 at 1:28 PM, Ricardo Ferreira < jricardoferre...@gmail.com> wrote: > Hi experts, > > I'm newbie in the Kafka world, so excuse me for such basic question. > > I'm in the process of designing a client for Kafka, and after few hours of > study, I was told that to achieve a proper level of parallelism, it is a > best practice having one thread for each partition of an topic. > > My question is that this rule-of-thumb also applies for multiple consumer > applications. For instance: > > Considering a topic with 4 partitions, it is OK to have one consumer > application with 4 threads, just like would be OK to have two consumer > applications with 2 threads each. But what about having two consumer > applications with 4 threads each? It would break any load-balancing made by > Kafka brokers? > > Anyway, I'd like to understand if the proper number of threads that should > match the number of partitions is per application or if there is some other > best practice. > > Thanks in advance, > > Ricardo Ferreira >
