Does anyone know how could I perform a load balance to distribute equally the messages to all consumers within the same consumer group having multiple producers?
Is this a conceptual flaw on Kafka, wasn't it thought for equal distribution with multiple producers or am I missing something? I've asked on Stack Overflow, on Kafka users mailing group, here (on Kafka Devs) and on Slack - and still have no definitive answer (actually most of the time I got no answer at all) Would something like this even be possible in the way Kafka is currently designed? How does proposing for a KIP work? Thanks, On Thu, May 28, 2020, 3:44 PM Vinicius Scheidegger < vinicius.scheideg...@gmail.com> wrote: > Hi, > > I'm trying to understand a little bit more about how Kafka works. > I have a design with multiple producers writing to a single topic and > multiple consumers in a single Consumer Group consuming message from this > topic. > > My idea is to distribute the messages from all producers equally. From > reading the documentation I understood that the partition is always > selected by the producer. Is that correct? > > I'd also like to know if there is an out of the box option to assign the > partition via a round robin *on the broker side *to guarantee equal > distribution of the load - if possible to each consumer, but if not > possible, at least to each partition. > > If my understanding is correct, it looks like in a multiple producer > scenario there is lack of support from Kafka regarding load balancing and > customers have to either stick to the hash of the key (random distribution, > although it would guarantee same key goes to the same partition) or they > have to create their own logic on the producer side (i.e. by sharing memory) > > Am I missing something? > > Thank you, > > Vinicius Scheidegger >
