HI Vinicius, This question seems like a better fit for the user mailing list rather than the developer mailing list.
Anyway, if I understand correctly, you are asking if the producer can choose to assign partitions in a round-robin fashion rather than based on the key. The answer is, you can, by using RoundRobinPartitioner. (again, if I'm understanding the question correctly). best, Colin On Tue, Jun 9, 2020, at 00:48, Vinicius Scheidegger wrote: > Anyone? > > On Fri, Jun 5, 2020 at 2:42 PM Vinicius Scheidegger < > vinicius.scheideg...@gmail.com> wrote: > > > Does anyone know how could I perform a load balance to distribute equally > > the messages to all consumers within the same consumer group having > > multiple producers? > > > > Is this a conceptual flaw on Kafka, wasn't it thought for equal > > distribution with multiple producers or am I missing something? > > I've asked on Stack Overflow, on Kafka users mailing group, here (on Kafka > > Devs) and on Slack - and still have no definitive answer (actually most of > > the time I got no answer at all) > > > > Would something like this even be possible in the way Kafka is currently > > designed? > > How does proposing for a KIP work? > > > > Thanks, > > > > > > > > On Thu, May 28, 2020, 3:44 PM Vinicius Scheidegger < > > vinicius.scheideg...@gmail.com> wrote: > > > >> Hi, > >> > >> I'm trying to understand a little bit more about how Kafka works. > >> I have a design with multiple producers writing to a single topic and > >> multiple consumers in a single Consumer Group consuming message from this > >> topic. > >> > >> My idea is to distribute the messages from all producers equally. From > >> reading the documentation I understood that the partition is always > >> selected by the producer. Is that correct? > >> > >> I'd also like to know if there is an out of the box option to assign the > >> partition via a round robin *on the broker side *to guarantee equal > >> distribution of the load - if possible to each consumer, but if not > >> possible, at least to each partition. > >> > >> If my understanding is correct, it looks like in a multiple producer > >> scenario there is lack of support from Kafka regarding load balancing and > >> customers have to either stick to the hash of the key (random distribution, > >> although it would guarantee same key goes to the same partition) or they > >> have to create their own logic on the producer side (i.e. by sharing > >> memory) > >> > >> Am I missing something? > >> > >> Thank you, > >> > >> Vinicius Scheidegger > >> > > >
