Re: Broker side round robin on topic partitions when receiving messages

I understand your point of view... My requirement is *exact* balancing - parts of our current flow have a consumption processing of around 5 minutes... (and this is an important/expensive part - because it's CPU and memory intensive and we'd like to avoid queueing) so we need EQUAL load balancing -

Re: Broker side round robin on topic partitions when receiving messages

This is a bit frustrating since you keep saying that the load is not balanced, but the load actually is balanced, it's just balanced in an approximate fashion. If you need exact balancing (for example, because you're creating a job scheduler or something), then you need to use a different strat

Re: Broker side round robin on topic partitions when receiving messages

Hi Collin, One producer shouldn't need to know about the other to distribute the load equally, but what Kafka has now is roughly equal... If you have a single producer RounRobinPartitioner works fine, if you have 10 producers you can have 7/8 messages in one partition while another partition has n

Re: Broker side round robin on topic partitions when receiving messages

Hi Vinicius, It's actually not necessary for one producer to know about the others to get an even distribution across partitions, right? All that's really required is that all producers produce a roughly equal amount of data to each partition, which is what RoundRobinPartitioner is designed to

Re: Broker side round robin on topic partitions when receiving messages

Vinicius, If you have an idea of what you want , and how it is going to improve current Kafka offering perhaps you could submit a KiP ? The guidelines are provided on Apache Kafka Wiki (Confluence). Thanks, On Sun, 14 Jun 2020 at 22:26, Vinicius Scheidegger < vinicius.scheideg...@gmail.com> wr

Re: Broker side round robin on topic partitions when receiving messages

Hi Collin, Thanks for the reply. Actually the RoundRobinPartitioner won't do an equal distribution when working with multiple producers. One producer does not know the others. If you consider that producers are randomly producing messages, in the worst case scenario all producers can be synced and

Re: Broker side round robin on topic partitions when receiving messages

HI Vinicius, This question seems like a better fit for the user mailing list rather than the developer mailing list. Anyway, if I understand correctly, you are asking if the producer can choose to assign partitions in a round-robin fashion rather than based on the key. The answer is, you can,

Re: Broker side round robin on topic partitions when receiving messages

Anyone? On Fri, Jun 5, 2020 at 2:42 PM Vinicius Scheidegger < vinicius.scheideg...@gmail.com> wrote: > Does anyone know how could I perform a load balance to distribute equally > the messages to all consumers within the same consumer group having > multiple producers? > > Is this a conceptual fla

Re: Broker side round robin on topic partitions when receiving messages

Does anyone know how could I perform a load balance to distribute equally the messages to all consumers within the same consumer group having multiple producers? Is this a conceptual flaw on Kafka, wasn't it thought for equal distribution with multiple producers or am I missing something? I've ask

Broker side round robin on topic partitions when receiving messages

Hi, I'm trying to understand a little bit more about how Kafka works. I have a design with multiple producers writing to a single topic and multiple consumers in a single Consumer Group consuming message from this topic. My idea is to distribute the messages from all producers equally. From readi