Hi guys, I am new to Kafka and I am facing a problem I am not able to sort out.
To smooth traffic over all my brokers' partitions, I have coded a custom Paritioner for my producers, using a simple round robin algorithm that jumps from a partition to another on every batch of messages (corresponding to batch.num.messages value). It looks like that : https://gist.github.com/sfalquier/4c0c7f36dd96d642b416 With that fix, every partitions are used equally, but the amount of requests from the producers to the brokers have been multiplied by 2. I do not understand since all producers are async with batch.num.messages=200 and the amount of messages processed is still the same as before. Why do producers need more requests to do the job? As internal traffic is a bit critical on our platform, I would really like to reduce producers' requests volume if possible. Any idea? Any suggestion? Regards, Sébastien
