Re: Uneven distribution of messages in topic's partitions

2020-06-20 Thread Ricardo Ferreira
Nag, Technically the `DefaultPartitioner` uses Mumur2 as you can see in the implementation code from Kafka's trunk: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java The `RoundRobinPartitioner` should be used i

Re: Uneven distribution of messages in topic's partitions

2020-06-19 Thread Nag Y
Hi Ricardo , Just follow up question to add , I believe the defaultpartioner uses mumur3 as default . Should RoundRobinPartitioner class be used to have an equal distribution to maximum extent.instead of default partitioner ? Is StickyPartitioner (mentioned above) is different from Roun

Re: Uneven distribution of messages in topic's partitions

2020-06-19 Thread Ricardo Ferreira
Hi Hemant, Being able to lookup specific records by key is not possible in Kafka. As a distributed streaming platform based on the concept of a commit log Kafka organizes data sequentially where each record has an offset that uniquely identifies not who the record is but where within the log i

Re: Uneven distribution of messages in topic's partitions

2020-06-19 Thread Hemant Bairwa
Thanks Ricardo. I need some information on more use case. In my application I need to use Kafka to maintain the different workflow states of message items while processing through different processes. For example in my application all messages transits from Process A to Process Z and I need to mai

Re: Uneven distribution of messages in topic's partitions

2020-06-18 Thread Ricardo Ferreira
Hemant, This behavior might be the result of the version of AK (Apache Kafka) that you are using. Before AK 2.4 the default behavior for the DefaultPartitioner was to load balance data production across the partitions as you described. But it was found that this behavior would cause performan

Uneven distribution of messages in topic's partitions

2020-06-17 Thread Hemant Bairwa
Hello All I have a single producer service which is queuing message into a topic with let say 12 partitions. I want to evenly distribute the messages across all the partitions in a round robin fashion. Even after using default partitioning and keeping key 'NULL', the messages are not getting distr