Ivan Kunz created KAFKA-1497:
--------------------------------
Summary: Change producer load-balancing algorithm in MirrorMaker
Key: KAFKA-1497
URL: https://issues.apache.org/jira/browse/KAFKA-1497
Project: Kafka
Issue Type: Improvement
Components: core
Affects Versions: 0.8.1.1
Reporter: Ivan Kunz
Currently the MirrorMaker uses the following way of spreading the load into
configured producers :
val producerId =
Utils.abs(java.util.Arrays.hashCode(msgAndMetadata.key)) % producers.size()
This way if the producer side of MM uses different than the default
"partitioner.class" messages within the same partition can get re-ordered. Also
hashCode does not produce the same results on different machines (verified by
testing) so cannot be safely used for partitioning between distributed systems
connected via MM (for us message order preservation within a partition is a
critical feature).
It would be great if the code above is changed to utilize the configured
"partitioner.class".
Something along the lines of :
At the initialization:
mmpartitioner = Utils.createObject[Partitioner](config.partitionerClass,
config.props)
During the processing:
val producerId =
mmpartitioner.partition(msgAndMetadata.key,producers.size())
This way the messages consumed and produced by MM can remain in the same order.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
