Hi all, I've been struggling with this for awhile and need some help please. :)
I have a source cluster running Kafka 0.8.1.1 with about 900 topics. I have a target cluster running Kafka 0.10.2 with those same topics pre-created with the same number of partitions per topic as the source cluster. I'm running Mirror Maker from the 0.8.1.1 Kafka version, and cannot seem to get the produced records to go anywhere besides partition 0 of all these topics. :( I found this page to be helpful WRT performance tuning, but it did not solve my partition producer problem: https://community.hortonworks.com/articles/79891/kafka-mirror-maker-best-practices.html My consumer.config: ====================================== zookeeper.connect=zk1:2181,zk2:2181,zk3:2181 group.id=prod_perf_mirrormaker2_new1 num.consumer.fetchers=1100 client.id=MirrorMakerConsumer_perf ====================================== My producer config: ====================================== metadata.broker.list=kafka1:9092,kafka2:9092,kafka3:9092 compression.codec=gzip producer.type=async batch.num.messages=1000 message.send.max.retries=2 client.id=MirrorMakerProducer_perf ====================================== My start script: ====================================== #!/bin/sh export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G" /home/chris.neal/kafka_2.9.2-0.8.1.1/bin/kafka-run-class.sh kafka.tools.MirrorMaker \ --producer.config /home/chris.neal/mirror_maker/prod_perf/producer.properties \ --consumer.config /home/chris.neal/mirror_maker/prod_perf/consumer.properties \ --num.streams=10 \ --whitelist '<snip long list of about 60 topics>' ====================================== I do know that there is some combination of "num.streams" and "num.consumer.fetchers" that should get it working, but I can't figure out what that combination is. All topics have the same setup of 10 partitions per topic. Can anyone shed some light on how to make this go across all 10 partitions, and not just partition 0? Thank you SO much for your time and help. Chris
