Thanks Manikumar for the reply :) For errors in the logs, I have a few WARN messages about "Reconnect due to socked error: null", and an occasional "Connection timed out", but that's it.
I definitely can turn on trace logs. Is there something specific you know of that I should look for? WRT keys, all 10 partitions are used on the source cluster, so doesn't that indicate that the keys are different? I don't produce the data, so I'm not 100% sure, but I can try and get ahold of the producer's source code and see how they are being created (if at all). I did validate that the partition counts are the same in both clusters. I was trying to determine if I had the right values set for num.streams and num.consumer.fetchers based on consuming from/producing to 267 topics with 10 partitions each...or if that even matters in this case. Chris On Thu, Nov 2, 2017 at 12:58 AM, Manikumar <manikumar.re...@gmail.com> wrote: > Any exceptions in the mirror maker logs? may be you can enable mirror maker > trace logs. > maybe all messages are having same key? Can you recheck partition count on > target cluster? > > > On Thu, Nov 2, 2017 at 2:45 AM, Chris Neal <cwn...@gmail.com> wrote: > > > Apologies for bumping my own post, but really hoping someone has > experience > > with MirrorMaker who might be able to help with my question below. :) > > > > Thanks! > > Chris > > > > On Fri, Oct 27, 2017 at 10:59 AM, Chris Neal <cwn...@gmail.com> wrote: > > > > > Hi all, > > > > > > I've been struggling with this for awhile and need some help please. :) > > > > > > I have a source cluster running Kafka 0.8.1.1 with about 900 topics. > > > I have a target cluster running Kafka 0.10.2 with those same topics > > > pre-created with the same number of partitions per topic as the source > > > cluster. > > > > > > I'm running Mirror Maker from the 0.8.1.1 Kafka version, and cannot > seem > > > to get the produced records to go anywhere besides partition 0 of all > > these > > > topics. :( > > > > > > I found this page to be helpful WRT performance tuning, but it did not > > > solve my partition producer problem: > > > > > > https://community.hortonworks.com/articles/79891/kafka- > > > mirror-maker-best-practices.html > > > > > > My consumer.config: > > > ====================================== > > > zookeeper.connect=zk1:2181,zk2:2181,zk3:2181 > > > group.id=prod_perf_mirrormaker2_new1 > > > num.consumer.fetchers=1100 > > > client.id=MirrorMakerConsumer_perf > > > ====================================== > > > > > > My producer config: > > > ====================================== > > > metadata.broker.list=kafka1:9092,kafka2:9092,kafka3:9092 > > > compression.codec=gzip > > > producer.type=async > > > batch.num.messages=1000 > > > message.send.max.retries=2 > > > client.id=MirrorMakerProducer_perf > > > ====================================== > > > > > > My start script: > > > ====================================== > > > #!/bin/sh > > > > > > export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G" > > > > > > /home/chris.neal/kafka_2.9.2-0.8.1.1/bin/kafka-run-class.sh > > > kafka.tools.MirrorMaker \ > > > --producer.config /home/chris.neal/mirror_maker/ > > prod_perf/producer.properties > > > \ > > > --consumer.config /home/chris.neal/mirror_maker/ > > prod_perf/consumer.properties > > > \ > > > --num.streams=10 \ > > > --whitelist '<snip long list of about 60 topics>' > > > ====================================== > > > > > > I do know that there is some combination of "num.streams" and > > > "num.consumer.fetchers" that should get it working, but I can't figure > > out > > > what that combination is. > > > > > > All topics have the same setup of 10 partitions per topic. > > > > > > Can anyone shed some light on how to make this go across all 10 > > > partitions, and not just partition 0? > > > > > > Thank you SO much for your time and help. > > > Chris > > > > > >
