Again, thank you for the feedback. That link was very helpful! I adjusted my consumer/producer configs to be:
consumer: zookeeper.connect=[server_list_here] group.id=MirrorMaker exclude.internal.topics=true client.id=MirrorMakerConsumer producer: metadata.broker.list=[server_list_here] compression.codec=gzip producer.type=async batch.size=100 message.send.max.retries=2 client.id=MirrorMakerProducer script file: /home/chris.neal/kafka_2.10-0.8.1.1/bin/kafka-run-class.sh kafka.tools.MirrorMaker \ --producer.config /home/chris.neal/mirror_maker/producer.properties \ --consumer.config /home/chris.neal/mirror_maker/consumer.properties \ --num.streams=18 \ --num.producers=18 \ --whitelist '[topic_list_here]' The num.streams matches the *total* of all partitions for the topics I am consuming. The topics on both the source/target cluster have the same partitions as well. I did have to adjust one to make this true. But...The messages still all go to partition 1 for all the topics. :( The only time I have been able to get it to round-robin is when I pick just one topic. That really makes me think it is still a configuration issue that is causing my problems (probably num.streams). Thanks again for working through this with me. VERY much appreciate your help. Chris On Fri, Mar 24, 2017 at 10:48 AM, Manikumar <manikumar.re...@gmail.com> wrote: > producer distributes the non-keyed messages to available target partitions > in a > round-robin fashion. > > You don't need to set num.consumer.fetchers, partition.assignment.strategy > props. > Use the --num.streams option to specify the number of consumer threads to > create. > > https://community.hortonworks.com/articles/79891/kafka- > mirror-maker-best-practices.html > > On Fri, Mar 24, 2017 at 8:49 PM, Chris Neal <cwn...@gmail.com> wrote: > > > Thanks very much for the reply Manikumar! > > > > I found that there were a few topics on the source cluster that had more > > than two partitions, but all topics on the target cluster had 2 > > partitions. I did a test between one topic that had 2 on each, and I did > > get messages to both partitions as expected. > > > > Is there a way to get this same behavior if the number of partitions on > the > > source and target topic are different? > > > > What should I set the num.consumer.fetchers property to? I believe I > > remember reading that it needed to be at least equal to the number of > > partitions on the topic. > > > > For example, if I'm reading from: > > > > topicA: 2 partitions > > topicB: 3 partitions > > topicC: 1 partition > > > > should i set num.consumer.fetchers to 3, or 6? :) > > > > Again, thank you!! > > > > > > On Fri, Mar 24, 2017 at 12:57 AM, Manikumar <manikumar.re...@gmail.com> > > wrote: > > > > > Are you sure target cluster topics have more than one partition? > > > If you are sending keyed messages, they may be going to the same > > partition. > > > > > > On Thu, Mar 23, 2017 at 11:15 PM, Chris Neal <cwn...@gmail.com> wrote: > > > > > > > Hi everyone, > > > > > > > > I am using MirrorMaker to consume from a 0.8.2.2 cluster and produce > > to a > > > > 0.10.2 cluster. All the topics have two partitions on both clusters. > > My > > > > consumer.properties is: > > > > > > > > zookeeper.connect=[string of servers] > > > > group.id=MirrorMaker > > > > num.consumer.fetchers=2 > > > > partition.assignment.strategy=roundrobin > > > > > > > > My producer.properties is: > > > > > > > > metadata.broker.list=[string of servers] > > > > compression.codec=gzip > > > > producer.type=async > > > > message.send.max.retries=2 > > > > > > > > My understanding from the documentation is that this *should* work, > but > > > it > > > > is not. All my data is going into partition 1 on the target cluster > > for > > > > all topics. > > > > > > > > Can someone help me understand what I'm missing here? > > > > > > > > Thank you for your time! > > > > Chris > > > > > > > > > >
