producer distributes the non-keyed messages to available target partitions in a round-robin fashion.
You don't need to set num.consumer.fetchers, partition.assignment.strategy props. Use the --num.streams option to specify the number of consumer threads to create. https://community.hortonworks.com/articles/79891/kafka-mirror-maker-best-practices.html On Fri, Mar 24, 2017 at 8:49 PM, Chris Neal <cwn...@gmail.com> wrote: > Thanks very much for the reply Manikumar! > > I found that there were a few topics on the source cluster that had more > than two partitions, but all topics on the target cluster had 2 > partitions. I did a test between one topic that had 2 on each, and I did > get messages to both partitions as expected. > > Is there a way to get this same behavior if the number of partitions on the > source and target topic are different? > > What should I set the num.consumer.fetchers property to? I believe I > remember reading that it needed to be at least equal to the number of > partitions on the topic. > > For example, if I'm reading from: > > topicA: 2 partitions > topicB: 3 partitions > topicC: 1 partition > > should i set num.consumer.fetchers to 3, or 6? :) > > Again, thank you!! > > > On Fri, Mar 24, 2017 at 12:57 AM, Manikumar <manikumar.re...@gmail.com> > wrote: > > > Are you sure target cluster topics have more than one partition? > > If you are sending keyed messages, they may be going to the same > partition. > > > > On Thu, Mar 23, 2017 at 11:15 PM, Chris Neal <cwn...@gmail.com> wrote: > > > > > Hi everyone, > > > > > > I am using MirrorMaker to consume from a 0.8.2.2 cluster and produce > to a > > > 0.10.2 cluster. All the topics have two partitions on both clusters. > My > > > consumer.properties is: > > > > > > zookeeper.connect=[string of servers] > > > group.id=MirrorMaker > > > num.consumer.fetchers=2 > > > partition.assignment.strategy=roundrobin > > > > > > My producer.properties is: > > > > > > metadata.broker.list=[string of servers] > > > compression.codec=gzip > > > producer.type=async > > > message.send.max.retries=2 > > > > > > My understanding from the documentation is that this *should* work, but > > it > > > is not. All my data is going into partition 1 on the target cluster > for > > > all topics. > > > > > > Can someone help me understand what I'm missing here? > > > > > > Thank you for your time! > > > Chris > > > > > >
