Again, thank you for the feedback.  That link was very helpful!

I adjusted my consumer/producer configs to be:

consumer:
zookeeper.connect=[server_list_here]
group.id=MirrorMaker
exclude.internal.topics=true
client.id=MirrorMakerConsumer

producer:
metadata.broker.list=[server_list_here]
compression.codec=gzip
producer.type=async
batch.size=100
message.send.max.retries=2
client.id=MirrorMakerProducer

script file:
/home/chris.neal/kafka_2.10-0.8.1.1/bin/kafka-run-class.sh
kafka.tools.MirrorMaker \
--producer.config /home/chris.neal/mirror_maker/producer.properties \
--consumer.config /home/chris.neal/mirror_maker/consumer.properties \
--num.streams=18 \
--num.producers=18 \
--whitelist '[topic_list_here]'

The num.streams matches the *total* of all partitions for the topics I am
consuming.  The topics on both the source/target cluster have the same
partitions as well.  I did have to adjust one to make this true.

But...The messages still all go to partition 1 for all the topics. :(

The only time I have been able to get it to round-robin is when I pick just
one topic.  That really makes me think it is still a configuration issue
that is causing my problems (probably num.streams).

Thanks again for working through this with me.  VERY much appreciate your
help.
Chris


On Fri, Mar 24, 2017 at 10:48 AM, Manikumar <manikumar.re...@gmail.com>
wrote:

> producer distributes the non-keyed messages to available target partitions
> in a
> round-robin fashion.
>
> You don't need to set num.consumer.fetchers, partition.assignment.strategy
> props.
> Use the --num.streams option to specify the number of consumer threads to
> create.
>
> https://community.hortonworks.com/articles/79891/kafka-
> mirror-maker-best-practices.html
>
> On Fri, Mar 24, 2017 at 8:49 PM, Chris Neal <cwn...@gmail.com> wrote:
>
> > Thanks very much for the reply Manikumar!
> >
> > I found that there were a few topics on the source cluster that had more
> > than two partitions, but all topics on the target cluster had 2
> > partitions.  I did a test between one topic that had 2 on each, and I did
> > get messages to both partitions as expected.
> >
> > Is there a way to get this same behavior if the number of partitions on
> the
> > source and target topic are different?
> >
> > What should I set the num.consumer.fetchers property to?  I believe I
> > remember reading that it needed to be at least equal to the number of
> > partitions on the topic.
> >
> > For example, if I'm reading from:
> >
> > topicA:  2 partitions
> > topicB:  3 partitions
> > topicC:  1 partition
> >
> > should i set  num.consumer.fetchers to 3, or 6? :)
> >
> > Again, thank you!!
> >
> >
> > On Fri, Mar 24, 2017 at 12:57 AM, Manikumar <manikumar.re...@gmail.com>
> > wrote:
> >
> > > Are you sure target cluster topics have more than one partition?
> > > If you are sending keyed messages, they may be going to the same
> > partition.
> > >
> > > On Thu, Mar 23, 2017 at 11:15 PM, Chris Neal <cwn...@gmail.com> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I am using MirrorMaker to consume from a 0.8.2.2 cluster and produce
> > to a
> > > > 0.10.2 cluster.  All the topics have two partitions on both clusters.
> > My
> > > > consumer.properties is:
> > > >
> > > > zookeeper.connect=[string of servers]
> > > > group.id=MirrorMaker
> > > > num.consumer.fetchers=2
> > > > partition.assignment.strategy=roundrobin
> > > >
> > > > My producer.properties is:
> > > >
> > > > metadata.broker.list=[string of servers]
> > > > compression.codec=gzip
> > > > producer.type=async
> > > > message.send.max.retries=2
> > > >
> > > > My understanding from the documentation is that this *should* work,
> but
> > > it
> > > > is not.  All my data is going into partition 1 on the target cluster
> > for
> > > > all topics.
> > > >
> > > > Can someone help me understand what I'm missing here?
> > > >
> > > > Thank you for your time!
> > > > Chris
> > > >
> > >
> >
>

Reply via email to