lol...well, I take it all back. Now I can't get it to work at all :( Here's what I have:
*consumer.properties* zookeeper.connect=[server_list] # I was changing the group.id each time in case that was causing some issues group.id=MirrorMakerTest7 client.id=MirrorMakerConsumer *producer.properties* metadata.broker.list=[server_list] compression.codec=gzip producer.type=async batch.num.messages=100 message.send.max.retries=2 client.id=MirrorMakerProducer *script:* #!/bin/sh export KAFKA_HEAP_OPTS="-Xmx32M -Xms32M" /home/chris.neal/kafka_2.10-0.8.1.1/bin/kafka-run-class.sh kafka.tools.MirrorMaker \ --producer.config /home/chris.neal/mirror_maker/test3/producer.properties \ --consumer.config /home/chris.neal/mirror_maker/test3/consumer.properties \ --num.streams=2 \ --num.producers=2 \ --whitelist 'mytopic' The topic definition is as follows: *Source cluster:* Topic: mytopic PartitionCount:2 ReplicationFactor:2 Configs: Topic: mytopic Partition: 0 Leader: 0 Replicas: 0,1 Isr: 0,1 Topic: mytopic Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1 *Target cluster:* Topic: mytopic PartitionCount:2 ReplicationFactor:2 Configs: Topic: mytopic Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic: mytopic Partition: 1 Leader: 2 Replicas: 2,3 Isr: 2,3 The target has one more replica (3 servers in cluster vs. 2), but that's the only difference. I've spent seriously all day trying various configs, and I can't get this simple thing working... Sorry to keep bugging you for something that should be quite simple. :( Chris On Fri, Mar 24, 2017 at 12:37 PM, Chris Neal <cwn...@gmail.com> wrote: > Again, thank you for the feedback. That link was very helpful! > > I adjusted my consumer/producer configs to be: > > consumer: > zookeeper.connect=[server_list_here] > group.id=MirrorMaker > exclude.internal.topics=true > client.id=MirrorMakerConsumer > > producer: > metadata.broker.list=[server_list_here] > compression.codec=gzip > producer.type=async > batch.size=100 > message.send.max.retries=2 > client.id=MirrorMakerProducer > > script file: > /home/chris.neal/kafka_2.10-0.8.1.1/bin/kafka-run-class.sh > kafka.tools.MirrorMaker \ > --producer.config /home/chris.neal/mirror_maker/producer.properties \ > --consumer.config /home/chris.neal/mirror_maker/consumer.properties \ > --num.streams=18 \ > --num.producers=18 \ > --whitelist '[topic_list_here]' > > The num.streams matches the *total* of all partitions for the topics I am > consuming. The topics on both the source/target cluster have the same > partitions as well. I did have to adjust one to make this true. > > But...The messages still all go to partition 1 for all the topics. :( > > The only time I have been able to get it to round-robin is when I pick > just one topic. That really makes me think it is still a configuration > issue that is causing my problems (probably num.streams). > > Thanks again for working through this with me. VERY much appreciate your > help. > Chris > > > On Fri, Mar 24, 2017 at 10:48 AM, Manikumar <manikumar.re...@gmail.com> > wrote: > >> producer distributes the non-keyed messages to available target partitions >> in a >> round-robin fashion. >> >> You don't need to set num.consumer.fetchers, partition.assignment.strategy >> props. >> Use the --num.streams option to specify the number of consumer threads to >> create. >> >> https://community.hortonworks.com/articles/79891/kafka-mirro >> r-maker-best-practices.html >> >> On Fri, Mar 24, 2017 at 8:49 PM, Chris Neal <cwn...@gmail.com> wrote: >> >> > Thanks very much for the reply Manikumar! >> > >> > I found that there were a few topics on the source cluster that had more >> > than two partitions, but all topics on the target cluster had 2 >> > partitions. I did a test between one topic that had 2 on each, and I >> did >> > get messages to both partitions as expected. >> > >> > Is there a way to get this same behavior if the number of partitions on >> the >> > source and target topic are different? >> > >> > What should I set the num.consumer.fetchers property to? I believe I >> > remember reading that it needed to be at least equal to the number of >> > partitions on the topic. >> > >> > For example, if I'm reading from: >> > >> > topicA: 2 partitions >> > topicB: 3 partitions >> > topicC: 1 partition >> > >> > should i set num.consumer.fetchers to 3, or 6? :) >> > >> > Again, thank you!! >> > >> > >> > On Fri, Mar 24, 2017 at 12:57 AM, Manikumar <manikumar.re...@gmail.com> >> > wrote: >> > >> > > Are you sure target cluster topics have more than one partition? >> > > If you are sending keyed messages, they may be going to the same >> > partition. >> > > >> > > On Thu, Mar 23, 2017 at 11:15 PM, Chris Neal <cwn...@gmail.com> >> wrote: >> > > >> > > > Hi everyone, >> > > > >> > > > I am using MirrorMaker to consume from a 0.8.2.2 cluster and produce >> > to a >> > > > 0.10.2 cluster. All the topics have two partitions on both >> clusters. >> > My >> > > > consumer.properties is: >> > > > >> > > > zookeeper.connect=[string of servers] >> > > > group.id=MirrorMaker >> > > > num.consumer.fetchers=2 >> > > > partition.assignment.strategy=roundrobin >> > > > >> > > > My producer.properties is: >> > > > >> > > > metadata.broker.list=[string of servers] >> > > > compression.codec=gzip >> > > > producer.type=async >> > > > message.send.max.retries=2 >> > > > >> > > > My understanding from the documentation is that this *should* work, >> but >> > > it >> > > > is not. All my data is going into partition 1 on the target cluster >> > for >> > > > all topics. >> > > > >> > > > Can someone help me understand what I'm missing here? >> > > > >> > > > Thank you for your time! >> > > > Chris >> > > > >> > > >> > >> > >
