Meghana, As Ewen said in KAFKA-1650 we removed the support of multiple consumer configs in MM, so that you cannot get one MM instance to pull from multiple origin clusters, but instead you need to use one MM instance for each origin.
The motivation for dropping this support is mainly operational: with the previous settings if one of the original cluster falls into a bad state, we cannot stop / debug that MM since it also pulls from other clusters. And other minor issues include metics reporting, etc. For example at LI people use one MM instance for each origin cluster, that are co-located with the destination cluster, so if you need to mirror from cluster A / B / C to D, you will have a A->D / B->D / C->D MM. Moving forward we are also trying to add exact mirroring features into MM, which would better be a 1-to-1 mapping. Guozhang On Thu, Dec 10, 2015 at 3:21 PM, Ewen Cheslack-Postava <e...@confluent.io> wrote: > Meghana, > > It looks like this functionality was removed in > https://issues.apache.org/jira/browse/KAFKA-1650, although I don't see > explicit discussion of the issue in the JIRA so I'm not sure of the exact > motivation. Maybe Becket or Guozhang can offer some insight about if it is > necessary (that JIRA fixes a data loss issue) or an accidental > incompatibility. > > -Ewen > > On Thu, Dec 10, 2015 at 2:32 PM, Meghana Narasimhan < > mnarasim...@bandwidth.com> wrote: > > > Hi, > > > > We upgraded to the Confluent 2.0 platform today (we were earlier using > the > > Kafka 0.8 without the confluent platform). With the latest Kafka 0.9 that > > is packaged with the platform we are facing issues starting the Mirror > > maker with multiple consumer config files. > > > > The original command that we were using with kafka 0.8 was, > > > > bin/kafka-run-class kafka.tools.MirrorMaker --consumer.config > > mirrorCluster1Consumer.properties --consumer.config > > mirrorCluster2Consumer.properties --consumer.config > > mirrorCluster3Consumer.properties --consumer.config > > mirrorCluster4Consumer.properties --producer.config > > mirrorProducer.properties --whitelist "topic1,topic2" > > > > With the current Kafka 0.9 release, the tool throws the following error > > > > ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$) > > joptsimple.MultipleArgumentsForOptionException: Found multiple arguments > > for option ['consumer.config'], but you asked for only one > > at joptsimple.OptionSet.valueOf(OptionSet.java:177) > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:225) > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > Exception in thread "main" java.lang.NullPointerException > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:276) > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > So I tried setting the --num.streams=4 option in the command, and it > still > > throws the same error, next I tried to use the --new.consumer option but > > the tool fails again with the same error. > > > > Not sure how to get the Mirrormaker tool running with multiple consumer > > config files. > > > > Could not find much documentation regarding this except for the one in > the > > kafka documentation ( > > http://kafka.apache.org/documentation.html#basic_ops_mirror_maker). > > > > ps : I was unable to post this in the confluent mailing list and hence > > posting here. > > > > Thanks, > > Meghana > > > > > > -- > Thanks, > Ewen > -- -- Guozhang
