Hi Churu,
The documentations needs to be updated.
Basically, you need to specify the producer and consumer config files as
part of the parameters:
bin/kafka-run-class.sh kafka.tools.MirrorMaker
Missing required argument "[consumer.config]"
Option
Description
------
-----------
--blacklist <Java regex (String)> Blacklist of topics to
mirror.
*--consumer.config <config file> Consumer config to consume from
a *
source cluster. You may
specify
multiple of
these.
--help Print this
message.
--num.producers <Integer: Number of Number of producer instances
(default:
producers>
1)
--num.streams <Integer: Number of Number of consumption
streams.
threads> (default:
1)
*--producer.config <config file> Embedded producer
config. *--queue.size <Integer: Queue size in Number of
messages that are buffered
terms of number of messages> between the consumer and
producer
(default:
10000)
--whitelist <Java regex (String)> Whitelist of topics to
mirror.
The configs of producer/consumer can be found here:
http://kafka.apache.org/documentation.html#configuration
For most of them you can just go with the defaults, but for some of them
such as the metadata.broker.list and zookeeper.connect string you are
required to specify the value.
Guozhang
On Mon, Mar 24, 2014 at 2:29 PM, Churu Tang <ct...@rubiconproject.com>wrote:
> Hi,
>
> 1. In documentation 0.8.1, the mirror maker command is:
> bin/kafka-run-class.sh kafka.tools.MirrorMaker --zookeeper
> zk_host:port/chroot
> --consumer.config consumer-1.properties --consumer.config
> consumer-2.properties
> --producer.config producer.properties --whitelist my-topic
>
> However, the zookeeper option seems not allowed.
> [kafka_2.10-0.8.1]]$ bin/kafka-run-class.sh kafka.tools.MirrorMaker
> --zookeeper localhost:2181 --consumer.config config/consumer1.properties
> --producer.config config/producer.properties --whitelist="test"
> Exception in thread "main" joptsimple.UnrecognizedOptionException:
> 'zookeeper' is not a recognized option
> at
> joptsimple.OptionException.unrecognizedOption(OptionException.java:93)
> at
> joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:402)
> at
> joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:55)
> at joptsimple.OptionParser.parse(OptionParser.java:392)
> at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:90)
> at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
>
> 2. After I remove the zookeeper option, the command seems still not work.
> I have ensured that the configuration files will direct the consumer to the
> source cluster's ZooKeeper, and the producer to the mirror cluster's
> ZooKeeper. The following is the output for both target and source clusters.
> Is there any other thing that I need to watch out? Thank you very much!
>
> At target side:
> [kafka_2.10-0.8.1]$ bin/kafka-run-class.sh kafka.tools.MirrorMaker
> --consumer.config config/consumer1.properties --producer.config
> config/producer.properties --whitelist="test"
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
>
> At source zookeeper side:
> [2014-03-24 14:12:19,285] INFO Accepted socket connection from /
> 10.96.120.31:60102 (org.apache.zookeeper.server.NIOServerCnxn)
> [2014-03-24 14:12:19,291] INFO Client attempting to establish new session
> at /10.96.120.31:60102 (org.apache.zookeeper.server.NIOServerCnxn)
> [2014-03-24 14:12:19,295] INFO Established session 0x144f5e0cb510003 with
> negotiated timeout 6000 for client
> /10.96.120.31:60102(org.apache.zookeeper.server.NIOServerCnxn)
>
> At source broker side:
> [2014-03-24 14:12:20,386] INFO Closing socket connection to /10.96.120.31.
> (kafka.network.Processor)
>
> Cheers,
> Churu
>
>
>
--
-- Guozhang
