Tao,
I updated the mirrorconsumer.properties config file as you suggested, and upped the MM's log level to DEBUG. I have the output of the DEBUG logger here in this pastebin, if you could take a minute to look for anything in its contents that would indicate a problem that would be extremely helpful. Note that my servers hostnames are of the form ad-010X or ba-0X where X is some integer between 1 and 4. http://pastebin.com/rBsxx15A When I run the mirrormaker and then spin up a console consumer to read from the source cluster, I get 0 messages consumed. Alex On Sun, Feb 15, 2015 at 3:00 AM, tao xiao <xiaotao...@gmail.com> wrote: > Alex, > > Are you sure you have data continually being sent to the topic in source > cluster after you bring up MM? By default auto.offset.reset=largest in MM > consumer config which means MM only fetches the largest offset if the > consumer group has no initial offset in zookeeper. > > You can have MM print more log by changing the log level in > config/tools-log4j.properties > > On Sun, Feb 15, 2015 at 8:39 AM, Alex Melville <amelvi...@g.hmc.edu> > wrote: > > > Hi Kafka'ers, > > > > > > I am trying to get the Mirrormaker working with two separate clusters, > one > > as the source and the other as the target. The topic I'm trying to copy > > over exists on both the source and target clusters. Here are the relevant > > entries in my consumer and producer properties files, which I'm > specifying > > the command I run to start the MM: > > > > *mirrorconsumer.properties:* > > zookeeper.connect=ad-0104:2181 > > zookeeper.connection.timeout.ms=6000 > > group.id=test-consumer-group > > > > > > *mirrorproducer.properties:* > > metadata.broker.list=ba-02:9092,ba-03:9092 > > producer.type=sync > > compression.codec=none > > serializer.class=kafka.serializer.DefaultEncoder > > > > > > Then I run the following command: > > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config > > ../config/mirrorconsumer.properties --producer.config > > ../config/mirrorproducer.properties --whitelist consolemm > > > > so consolemm is the topic I'm trying to copy over. I've created consolemm > > and have used to console-consumer to verify that there are messages in > the > > topic. > > > > When I run this command... nothing happens. The process keeps running and > > prints nothing to the Terminal. If I look in the output of the zookeeper > on > > the source cluster I get only the following: > > > > [2015-02-15 00:34:06,102] INFO Accepted socket connection from / > > 10.7.162.75:42819 (org.apache.zookeeper.server.NIOServerCnxnFactory) > > [2015-02-15 00:34:06,104] INFO Client attempting to establish new session > > at /10.7.162.75:42819 (org.apache.zookeeper.server.ZooKeeperServer) > > [2015-02-15 00:34:06,106] INFO Established session 0x14b668b0fbe0033 with > > negotiated timeout 6000 for client /10.7.162.75:42819 > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > and when I look at the output of one of the brokers on the source > cluster I > > get: > > > > [2015-02-15 00:32:14,382] INFO Closing socket connection to /10.7.162.75 > . > > (kafka.network.Processor) > > > > and there is no output on the zookeeper on the target cluster. > > > > > > > > Any advice on what is causing MM to not properly copy over data to the > > target cluster would be extremely helpful. > > > > -Alex > > > > > > -- > Regards, > Tao >
