Looks like you only have 4 messages in your topic and no more messages got sent
2015-02-19 20:09:34,661] DEBUG initial fetch offset of consolemm:0: fetched offset = 4: consumed offset = 4 is 4 (kafka.consumer.PartitionTopicInfo You can try sending more messages to topic or give the MM a different consumer group id and set auto.offset.reset=smallest On Friday, February 20, 2015, Alex Melville <amelvi...@g.hmc.edu> wrote: > Tao, > > > I updated the mirrorconsumer.properties config file as you suggested, and > upped the MM's log level to DEBUG. I have the output of the DEBUG logger > here in this pastebin, if you could take a minute to look for anything in > its contents that would indicate a problem that would be extremely helpful. > Note that my servers hostnames are of the form ad-010X or ba-0X where X is > some integer between 1 and 4. > > http://pastebin.com/rBsxx15A > > When I run the mirrormaker and then spin up a console consumer to read from > the source cluster, I get 0 messages consumed. > > > Alex > > On Sun, Feb 15, 2015 at 3:00 AM, tao xiao <xiaotao...@gmail.com > <javascript:;>> wrote: > > > Alex, > > > > Are you sure you have data continually being sent to the topic in source > > cluster after you bring up MM? By default auto.offset.reset=largest in MM > > consumer config which means MM only fetches the largest offset if the > > consumer group has no initial offset in zookeeper. > > > > You can have MM print more log by changing the log level in > > config/tools-log4j.properties > > > > On Sun, Feb 15, 2015 at 8:39 AM, Alex Melville <amelvi...@g.hmc.edu > <javascript:;>> > > wrote: > > > > > Hi Kafka'ers, > > > > > > > > > I am trying to get the Mirrormaker working with two separate clusters, > > one > > > as the source and the other as the target. The topic I'm trying to copy > > > over exists on both the source and target clusters. Here are the > relevant > > > entries in my consumer and producer properties files, which I'm > > specifying > > > the command I run to start the MM: > > > > > > *mirrorconsumer.properties:* > > > zookeeper.connect=ad-0104:2181 > > > zookeeper.connection.timeout.ms=6000 > > > group.id=test-consumer-group > > > > > > > > > *mirrorproducer.properties:* > > > metadata.broker.list=ba-02:9092,ba-03:9092 > > > producer.type=sync > > > compression.codec=none > > > serializer.class=kafka.serializer.DefaultEncoder > > > > > > > > > Then I run the following command: > > > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config > > > ../config/mirrorconsumer.properties --producer.config > > > ../config/mirrorproducer.properties --whitelist consolemm > > > > > > so consolemm is the topic I'm trying to copy over. I've created > consolemm > > > and have used to console-consumer to verify that there are messages in > > the > > > topic. > > > > > > When I run this command... nothing happens. The process keeps running > and > > > prints nothing to the Terminal. If I look in the output of the > zookeeper > > on > > > the source cluster I get only the following: > > > > > > [2015-02-15 00:34:06,102] INFO Accepted socket connection from / > > > 10.7.162.75:42819 (org.apache.zookeeper.server.NIOServerCnxnFactory) > > > [2015-02-15 00:34:06,104] INFO Client attempting to establish new > session > > > at /10.7.162.75:42819 (org.apache.zookeeper.server.ZooKeeperServer) > > > [2015-02-15 00:34:06,106] INFO Established session 0x14b668b0fbe0033 > with > > > negotiated timeout 6000 for client /10.7.162.75:42819 > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > > > > and when I look at the output of one of the brokers on the source > > cluster I > > > get: > > > > > > [2015-02-15 00:32:14,382] INFO Closing socket connection to / > 10.7.162.75 > > . > > > (kafka.network.Processor) > > > > > > and there is no output on the zookeeper on the target cluster. > > > > > > > > > > > > Any advice on what is causing MM to not properly copy over data to the > > > target cluster would be extremely helpful. > > > > > > -Alex > > > > > > > > > > > -- > > Regards, > > Tao > > > -- Regards, Tao
