Tao,

I updated the mirrorconsumer.properties config file as you suggested, and
upped the MM's log level to DEBUG. I have the output of the DEBUG logger
here in this pastebin, if you could take a minute to look for anything in
its contents that would indicate a problem that would be extremely helpful.
Note that my servers hostnames are of the form ad-010X or ba-0X where X is
some integer between 1 and 4.

http://pastebin.com/rBsxx15A

When I run the mirrormaker and then spin up a console consumer to read from
the source cluster, I get 0 messages consumed.


Alex

On Sun, Feb 15, 2015 at 3:00 AM, tao xiao <xiaotao...@gmail.com> wrote:

> Alex,
>
> Are you sure you have data continually being sent to the topic in source
> cluster after you bring up MM? By default auto.offset.reset=largest in MM
> consumer config which means MM only fetches the largest offset if the
> consumer group has no initial offset in zookeeper.
>
> You can have MM print more log by changing the log level in
> config/tools-log4j.properties
>
> On Sun, Feb 15, 2015 at 8:39 AM, Alex Melville <amelvi...@g.hmc.edu>
> wrote:
>
> > Hi Kafka'ers,
> >
> >
> > I am trying to get the Mirrormaker working with two separate clusters,
> one
> > as the source and the other as the target. The topic I'm trying to copy
> > over exists on both the source and target clusters. Here are the relevant
> > entries in my consumer and producer properties files, which I'm
> specifying
> > the command I run to start the MM:
> >
> > *mirrorconsumer.properties:*
> > zookeeper.connect=ad-0104:2181
> > zookeeper.connection.timeout.ms=6000
> > group.id=test-consumer-group
> >
> >
> > *mirrorproducer.properties:*
> > metadata.broker.list=ba-02:9092,ba-03:9092
> > producer.type=sync
> > compression.codec=none
> > serializer.class=kafka.serializer.DefaultEncoder
> >
> >
> > Then I run the following command:
> > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> >  ../config/mirrorconsumer.properties --producer.config
> > ../config/mirrorproducer.properties --whitelist consolemm
> >
> > so consolemm is the topic I'm trying to copy over. I've created consolemm
> > and have used to console-consumer to verify that there are messages in
> the
> > topic.
> >
> > When I run this command... nothing happens. The process keeps running and
> > prints nothing to the Terminal. If I look in the output of the zookeeper
> on
> > the source cluster I get only the following:
> >
> > [2015-02-15 00:34:06,102] INFO Accepted socket connection from /
> > 10.7.162.75:42819 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> > [2015-02-15 00:34:06,104] INFO Client attempting to establish new session
> > at /10.7.162.75:42819 (org.apache.zookeeper.server.ZooKeeperServer)
> > [2015-02-15 00:34:06,106] INFO Established session 0x14b668b0fbe0033 with
> > negotiated timeout 6000 for client /10.7.162.75:42819
> > (org.apache.zookeeper.server.ZooKeeperServer)
> >
> >
> > and when I look at the output of one of the brokers on the source
> cluster I
> > get:
> >
> > [2015-02-15 00:32:14,382] INFO Closing socket connection to /10.7.162.75
> .
> > (kafka.network.Processor)
> >
> > and there is no output on the zookeeper on the target cluster.
> >
> >
> >
> > Any advice on what is causing MM to not properly copy over data to the
> > target cluster would be extremely helpful.
> >
> > -Alex
> >
>
>
>
> --
> Regards,
> Tao
>

Reply via email to