Hi Sunilkalva, We are rewriting mirror maker in KAFKA-1997 with a handful of enhancement. With that new mirror maker, you will be able to mirror to a different topic by using the message handler.
Jiangjie (Becket) Qin On 3/9/15, 4:41 AM, "sunil kalva" <sambarc...@gmail.com> wrote: >I think it will be very usefull if we can mirror to a different topic name >on destination side. >We have a use case to merge data from multiple colos to one central colo. > >SunilKalva > >On Mon, Mar 9, 2015 at 4:29 PM, tao xiao <xiaotao...@gmail.com> wrote: > >> I don't think you can mirror messages to a different topic name in the >> current mirror maker implementation. Mirror maker sends the message to >> destination topic based on the topic name it reads from source >> >> On Mon, Mar 9, 2015 at 5:00 PM, sunil kalva <sambarc...@gmail.com> >>wrote: >> >> > Can i configure different topic name in destination cluster, i mean >>can i >> > have different topic names for source and destination cluster for >> > mirroring. If yes how can i map source topic with destination topic >>name >> ? >> > >> > SunilKalva >> > >> > On Mon, Mar 9, 2015 at 6:41 AM, tao xiao <xiaotao...@gmail.com> wrote: >> > >> > > Ctrl+c is clean shutdown. kill -9 is not >> > > >> > > On Mon, Mar 9, 2015 at 2:32 AM, Alex Melville <amelvi...@g.hmc.edu> >> > wrote: >> > > >> > > > What does a "clean shutdown" of the MM entail? So far I've just >>been >> > > using >> > > > Ctrl + C to send an interrupt to kill it. >> > > > >> > > > >> > > > Alex >> > > > >> > > > On Sat, Mar 7, 2015 at 10:59 PM, Jiangjie Qin >> > <j...@linkedin.com.invalid >> > > > >> > > > wrote: >> > > > >> > > > > If auto.offset.reset is set to smallest, it does not mean the >> > consumer >> > > > > will always consume from the smallest. It means that if no >>previous >> > > > offset >> > > > > commit is found for this consumer group, then it will consume >>from >> > the >> > > > > smallest. So for mirror maker, you probably want to always use >>the >> > same >> > > > > consumer group id. This could be configured in the consumer >>config >> > file >> > > > > you pass into mirror maker. >> > > > > Another thing about duplicate messages is that if mirror maker >>is >> > > > shutdown >> > > > > cleanly, next time when you start it again with same consumer >>group >> > id, >> > > > > there should be no duplicates. But if mirror maker shutdown >> > > > uncleanly(e.g. >> > > > > By a kill -9), then next time it starts up you might still have >> > > duplicate >> > > > > messages after the last committed offsets. >> > > > > >> > > > > Jiangjie (Becket) Qin >> > > > > >> > > > > On 3/7/15, 11:45 PM, "sunil kalva" <sambarc...@gmail.com> wrote: >> > > > > >> > > > > >Qin >> > > > > >Partition problem is solved by passing "--new.producer true" >> option >> > in >> > > > > >command line, but adding auto.offset.rese=smallest config, >>every >> > > time i >> > > > > >restart the Mirror tool it copies from starting ends up having >>lot >> > of >> > > > > >duplicate messages in destination cluster. >> > > > > >Could you please tell me how do i configure to make sure that >> > > > destination >> > > > > >cluster is always insync with source cluster. >> > > > > > >> > > > > >SunilKalva >> > > > > > >> > > > > >On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin >> > > <j...@linkedin.com.invalid >> > > > > >> > > > > >wrote: >> > > > > > >> > > > > >> For data not showing up, you need to make sure mirror maker >> > consumer >> > > > > >> auto.offset.reset is set to smallest, otherwise when you run >> > mirror >> > > > > >>maker >> > > > > >> for the first time, all the pre-existing messages won¹t be >> > consumed. >> > > > > >> For partition sticking, can you verify if your messages are >> keyed >> > > > > >>messages >> > > > > >> or not? If they are not keyed messages, can you check if you >>are >> > > using >> > > > > >>old >> > > > > >> producer or new producer? For old producer, the default >>behavior >> > is >> > > > > >> sticking to one partition for 10 min and then move to the >>next >> > > > > >>partition. >> > > > > >> So if you wait for more than 10 min, you should see messages >>in >> > two >> > > > > >> different partitions. >> > > > > >> >> > > > > >> Jiangjie (Becket) Qin >> > > > > >> >> > > > > >> On 3/7/15, 8:28 AM, "sunil kalva" <sambarc...@gmail.com> >>wrote: >> > > > > >> >> > > > > >> >And i also observed ,all the data is moving to one >>partition in >> > > > > >> >destination >> > > > > >> >cluster though i have multiple partitions for that topic in >> > source >> > > > and >> > > > > >> >destination clusters. >> > > > > >> > >> > > > > >> >SunilKalva >> > > > > >> > >> > > > > >> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva < >> > sambarc...@gmail.com> >> > > > > >>wrote: >> > > > > >> > >> > > > > >> >> I ran kafka mirroring tool after producing data in source >> > > cluster, >> > > > > >>and >> > > > > >> >> this is not copied to destination cluster. If i produce >>data >> > > after >> > > > > >> >>running >> > > > > >> >> tool those data are copied to destination cluster. Am i >> missing >> > > > > >> >>something ? >> > > > > >> >> >> > > > > >> >> -- >> > > > > >> >> SunilKalva >> > > > > >> >> >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > >> >-- >> > > > > >> >SunilKalva >> > > > > >> >> > > > > >> >> > > > > > >> > > > > > >> > > > > >-- >> > > > > >SunilKalva >> > > > > >> > > > > >> > > > >> > > >> > > >> > > >> > > -- >> > > Regards, >> > > Tao >> > > >> > >> > >> > >> > -- >> > SunilKalva >> > >> >> >> >> -- >> Regards, >> Tao >> > > > >-- >SunilKalva
