Can i configure different topic name in destination cluster, i mean can i have different topic names for source and destination cluster for mirroring. If yes how can i map source topic with destination topic name ?
SunilKalva On Mon, Mar 9, 2015 at 6:41 AM, tao xiao <xiaotao...@gmail.com> wrote: > Ctrl+c is clean shutdown. kill -9 is not > > On Mon, Mar 9, 2015 at 2:32 AM, Alex Melville <amelvi...@g.hmc.edu> wrote: > > > What does a "clean shutdown" of the MM entail? So far I've just been > using > > Ctrl + C to send an interrupt to kill it. > > > > > > Alex > > > > On Sat, Mar 7, 2015 at 10:59 PM, Jiangjie Qin <j...@linkedin.com.invalid > > > > wrote: > > > > > If auto.offset.reset is set to smallest, it does not mean the consumer > > > will always consume from the smallest. It means that if no previous > > offset > > > commit is found for this consumer group, then it will consume from the > > > smallest. So for mirror maker, you probably want to always use the same > > > consumer group id. This could be configured in the consumer config file > > > you pass into mirror maker. > > > Another thing about duplicate messages is that if mirror maker is > > shutdown > > > cleanly, next time when you start it again with same consumer group id, > > > there should be no duplicates. But if mirror maker shutdown > > uncleanly(e.g. > > > By a kill -9), then next time it starts up you might still have > duplicate > > > messages after the last committed offsets. > > > > > > Jiangjie (Becket) Qin > > > > > > On 3/7/15, 11:45 PM, "sunil kalva" <sambarc...@gmail.com> wrote: > > > > > > >Qin > > > >Partition problem is solved by passing "--new.producer true" option in > > > >command line, but adding auto.offset.rese=smallest config, every > time i > > > >restart the Mirror tool it copies from starting ends up having lot of > > > >duplicate messages in destination cluster. > > > >Could you please tell me how do i configure to make sure that > > destination > > > >cluster is always insync with source cluster. > > > > > > > >SunilKalva > > > > > > > >On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin > <j...@linkedin.com.invalid > > > > > > >wrote: > > > > > > > >> For data not showing up, you need to make sure mirror maker consumer > > > >> auto.offset.reset is set to smallest, otherwise when you run mirror > > > >>maker > > > >> for the first time, all the pre-existing messages won¹t be consumed. > > > >> For partition sticking, can you verify if your messages are keyed > > > >>messages > > > >> or not? If they are not keyed messages, can you check if you are > using > > > >>old > > > >> producer or new producer? For old producer, the default behavior is > > > >> sticking to one partition for 10 min and then move to the next > > > >>partition. > > > >> So if you wait for more than 10 min, you should see messages in two > > > >> different partitions. > > > >> > > > >> Jiangjie (Becket) Qin > > > >> > > > >> On 3/7/15, 8:28 AM, "sunil kalva" <sambarc...@gmail.com> wrote: > > > >> > > > >> >And i also observed ,all the data is moving to one partition in > > > >> >destination > > > >> >cluster though i have multiple partitions for that topic in source > > and > > > >> >destination clusters. > > > >> > > > > >> >SunilKalva > > > >> > > > > >> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva <sambarc...@gmail.com> > > > >>wrote: > > > >> > > > > >> >> I ran kafka mirroring tool after producing data in source > cluster, > > > >>and > > > >> >> this is not copied to destination cluster. If i produce data > after > > > >> >>running > > > >> >> tool those data are copied to destination cluster. Am i missing > > > >> >>something ? > > > >> >> > > > >> >> -- > > > >> >> SunilKalva > > > >> >> > > > >> > > > > >> > > > > >> > > > > >> >-- > > > >> >SunilKalva > > > >> > > > >> > > > > > > > > > > > >-- > > > >SunilKalva > > > > > > > > > > > > -- > Regards, > Tao > -- SunilKalva
