Ctrl+c is clean shutdown. kill -9 is not On Mon, Mar 9, 2015 at 2:32 AM, Alex Melville <amelvi...@g.hmc.edu> wrote:
> What does a "clean shutdown" of the MM entail? So far I've just been using > Ctrl + C to send an interrupt to kill it. > > > Alex > > On Sat, Mar 7, 2015 at 10:59 PM, Jiangjie Qin <j...@linkedin.com.invalid> > wrote: > > > If auto.offset.reset is set to smallest, it does not mean the consumer > > will always consume from the smallest. It means that if no previous > offset > > commit is found for this consumer group, then it will consume from the > > smallest. So for mirror maker, you probably want to always use the same > > consumer group id. This could be configured in the consumer config file > > you pass into mirror maker. > > Another thing about duplicate messages is that if mirror maker is > shutdown > > cleanly, next time when you start it again with same consumer group id, > > there should be no duplicates. But if mirror maker shutdown > uncleanly(e.g. > > By a kill -9), then next time it starts up you might still have duplicate > > messages after the last committed offsets. > > > > Jiangjie (Becket) Qin > > > > On 3/7/15, 11:45 PM, "sunil kalva" <sambarc...@gmail.com> wrote: > > > > >Qin > > >Partition problem is solved by passing "--new.producer true" option in > > >command line, but adding auto.offset.rese=smallest config, every time i > > >restart the Mirror tool it copies from starting ends up having lot of > > >duplicate messages in destination cluster. > > >Could you please tell me how do i configure to make sure that > destination > > >cluster is always insync with source cluster. > > > > > >SunilKalva > > > > > >On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin <j...@linkedin.com.invalid > > > > >wrote: > > > > > >> For data not showing up, you need to make sure mirror maker consumer > > >> auto.offset.reset is set to smallest, otherwise when you run mirror > > >>maker > > >> for the first time, all the pre-existing messages won¹t be consumed. > > >> For partition sticking, can you verify if your messages are keyed > > >>messages > > >> or not? If they are not keyed messages, can you check if you are using > > >>old > > >> producer or new producer? For old producer, the default behavior is > > >> sticking to one partition for 10 min and then move to the next > > >>partition. > > >> So if you wait for more than 10 min, you should see messages in two > > >> different partitions. > > >> > > >> Jiangjie (Becket) Qin > > >> > > >> On 3/7/15, 8:28 AM, "sunil kalva" <sambarc...@gmail.com> wrote: > > >> > > >> >And i also observed ,all the data is moving to one partition in > > >> >destination > > >> >cluster though i have multiple partitions for that topic in source > and > > >> >destination clusters. > > >> > > > >> >SunilKalva > > >> > > > >> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva <sambarc...@gmail.com> > > >>wrote: > > >> > > > >> >> I ran kafka mirroring tool after producing data in source cluster, > > >>and > > >> >> this is not copied to destination cluster. If i produce data after > > >> >>running > > >> >> tool those data are copied to destination cluster. Am i missing > > >> >>something ? > > >> >> > > >> >> -- > > >> >> SunilKalva > > >> >> > > >> > > > >> > > > >> > > > >> >-- > > >> >SunilKalva > > >> > > >> > > > > > > > > >-- > > >SunilKalva > > > > > -- Regards, Tao
