Qin Partition problem is solved by passing "--new.producer true" option in command line, but adding auto.offset.rese=smallest config, every time i restart the Mirror tool it copies from starting ends up having lot of duplicate messages in destination cluster. Could you please tell me how do i configure to make sure that destination cluster is always insync with source cluster.
SunilKalva On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin <j...@linkedin.com.invalid> wrote: > For data not showing up, you need to make sure mirror maker consumer > auto.offset.reset is set to smallest, otherwise when you run mirror maker > for the first time, all the pre-existing messages won¹t be consumed. > For partition sticking, can you verify if your messages are keyed messages > or not? If they are not keyed messages, can you check if you are using old > producer or new producer? For old producer, the default behavior is > sticking to one partition for 10 min and then move to the next partition. > So if you wait for more than 10 min, you should see messages in two > different partitions. > > Jiangjie (Becket) Qin > > On 3/7/15, 8:28 AM, "sunil kalva" <sambarc...@gmail.com> wrote: > > >And i also observed ,all the data is moving to one partition in > >destination > >cluster though i have multiple partitions for that topic in source and > >destination clusters. > > > >SunilKalva > > > >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva <sambarc...@gmail.com> wrote: > > > >> I ran kafka mirroring tool after producing data in source cluster, and > >> this is not copied to destination cluster. If i produce data after > >>running > >> tool those data are copied to destination cluster. Am i missing > >>something ? > >> > >> -- > >> SunilKalva > >> > > > > > > > >-- > >SunilKalva > > -- SunilKalva
