Hello, it has been figured out. The method was not the best way to measure number of messages in a topic. Can't measure topics which have compactaion with kafka.tools.GetOffsetShell script.
BR On Thu, Jan 23, 2020 at 3:03 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > Nils, are those record counts or offsets? > > Ryanne > > On Thu, Jan 23, 2020, 2:46 AM Nils Hermansson < > nils.hermans...@getdreams.com> > wrote: > > > Hello, > > I have setup MM2 and the config is similar to the example config. > > > > > https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties > > Primary has 3 brokers and secondary has 1 broker. > > > > I wrote a script to diff number of messages in topics between primary and > > secondary cluster. > > > > To get number of messages in topic I use. > > bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list > > localhost:9092 --topic mytopic --time -1 --offsets 1 | awk -F ":" '{sum > += > > $3} END {print sum}' > > > > Is there some reason not all messages gets replicated? > > > > A first thought was that it may have been because I wiped the secondary > > cluster and started from beginning. So i switched the name of the > secondary > > cluster and prefix in mm2 config. But I ended up with the same diff. > > > > Result: > > topic: topic1, primary: 236, secondary: 119, diff 117 > > topic: topic2, primary: 2910, secondary: 1223, diff 1687 > > topic: topic3, primary: 2248, secondary: 2248, diff 0 > > topic: topic4, primary: 627, secondary: 91, diff 536 > > topic: topic5, primary: 84, secondary: 77, diff 7 > > topic: topic6, primary: 333, secondary: 156, diff 177 > > topic: topic7, primary: 158, secondary: 45, diff 113 > > topic: topic8, primary: 247, secondary: 48, diff 199 > > ... > > ... > > > > > > BR > > Nils > > >
