Anyone can help me understand this? On Mon, Nov 16, 2015 at 11:21 PM, allen chan <allen.michael.c...@gmail.com> wrote:
> According to documentation, offsets by default are committed every 10 > secs. Shouldnt that be frequent enough that JMX would be accurate? > > autocommit.interval.ms10000is the frequency that the consumed offsets are > committed to zookeeper. > > On Mon, Nov 16, 2015 at 3:31 PM, allen chan <allen.michael.c...@gmail.com> > wrote: > >> So to make JMX accurate, we need to tweak the frequency of commits? What >> setting would that be? >> >> On Mon, Nov 16, 2015 at 8:40 AM, Scott Reynolds <sreyno...@twilio.com> >> wrote: >> >>> On Mon, Nov 16, 2015 at 8:27 AM, Abu-Obeid, Osama < >>> osama.abu-ob...@morganstanley.com> wrote: >>> >>> > I can observe the same thing: >>> > >>> > - Lag values read through the Kafka consumer JMX is 0 >>> > >>> This metric includes *uncommitted* offsets >>> >>> - Lag values read through kafka-run-class.sh >>> > kafka.tools.ConsumerOffsetChecker is on average 200K-400K >>> > >>> * This is just the *committed* offsets >>> >>> >>> > When the Lag value in the Kafka consumer JMX is high (for example 5M), >>> > ConsumerOffsetChecker shows a matching number. >>> > >>> > I am running kafka_2.10-0.8.2.1 >>> > >>> > Osama >>> > >>> > -----Original Message----- >>> > From: allen chan [mailto:allen.michael.c...@gmail.com] >>> > Sent: Saturday, November 14, 2015 8:34 PM >>> > To: users@kafka.apache.org >>> > Subject: Re: consumer offset tool and JMX metrics do not match >>> > >>> > I believe producers / brokers / and consumers has been restarted at >>> > different times. >>> > What do you think the issue is? >>> > >>> > On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj < >>> prabhbha...@gmail.com> >>> > wrote: >>> > >>> > > Hi, >>> > > >>> > > Did any of your brokers/consumers undergo a restart between the time >>> > > after you had started the consumption and until you see this issue ? >>> > > >>> > > Thanks, >>> > > Prabhjot >>> > > >>> > > >>> > > >>> > > On Sat, Nov 14, 2015 at 5:53 AM, allen chan >>> > > <allen.michael.c...@gmail.com> >>> > > wrote: >>> > > >>> > > > I also looked at this metric in JMX and it is also 0 >>> > > > >>> > > >>> *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logst >>> > > ash* >>> > > > >>> > > > On Fri, Nov 13, 2015 at 4:06 PM, allen chan < >>> > > allen.michael.c...@gmail.com> >>> > > > wrote: >>> > > > >>> > > > > Hi All, >>> > > > > >>> > > > > I am comparing the output from kafka.tools.ConsumerOffsetChecker >>> > > > > vs JMX >>> > > > > >>> > > > >>> > > >>> (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstas >>> > > h,topic=logstash_fdm,partition=*) >>> > > > > and they do not match. >>> > > > > >>> > > > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX >>> > > > > shows 0 >>> > > > for >>> > > > > all partitions. >>> > > > > >>> > > > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to >>> JMX. >>> > > > > >>> > > > > Has anyone seen this issue before? I am trying to use JMX to pull >>> > > > > stats into monitoring system >>> > > > > >>> > > > > >>> > > > > -- >>> > > > > Allen Michael Chan >>> > > > > >>> > > > >>> > > > >>> > > > >>> > > > -- >>> > > > Allen Michael Chan >>> > > > >>> > > >>> > > >>> > > >>> > > -- >>> > > --------------------------------------------------------- >>> > > "There are only 10 types of people in the world: Those who understand >>> > > binary, and those who don't" >>> > > >>> > >>> > >>> > >>> > -- >>> > Allen Michael Chan >>> > >>> > >>> > >>> > >>> -------------------------------------------------------------------------------- >>> > >>> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the >>> > opinions or views contained herein are not intended to be, and do not >>> > constitute, advice within the meaning of Section 975 of the Dodd-Frank >>> Wall >>> > Street Reform and Consumer Protection Act. If you have received this >>> > communication in error, please destroy all electronic and paper >>> copies; do >>> > not disclose, use or act upon the information; and notify the sender >>> > immediately. Mistransmission is not intended to waive confidentiality >>> or >>> > privilege. Morgan Stanley reserves the right, to the extent permitted >>> under >>> > applicable law, to monitor electronic communications. This message is >>> > subject to terms available at the following link: >>> > http://www.morganstanley.com/disclaimers. If you cannot access these >>> > links, please notify us by reply message and we will send the contents >>> to >>> > you. By messaging with Morgan Stanley you consent to the foregoing. >>> >> >> >> >> -- >> Allen Michael Chan >> > > > > -- > Allen Michael Chan > -- Allen Michael Chan
