Hi Xuyen, SPM for Kafka gets all this stuff already. I didn't look into whether/how exactly one can spot consumer lag, but if you spot a way to do it, please share.
There is a demo of SPM for Kafka if you go to https://apps.sematext.com/demoso you can see all Kafka performance graphs and see if any of the metrics there show consumer lag directly or indirectly. Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Mon, Jan 27, 2014 at 12:49 PM, Xuyen On <x...@ancestry.com> wrote: > Oh, I should note that the built in consumer offsetchecker tool works for > me but I was hoping to use something like jmxtrans so that I could easily > export the data to ganglia, graphite or some other graphing tool. Jmxtrans > was recommended from the Kafka wiki and there's another project called > kafka-ganglia but it doesn't support 0.7.2 Kafka which is what we're using > right now. Any suggestions to get real time reporting of consumer lag > through a tool that can export to ganglia would be appreciated. > > Thanks, > > Xuyen > > -----Original Message----- > From: Xuyen On [mailto:x...@ancestry.com] > Sent: Monday, January 27, 2014 9:33 AM > To: users@kafka.apache.org > Subject: RE: Calcuate Consumer lag from JMX beans in Kafka 0.7.2 > > Thanks for your response Joel. > > I am currently trying out JMXTrans to get the stats from MBean and I can > read attributes fine but it doesn't support JMX Operations yet. What tool > do you use for your reporting? Is there another tool that supports JMX > operations so that I can use the getOffsetLag operation? > > Thanks, > > Xuyen > > -----Original Message----- > From: Joel Koshy [mailto:jjkosh...@gmail.com] > Sent: Friday, January 24, 2014 5:48 PM > To: users@kafka.apache.org > Subject: Re: Calcuate Consumer lag from JMX beans in Kafka 0.7.2 > > > kafka.logs.eventdata-0 > > Attributes > > Name // Name of partition > > Size //Is this the > current number of messages? > Size -> in bytes > > > NumberofSegments // Don't know what this is > Each partition contains multiple segment (files) on disk. > > > CurrentOffset //Is this the current > offset of a consumer? > > No - it is the (byte) offset up to which we have flushed. It is unrelated > to the current consumption point of consumers. > > > NumAppendedMessages // Don't know what this is > Simple counter of number of messages sent so far to this partition. > > > Can someone please tell me what these attributes mean? I am trying to > find out if I can use these values to calculate the consumer lag ie: Size - > CurrentOffset and report it to a chart or dashboard somewhere for real-time > analysis. > > So the above would be insufficient to measure consumer lag. You can use > the consumer offset checker tool and extract the lag from its output. There > is also an mbean operation on the consumers > (getOffsetLag) that you can use on a per-partition basis. > > Thanks, > > Joel > > >
