Re: Consumer Group Lag Reporting

2015-04-07 Thread Kyle Banker
Thanks, Otis. I actually already have a reporting and alerting infrastructure. I mainly wanted to confirm that parsing the output of the offset checker is the recommended practice for reporting consumer group offsets. Is this the case? If so, I wanted to find out if any work is under way to make t

Re: Consumer Group Lag Reporting

2015-04-06 Thread Otis Gospodnetic
Hi, We track our Consumer Lag and all other Kafka metrics with SPM whose agent collects this info from Kafka Consumers. Here's a chart of that: https://apps.sematext.com/spm-reports/s/mcBHvsXwgj -- this public chart doesn't show all filtering option, but you can filter l

Consumer Group Lag Reporting

2015-04-06 Thread Kyle Banker
What is the best practice for reporting the lag on individual consumer groups (e.g., to Graphite)? A recent form post (http://search-hadoop.com/m/4TaT4x9qWm1) seems to indicate that parsing the output of the consumer offset checker tool and reporting that independently is what folks do. Is there a