It took a little while for this to post (aligning with company PR and all), but there's a detailed description of what Burrow is on the LinkedIn Engineering Blog now:
http://engineering.linkedin.com/apache-kafka/burrow-kafka-consumer-monitoring-reinvented -Todd On Thu, Jun 4, 2015 at 6:43 PM, Todd Palino <tpal...@gmail.com> wrote: > I am very happy to introduce Burrow, an application to provide Kafka > consumer status as a service. Burrow is different than just a "lag checker": > > * Multiple Kafka cluster support - Burrow supports any number of Kafka > clusters in a single instance. You can also run multiple copies of Burrow > in parallel and only one of them will send out notifications. > > * All consumers, all partitions - If the consumer is committing offsets to > Kafka (not Zookeeper), it will be available in Burrow automatically. Every > partition it consumes will be monitored simultaneously, avoiding the trap > of just watching the worst partition (MaxLag) or spot checking individual > topics. > > * Status can be checked via HTTP request - There's an internal HTTP server > that provides topic and consumer lists, can give you the latest offsets for > a topic either from the brokers or from the consumer, and lets you check > consumer status. > > * Continuously monitor groups with output via email or a call to an > external HTTP endpoint - Configure emails to send for bad groups, checked > continuously. Or you can have Burrow call an HTTP endpoint into another > system for handling alerts. > > * No thresholds - Status is determined over a sliding window and does not > rely on a fixed limit. When a consumer is checked, it has a status > indicator that tells whether it is OK, a warning, or an error, and the > partitions that caused it to be bad are provided. > > > Burrow was created to address specific problems that LinkedIn has with > monitoring consumers, in particular wildcard consumers like mirror makers > and our audit consumers. Instead of checking offsets for specific consumers > periodically, it monitors the stream of all committed offsets > (__consumer_offsets) and continually calculates lag over a sliding window. > > We welcome all feedback, comments, and contributors. This project is very > much under active development for us (we're using it in some of our > environments now, and working on getting it running everywhere to replace > our previous monitoring system). > > Burrow is written in Go, published under the Apache License, and hosted on > GitHub at: > https://github.com/linkedin/Burrow > > Documentation is on the GitHub wiki at: > https://github.com/linkedin/Burrow/wiki > > -Todd > >
