Hey Abhimanyu, Not directly answering your questions but in the past we used burrow at my current company and we had a horrible time with it. It would crash daily and its lag metrics were very different to what was returned when you would run the kafka-consumer-group describe command as you noted.
My co-worker ended up building our own solution that basically just wraps around the command line tools. https://github.com/zalando-incubator/remora > 2. Since burrow returns JSON is there any visualization tools that can be used to monitor the end results. We've an monitoring solution (https://github.com/zalando/zmon) that polls the HTTP endpoint every 60 seconds and places the data into kariosdb. From there we've a time series db to query directly from grafana. It should be possible to throw a simple poller script together that does this for you. > 3. On hitting group describe command and burrow group lag command results are different burrow result is somewhat delayed then the results that I am getting while hitting group describes command on Kafka broker and I am getting the different result. They use a different lag calculation method https://github.com/linkedin/Burrow/wiki/Consumer-Lag-Evaluation-Rules describes it. On 28 May 2017 at 12:59, Abhimanyu Nagrath <abhimanyunagr...@gmail.com> wrote: > Hi , > > I am using burrow to monitor kafka Lags and I am having following queries : > > 1.On hitting the API /v2/kafka/local/consumer/group1/lag I am not able to > view all the topics details present in that group and getting complete: > false in the above JSON. What does this mean? Below mentioned is the json > result of the above query. > { > > "error": false, > "message": "consumer group status returned", > "status": { > "cluster": "local", > "group": "group1", > "status": "OK", > "complete": false, > "partitions": [ > { > "topic": "topic1", > "partition": 1, > "status": "OK", > "start": { > "offset": 144, > "timestamp": 1494566913489, > "lag": 0, > "max_offset": 144 > }, > "end": { > "offset": 144, > "timestamp": 1494566999000, > "lag": 0, > "max_offset": 144 > } > } > ], > "partition_count": 17, > "maxlag": null, > "totallag": 0 > }, > "request": { > "url": "/v2/kafka/local/consumer/group1/lag", > "host": "", > "cluster": "local", > "group": "group1", > "topic": "" > } > } > > > 2. Since burrow returns JSON is there any visualization tools that can be > used to monitor the end results. > > 3. On hitting group describe command and burrow group lag command results > are different burrow result is somewhat delayed then the results that I am > getting while hitting group describes command on Kafka broker and I am > getting the different result. > > > > Below mentioned is my burrow configuration: > > > [general] > logdir=log > logconfig=/root/go/src/github.com/linkedin/Burrow/config/logging.cfg > pidfile=burrow.pid > client-id=burrow-lagchecker > group-blacklist=^(console-consumer-|python-kafka-consumer-).*$ > #group-whitelist=^(my-important-consumer).*$ > > [zookeeper] > hostname=<zookeeper ip> > port=2181 > timeout=6 > lock-path=/burrow/notifier > > [kafka "local"] > broker=<Kafka Ip> > broker-port=9092 > zookeeper=<zookeeper ip> > zookeeper-port=2181 > zookeeper-path=/ > offsets-topic=__consumer_offsets > > #[storm "local"] > #zookeeper=zkhost01.example.com > #zookeeper-port=2181 > #zookeeper-path=/kafka-cluster/stormconsumers > > [tickers] > broker-offsets=20 > > [lagcheck] > intervals=10 > expire-group=604800 > > [notify] > interval=10 > > [httpserver] > server=on > port=8000 > ; Alternatively, use listen (cannot be specified when port is) > ; listen=host:port > ; listen=host2:port2 > > [smtp] > server=mailserver.example.com > port=25 > from=burrow-nore...@example.com > template=config/default-email.tmpl > > [emailnotifier "b...@example.com"] > group=local,critical-consumer-group > group=local,other-consumer-group > interval=60 > > [notify] > interval=10 > > [httpnotifier] > url=http://notification.server.example.com:9000/v1/alert > interval=60 > extra=app=burrow > extra=tier=STG > template-post=config/default-http-post.tmpl > template-delete=config/default-http-delete.tmpl > timeout=5 > keepalive=30 > > So Can you please let me know what I am missing and how to fix these > issues.Any help would be appreciated. > > > > Regards, > Abhimanyu >
