Sorry I want to know you want to monitor kafka producers or kafka brokers and zookeepers ? It's seems you will want to monitor monitor Exceptions eg Leader Not Found, Queue is full, resend fail etc are kafka cluster
2014-06-25 8:20 GMT+08:00 Bhavesh Mistry <mistry.p.bhav...@gmail.com>: > We use Kafka as Transport Layer to transport application logs. How do we > monitor Producers at large scales about 6000 boxes x 4 topic per box so > roughly 24000 producers (spread across multiple data center.. we have > brokers per DC). We do the monitoring based on logs. I have tried > intercepting logs via Log4J custom implementation which only intercept WARN > and ERROR and FATAL events org.apache.log4j.AppenderSkeleton append method > which send its logs to brokers (This is working but after load testing it > is causing deadlock some times between ProducerSendThread and Producer). > > I know there are JMX monitoring MBeans available which we can pull the > data, but I would like to monitor Exceptions eg Leader Not Found, Queue is > full, resend fail etc in Kafka Library. > > How does LinkedIn monitor the Producers ? > > Thanks, > > Bhavesh > -- long is the way and hard that out of Hell leads up to light
