On the broker side, we have jmx beans for producer/consumer request rate and time from all clients. Each producer/consumer client has jmx beans that tracks its own request rate and time.
Thanks, Jun On Thu, Jul 25, 2013 at 2:35 AM, Otis Gospodnetic < otis_gospodne...@yahoo.com> wrote: > Hi, > > Oh, I know about Yammer metrics, Ganglia, Graphite, and friends.... > > My main Q was: can I get all Consumer and all Producer stats via Broker's > JMX. I think the answer is no :( > So I'm forced to look at the JMX of individual applications that act as > Kafka Consumers or Producers. > > Btw. this is for SPM - http://sematext.com/spm - support for monitoring > Kafka should be in the early August release. > > As a matter of fact, there is an SPM Reporter from Yammer metrics, so > people should be able to get their Kafka 0.8 metrics graphed as custom > metrics in SPM already. > > Thanks for the patience and the info, Jay! > > Otis > ---- > Performance Monitoring for Solr / ElasticSearch / HBase / Hadoop - > http://sematext.com/spm > > > > >________________________________ > > From: Jay Kreps <jay.kr...@gmail.com> > >To: "users@kafka.apache.org" <users@kafka.apache.org>; Otis Gospodnetic < > otis_gospodne...@yahoo.com> > >Sent: Thursday, July 25, 2013 1:07 AM > >Subject: Re: Metrics: via Broker vs. Producer vs. Consumer > > > > > >Yeah totally that was what I thought you are saying. So what we have is a > >yammer metrics which allows you to plug in any metrics reporter (it will > do > >jmx or whatever). But all this does it let you customize how the > individual > >java processes output metrics. We don't solve the problem of putting all > >these together into a central dashboard. But these can be hooked up to a > >standard monitoring thingy that pulls in metrics from all the java > >processes in your environment (brokers, consumers, non-kafka related > >things, etc) and let's you see these stats in aggregate across all > machines. > > > >I'm actually kind of ignorant of the state of open source monitoring > >thingies because at linkedin we have a pretty badass in-house thing that > >does this and is used for everything. I guess a lot of people used to use > >ganglia or nagios for this but nowadays I hear all the cool kids use > >graphite? > > > >At various times we have tossed around the idea of having a "Kafka > >Monitoring Dashboard" that had a lot of stats on it. This would allow us > to > >make out-of-the-box monitoring a lot easier for Kafka. The feedback we > have > >gotten previously is that everyone has some inhouse tool they already use > >for graphing/alerting/etc and they don't really want a one-off solution > for > >Kafka. This is definitely true for our own usage. > > > >-Jay > > > > > > > >On Wed, Jul 24, 2013 at 10:04 AM, Otis Gospodnetic < > >otis_gospodne...@yahoo.com> wrote: > > > >> Hi, > >> > >> Ah, I think I didn't ask my question clearly. Another try: > >> * If I have a javaagent attached to the Kafka process, I'll be able to > >> connect to its JMX and get all the Broker metrics for that Broker > process. > >> * If I have another Broker process, I'll need to attach my agent to this > >> process, too, to get all Broker metrics associated with this second > Broker > >> process. > >> > >> > >> So far OK - like you said, I can sum, average, etc. > >> > >> But what if I want to get all Producer metrics? What do I need to do? I > >> *believe* I would have to attach the javaagent to whichever app is > acting > >> as a Kafka Consumer and get Consumer stats from the JMX associated with > the > >> JVM process running that app. > >> Is this correct? > >> Is there any way to avoid that and get all Consumer and all Producer > >> metrics using the javaagent attached to one of the Broker processes? > >> > >> Thanks, > >> Otis > >> ---- > >> Performance Monitoring for Solr / ElasticSearch / HBase / Hadoop - > >> http://sematext.com/spm > >> > >> > >> > >> > >> >________________________________ > >> > From: Jay Kreps <jay.kr...@gmail.com> > >> >To: "users@kafka.apache.org" <users@kafka.apache.org> > >> >Sent: Wednesday, July 24, 2013 5:22 PM > >> >Subject: Re: Metrics: via Broker vs. Producer vs. Consumer > >> > > >> > > >> >Yeah all our monitoring is just of the local process (basically just a > >> >counter exposed through yammer metrics which support jmx and other > >> >outputs). If I understand what you want instead of having a counter > that > >> >tracks, say, produce requests per second for a single broker you want > one > >> >that covers the whole cluster. Obviously this would require collecting > the > >> >local count and aggregating across all the brokers. > >> > > >> >Our assumption is that you already have a separate monitoring system > which > >> >can slurp all these up, aggregate them, graph them, and alert off them. > >> >There are a number of open source thingies like this and I think most > >> >bigger shops have something they use. Our assumption is that trying to > do > >> a > >> >kafka-specific monitoring system wouldn't work for most people because > >> they > >> >are wedded to their current setup and just want to integrate with that. > >> > > >> >I'm not sure how valid any of those assumptions actually are. > >> > > >> >-Jay > >> > > >> > > >> >On Wed, Jul 24, 2013 at 7:29 AM, Otis Gospodnetic < > >> >otis.gospodne...@gmail.com> wrote: > >> > > >> >> Hi, > >> >> > >> >> I was looking at > >> >> > >> >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring > >> >> and noticed there is no information about which metrics are available > >> >> in which process/JVM/JMX. > >> >> > >> >> Some are available in the Broker process, but some are only available > >> >> from the JVM running Consumer and some only from the JVM running > >> >> Producer. And yet some Producer and Consumer metrics are, I > *believe* > >> >> available from Broker's JMX. > >> >> > >> >> Would it be possible for somebody in the know to mark the metrics in > >> >> > >> >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring > >> >> so one can tell where to get it? > >> >> > >> >> Also, why is it that the Broker process doesn't have *all* metrics, > >> >> including Producer and Consumer one? Is that because there can be N > >> >> Brokers and each P or C talk to one Broker at a time and thus there > is > >> >> no single process/JMX that can know *all* stats for *all* Brokers and > >> >> for *all* Ps and Cs? > >> >> > >> >> Thank you! > >> >> Otis > >> >> -- > >> >> Performance Monitoring -- http://sematext.com/spm > >> >> Solr & ElasticSearch Support -- http://sematext.com/ > >> >> > >> > > >> > > >> > > >> > > > > > > >
