Hi Francois, We had the exact same problem. We embed Kafka in our service container, and we use yammer metrics to see data about the whole app (e.g. kafka, the jvm, the service container wrapping it). However, as you observed, by default, kafka produces an insane amount of metrics. So what we did, is using the yammer library, you can disable specific metrics by removing metrics from the yammer MetricsRegistry, which you can access from guice (if you are using guice). I implemented the MetricsRegistryListener, and added the ability to remove metric names by regex, so I can still have some metrics show up (like the simple 'AllTopic' counts for messages/bytes sent from the producer), but block everything else that's per topic, etc....
Jason On Fri, Sep 19, 2014 at 11:34 PM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hi, > > I don't have any source or configs handy to check things, but you are > saying you've configured Kafka to use GraphiteReporter, right? So why not > remove that config, so metrics stop being sent to Graphite if your Graphite > setup is suffering? If you do that and you still want to see your Kafka > metrics, you can always use SPM <http://sematext.com/spm/> for Kafka > (though some of the graphs will be empty until we get KAFKA-1481, or > something else that improves metrics, in). If you just want to use it > temporarily, just use the free 30-day trial version until you beef up your > Graphite setup. > > Otis > -- > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > Solr & Elasticsearch Support * http://sematext.com/ > > > On Fri, Sep 19, 2014 at 10:08 AM, François Langelier < > f.langel...@gmail.com> > wrote: > > > Hi Daniel, > > > > Thank you for your answer. > > > > It's possible that I didn't understood something, if so correct me > please. > > > > From what I understood, from the kafka doc #monitoring > > <http://kafka.apache.org/documentation.html#monitoring>, kafka use > Yammer > > Metrics for monitoring the servers (the brokers) and the clients > (producers > > and consumers). > > > > Our web site is also using Yammer Metrics and push that to our Graphite > > server and our web site also produce message in kafka. > > From what I read, the Yammer Metrics GraphiteReporter is a kind of > > Singleton, once I Enable it, it is working for all the process. (But I > > might be wrong here...) > > > > We recently upgrade kafka from 0.7.2 to 0.8.1.1 and since the upgrade, > > kafka is monitoring in our Graphite Server and is hammering it, so we > > aren't able to use it because we always get timeout... > > > > SO, I was wondering if there is a way to disable the kafka monitoring to > > our Graphite server. > > > > We are using the code in the tag 0.8.1.1 on github, so if the > kafka-ganglia > > isn't in the tag, we aren't using it :) > > > > > > François Langelier > > Étudiant en génie Logiciel - École de Technologie Supérieure > > <http://www.etsmtl.ca/> > > Capitaine Club Capra <http://capra.etsmtl.ca/> > > VP-Communication - CS Games <http://csgames.org> 2014 > > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014 > > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/> > > Comité Organisateur Olympiades ÉTS 2012 > > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior > > > > On Wed, Sep 17, 2014 at 11:05 PM, Daniel Compton <d...@danielcompton.net > > > > wrote: > > > > > Hi Francois > > > > > > I didn't quite understand how you've set up your metrics reporting. Are > > you > > > using the https://github.com/criteo/kafka-ganglia metrics reporter? If > > so > > > then you should be able to adjust the config to exclude the metrics you > > > don't want, with kafka.ganglia.metrics.exclude.regex. > > > > > > > > > On 18 September 2014 07:55, François Langelier <f.langel...@gmail.com> > > > wrote: > > > > > > > Hi all! > > > > > > > > We are using yammer metrics to monitor some parts of our system. > > > > > > > > Since we upgrade from kafka 0.7.2 to 0.8.1.1, we saw a lot more data > > > > getting in our graphite server and from what I saw, it looks like it > > all > > > > come from our producers. > > > > > > > > From what i understand, since we already use graphite, our > > > graphiteReporter > > > > is enable in our main web site and our kafka producers are having fun > > > using > > > > it too to monitor in graphite. > > > > > > > > The problem is that right now kafka is hammering of graphite server > and > > > we > > > > have difficulty to saw our monitored data... > > > > > > > > Is there a way to deactivate the monitoring of our kafka producers? > > > > > > > > > > > > François Langelier > > > > Étudiant en génie Logiciel - École de Technologie Supérieure > > > > <http://www.etsmtl.ca/> > > > > Capitaine Club Capra <http://capra.etsmtl.ca/> > > > > VP-Communication - CS Games <http://csgames.org> 2014 > > > > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014 > > > > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/> > > > > Comité Organisateur Olympiades ÉTS 2012 > > > > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior > > > > > > > > > >
