Good to know! Thanks Jason, I'll look at it ASAP! :)
François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure <http://www.etsmtl.ca/> Membre Club Capra <http://capra.etsmtl.ca/> VP-Communication - CS Games <http://csgames.org> 2014 Jeux de Génie <http://www.jdgets.com/> 2011 à 2014 Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/> Comité Organisateur Olympiades ÉTS 2012 Compétition Québécoise d'Ingénierie 2012 - Compétition Senior On Thu, Nov 6, 2014 at 12:30 PM, Jason Rosenberg <j...@squareup.com> wrote: > Hi Francois, > > We had the exact same problem. We embed Kafka in our service container, > and we use yammer metrics to see data about the whole app (e.g. kafka, the > jvm, the service container wrapping it). However, as you observed, by > default, kafka produces an insane amount of metrics. So what we did, is > using the yammer library, you can disable specific metrics by removing > metrics from the yammer MetricsRegistry, which you can access from guice > (if you are using guice). I implemented the MetricsRegistryListener, and > added the ability to remove metric names by regex, so I can still have some > metrics show up (like the simple 'AllTopic' counts for messages/bytes sent > from the producer), but block everything else that's per topic, etc.... > > Jason > > On Fri, Sep 19, 2014 at 11:34 PM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > > > Hi, > > > > I don't have any source or configs handy to check things, but you are > > saying you've configured Kafka to use GraphiteReporter, right? So why > not > > remove that config, so metrics stop being sent to Graphite if your > Graphite > > setup is suffering? If you do that and you still want to see your Kafka > > metrics, you can always use SPM <http://sematext.com/spm/> for Kafka > > (though some of the graphs will be empty until we get KAFKA-1481, or > > something else that improves metrics, in). If you just want to use it > > temporarily, just use the free 30-day trial version until you beef up > your > > Graphite setup. > > > > Otis > > -- > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > On Fri, Sep 19, 2014 at 10:08 AM, François Langelier < > > f.langel...@gmail.com> > > wrote: > > > > > Hi Daniel, > > > > > > Thank you for your answer. > > > > > > It's possible that I didn't understood something, if so correct me > > please. > > > > > > From what I understood, from the kafka doc #monitoring > > > <http://kafka.apache.org/documentation.html#monitoring>, kafka use > > Yammer > > > Metrics for monitoring the servers (the brokers) and the clients > > (producers > > > and consumers). > > > > > > Our web site is also using Yammer Metrics and push that to our Graphite > > > server and our web site also produce message in kafka. > > > From what I read, the Yammer Metrics GraphiteReporter is a kind of > > > Singleton, once I Enable it, it is working for all the process. (But I > > > might be wrong here...) > > > > > > We recently upgrade kafka from 0.7.2 to 0.8.1.1 and since the upgrade, > > > kafka is monitoring in our Graphite Server and is hammering it, so we > > > aren't able to use it because we always get timeout... > > > > > > SO, I was wondering if there is a way to disable the kafka monitoring > to > > > our Graphite server. > > > > > > We are using the code in the tag 0.8.1.1 on github, so if the > > kafka-ganglia > > > isn't in the tag, we aren't using it :) > > > > > > > > > François Langelier > > > Étudiant en génie Logiciel - École de Technologie Supérieure > > > <http://www.etsmtl.ca/> > > > Capitaine Club Capra <http://capra.etsmtl.ca/> > > > VP-Communication - CS Games <http://csgames.org> 2014 > > > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014 > > > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/> > > > Comité Organisateur Olympiades ÉTS 2012 > > > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior > > > > > > On Wed, Sep 17, 2014 at 11:05 PM, Daniel Compton < > d...@danielcompton.net > > > > > > wrote: > > > > > > > Hi Francois > > > > > > > > I didn't quite understand how you've set up your metrics reporting. > Are > > > you > > > > using the https://github.com/criteo/kafka-ganglia metrics reporter? > If > > > so > > > > then you should be able to adjust the config to exclude the metrics > you > > > > don't want, with kafka.ganglia.metrics.exclude.regex. > > > > > > > > > > > > On 18 September 2014 07:55, François Langelier < > f.langel...@gmail.com> > > > > wrote: > > > > > > > > > Hi all! > > > > > > > > > > We are using yammer metrics to monitor some parts of our system. > > > > > > > > > > Since we upgrade from kafka 0.7.2 to 0.8.1.1, we saw a lot more > data > > > > > getting in our graphite server and from what I saw, it looks like > it > > > all > > > > > come from our producers. > > > > > > > > > > From what i understand, since we already use graphite, our > > > > graphiteReporter > > > > > is enable in our main web site and our kafka producers are having > fun > > > > using > > > > > it too to monitor in graphite. > > > > > > > > > > The problem is that right now kafka is hammering of graphite server > > and > > > > we > > > > > have difficulty to saw our monitored data... > > > > > > > > > > Is there a way to deactivate the monitoring of our kafka producers? > > > > > > > > > > > > > > > François Langelier > > > > > Étudiant en génie Logiciel - École de Technologie Supérieure > > > > > <http://www.etsmtl.ca/> > > > > > Capitaine Club Capra <http://capra.etsmtl.ca/> > > > > > VP-Communication - CS Games <http://csgames.org> 2014 > > > > > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014 > > > > > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/> > > > > > Comité Organisateur Olympiades ÉTS 2012 > > > > > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior > > > > > > > > > > > > > > >
