Is it actually getting double-counted? I tried reproducing this locally but the BrokerTopicMetrics.Count lines up with the sum of the PerTopic.Counts for various metrics.
On Tue, Jan 27, 2015 at 03:29:37AM -0500, Jason Rosenberg wrote: > Ok, > > It looks like the yammer MetricName is not being created correctly for the > sub metrics that include a topic. E.g. a metric with an mbeanName like: > > kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic > > appears to be malformed. A yammer MetricName has 4 fields that are used in > creating a graphite metric, that are included in the constructor: > group, type, name, scope. > > In this case, the metric with the above mbeanName has these fields set in > the MetricName: > > group: "kafka.server" > type: "BrokerTopicMetrics" > name: "BytesInPerSec" > scope: null > > Thus, the topic metrics all look the same, and get lumped into the > top-level BrokerTopicMetrics (and thus that will now be double counted). It > looks like the fix for kafka-1481 was where things got broken. It seems to > have introduced ‘tags’ in the building of metric names, and then those tags > only get applied to the mbeanName, but get excluded from the metric name: > https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f > > This is a pretty severe issue, since the yammer metrics for these stats > will be double counted in aggregate, and the per-topic stats will be > removed. > > I should note too, in my previous email, I thought that only the per-topic > BrokerTopicMetrics were missing, but also several other per-topic metrics > are missing too, e.g. under kafka.log, etc. > > Jason > > > On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg <j...@squareup.com> wrote: > > > I can confirm that the per topic metrics are not coming through to the > > yammer metrics registry. I do see them in jmx (via jconsole), but the > > MetricsRegistry does not have them. > > All the other metrics are coming through that appear in jmx. > > > > This is with single node instance running locally. > > > > Jason > > > > > > > > On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy <ku...@nmsworks.co.in> > > wrote: > > > >> If you are using multi-node cluster, then metrics may be reported from > >> other servers. > >> pl check all the servers in the cluster. > >> > >> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <kyleban...@gmail.com> > >> wrote: > >> > >> > I've been using a custom KafkaMetricsReporter to report Kafka broker > >> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and > >> messages in > >> > and out for all topics together and for each individual topic. > >> > > >> > After upgrading to v0.8.2.0, these metrics are no longer being reported. > >> > > >> > I'm only seeing the following: > >> > BrokerTopicMetrics > >> > - BytesInPerSec > >> > - BytesOutPerSec > >> > - BytesRejectedPerSec > >> > - MessagesInPerSec > >> > > >> > What's more, despite lots of successful writes to the cluster, the > >> values > >> > for these remaining metrics are all zero. > >> > > >> > I saw that there was some refactoring of metric naming code. Was the > >> > behavior supposed to have changed? > >> > > >> > Many thanks in advance. > >> > > >> > > > >
