Jay - I was thinking of a pure stub rather than just wrapping Kafka metrics
in a Coda gauge. I'd like the Timers, Meters etc to still be Coda meters -
that way the windows, exponential decays, etc are comparable to the rest of
the Coda metrics in our applications. At the same time, I don't want to
force Coda timers (or any other timers) on an app that won't make good use
of them.
Thanks again, C
On Sat, Feb 22, 2014 at 9:25 AM, Martin Kleppmann
<mkleppm...@linkedin.com>wrote:
> Not sure if you want yet another opinion added to the pile -- but since I
> had a similar problem on another project recently, I thought I'd weigh in.
> (On that project we were originally using Coda's library, but then switched
> to rolling our own metrics implementation because we needed to do a few
> things differently.)
>
> 1. Problems we encountered with Coda's library: it uses an
> exponentially-weighted moving average (EMWA) for rates (eg. messages/sec),
> and exponentially biased reservoir sampling for histograms (percentiles,
> averages). Those methods of calculation work well for events with a
> consistently high volume, but they give strange and misleading results for
> events that are bursty or rare (eg error rates). We found that a fixed-size
> window gives more predictable, easier-to-interpret results.
>
> 2. In defence of Coda's library, I think its histogram implementation is a
> good trade-off of memory for accuracy; I'm not totally convinced that your
> proposal (counts of events in a fixed set of buckets) would be much better.
> Would have to do some math to work out the expected accuracy in each case.
> The reservoir sampling can be configured to use a smaller sample if the
> default of 1028 samples is too expensive. Reservoir sampling also has the
> advantage that you don't need to hard-code a bucket distribution.
>
> 3. Quotas are an interesting use case. However, I'm not wild about using a
> QuotaViolationException for control flow -- I think an explicit conditional
> would be nicer than having to catch an exception. One question in that
> context: if a quota is exceeded, do you still want to count the event
> towards the metric, or do you want to stop counting it until the quota is
> replenished? The answer may depend on the particular metric.
>
> 4. If you decide to go with Coda's library, I would advocate isolating the
> dependency into a separate module and using it via a facade -- somewhat
> like using SLF4J instead of Log4j directly. It's ok for Coda's library to
> be the default metrics implementation, but it should be easy to swap it out
> for something different in case someone has a version conflict or differing
> requirements. The facade should be at a low level (individual events), not
> at the reporter level (which deals with pre-aggregated values, and is
> already pluggable).
>
> 5. If it's useful, I can probably contribute my simple (but imho
> effective) metrics library, for embedding into Kafka. It uses reservoir
> sampling for percentiles, like Coda's library, but uses a fixed-size window
> instead of an exponential bias, which avoids weird behaviour on bursty
> metrics.
>
> In summary, I would advocate one of the following approaches:
> - Coda Hale library via facade (allowing it to be swapped for something
> else), or
> - Own metrics implementation, provided that we have confidence in its
> implementation of percentiles.
>
> Martin
>
>
> On 22 Feb 2014, at 01:06, Jay Kreps <jay.kr...@gmail.com> wrote:
> > Hey guys,
> >
> > Just picking up this thread again. I do want to drive a conclusion as I
> > will run out of work to do on the producer soon and will need to add
> > metrics of some sort. We can vote on it, but I'm not sure if we actually
> > got everything discussed.
> >
> > Joel, I wasn't fully sure how to interpret your comment. I think you are
> > saying you are cool with the new metrics package as long as it really is
> > better. Do you have any comment on whether you think the benefits I
> > outlined are worth it? I agree with you that we could hold off on a
> second
> > repo until someone else would actually want to use our code.
> >
> > Jun, I'm not averse to doing a sampling-based histogram and doing some
> > comparison between the two approaches if you think this approach is
> > otherwise better.
> >
> > Sriram, originally I thought you preferred just sticking to Coda Hale,
> but
> > after your follow-up email I wasn't really sure...
> >
> > Joe/Clark, yes this code allows pluggable reporting so you could have a
> > metrics reporter that just wraps each metric in a Coda Hale Gauge if that
> > is useful. Though obviously if enough people were doing that I would
> think
> > it would be worth just using the Coda Hale package directly...
> >
> > -Jay
> >
> >
> >
> >
> > On Thu, Feb 13, 2014 at 3:34 PM, Clark Breyman <cl...@breyman.com>
> wrote:
> >
> >> Not requiring the client to link Coda/Yammer metrics sounds like a
> >> compelling reason to pivot to new interfaces. If that's the agreed
> >> direction, I'm hoping that we'd get the choice of backend to provide
> (e.g.
> >> facade on Yammer metrics for those with an investment in that) rather
> than
> >> force the new backend. Having a metrics factory seems better for this
> than
> >> directly instantiating the singleton registry.
> >>
> >>
> >> On Thu, Feb 13, 2014 at 2:39 PM, Joe Stein <joe.st...@stealth.ly>
> wrote:
> >>
> >>> Can we leave metrics and have multiple supported KafkaMetricsGroup
> >>> implementing a yammer based implementation?
> >>>
> >>> ProducerRequestStats with your configured analytics group?
> >>>
> >>> On Thu, Feb 13, 2014 at 11:37 AM, Jay Kreps <jay.kr...@gmail.com>
> wrote:
> >>>
> >>>> I think we discussed the scala/java stuff more fully previously.
> >>>> Essentially the client is embedded everywhere. Scala is very
> >> incompatible
> >>>> with itself so this makes it very hard to use for people using
> anything
> >>>> else in scala. Also Scala stack traces are very confusing. Basically
> we
> >>>> thought plain java code would be a lot easier for people to use. Even
> >> if
> >>>> Scala is more fun to write, that isn't really what we are optimizing
> >> for.
> >>>>
> >>>> -Jay
> >>>>
> >>>>
> >>>> On Thu, Feb 13, 2014 at 8:09 AM, S Ahmed <sahmed1...@gmail.com>
> wrote:
> >>>>
> >>>>> Jay, pretty impressive how you just write a 'quick version' like that
> >>> :)
> >>>>> Not to get off-topic but why didn't you write this in scala?
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Feb 12, 2014 at 6:54 PM, Joel Koshy <jjkosh...@gmail.com>
> >>> wrote:
> >>>>>
> >>>>>> I have not had a chance to review the new metrics code and its
> >>>>>> features carefully (apart from your write-up), but here are my
> >>> general
> >>>>>> thoughts:
> >>>>>>
> >>>>>> Implementing a metrics package correctly is difficult; more so for
> >>>>>> people like me, because I'm not a statistician. However, if this
> >> new
> >>>>>> package: {(i) functions correctly (and we need to define and prove
> >>>>>> correctness), (ii) is easy to use, (iii) serves all our current and
> >>>>>> anticipated monitoring needs, (iv) is not overly complex that it
> >>>>>> becomes a burden to maintain and we are better of with an available
> >>>>>> library;} then I think it makes sense to embed it and use it within
> >>>>>> the Kafka code. The main wins are: (i) predictability (no changing
> >>>>>> APIs and intimate knowledge of the code) and (ii) control with
> >>> respect
> >>>>>> to both functionality (e.g., there are hard-coded decay constants
> >> in
> >>>>>> metrics-core 2.x) and correctness (i.e., if we find a bug in the
> >>>>>> metrics package we have to submit a pull request and wait for it to
> >>>>>> become mainstream). I'm not sure it would help very much to pull
> >> it
> >>>>>> into a separate repo because that could potentially annul these
> >>>>>> benefits.
> >>>>>>
> >>>>>> Joel
> >>>>>>
> >>>>>> On Wed, Feb 12, 2014 at 02:50:43PM -0800, Jay Kreps wrote:
> >>>>>>> Sriram,
> >>>>>>>
> >>>>>>> Makes sense. I am cool moving this stuff into its own repo if
> >>> people
> >>>>>> think
> >>>>>>> that is better. I'm not sure it would get much contribution but
> >>> when
> >>>> I
> >>>>>>> started messing with this I did have a lot of grand ideas of
> >> making
> >>>>>> adding
> >>>>>>> metrics to a sensor dynamic so you could add more stuff in
> >>>>> real-time(via
> >>>>>>> jmx, say) and/or externalize all your metrics and config to a
> >>>> separate
> >>>>>> file
> >>>>>>> like log4j with only the points of instrumentation hard-coded.
> >>>>>>>
> >>>>>>> -Jay
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Feb 12, 2014 at 2:07 PM, Sriram Subramanian <
> >>>>>>> srsubraman...@linkedin.com> wrote:
> >>>>>>>
> >>>>>>>> I am actually neutral to this change. I found the replies were
> >>> more
> >>>>>>>> towards the implementation and features so far. I would like
> >> the
> >>>>>> community
> >>>>>>>> to think about the questions below before making a decision. My
> >>>>>> opinion on
> >>>>>>>> this is that it has potential to be its own project and it
> >> would
> >>>>>> attract
> >>>>>>>> developers who are specifically interested in contributing to
> >>>>> metrics.
> >>>>>> I
> >>>>>>>> am skeptical that the Kafka contributors would focus on
> >> improving
> >>>>> this
> >>>>>>>> library (apart from bug fixes) instead of
> >> developing/contributing
> >>>> to
> >>>>>> other
> >>>>>>>> core pieces. It would be useful to continue and keep it
> >> decoupled
> >>>>> from
> >>>>>>>> rest of Kafka (if it resides in the Kafka code base.) so that
> >> we
> >>>> can
> >>>>>> move
> >>>>>>>> it out anytime to its own project.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 2/12/14 1:21 PM, "Jay Kreps" <jay.kr...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hey Sriram,
> >>>>>>>>>
> >>>>>>>>> Not sure if these are actually meant as questions or more
> >> veiled
> >>>>>> comments.
> >>>>>>>>> In an case I tried to give my 2 cents inline.
> >>>>>>>>>
> >>>>>>>>> On Tue, Feb 11, 2014 at 11:12 PM, Sriram Subramanian <
> >>>>>>>>> srsubraman...@linkedin.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> I think answering the questions below would help to make a
> >>>> better
> >>>>>>>>>> decision. I am all for writing better code and having
> >> superior
> >>>>>>>>>> functionalities but it is worth thinking about stuff outside
> >>>> just
> >>>>>> code
> >>>>>>>>>> in
> >>>>>>>>>> this case -
> >>>>>>>>>>
> >>>>>>>>>> 1. Does metric form a core piece of kafka? Does it help
> >> kafka
> >>>>>> greatly in
> >>>>>>>>>> providing better core functionalities? I would always like a
> >>>>>> project to
> >>>>>>>>>> do
> >>>>>>>>>> one thing really well. Metrics is a non trivial amount of
> >>> code.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Metrics are obviously important, and obviously improving our
> >>>> metrics
> >>>>>>>>> system
> >>>>>>>>> would be good. That said this may or may not be better, and
> >> even
> >>>> if
> >>>>>> it is
> >>>>>>>>> better that betterness might not outweigh other
> >> considerations.
> >>>> That
> >>>>>> is
> >>>>>>>>> what we are discussing.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> 2. Does it make sense to be part of Kafka or its own
> >> project?
> >>> If
> >>>>>> this
> >>>>>>>>>> metrics library has the potential to be better than
> >>>> metrics-core,
> >>>>> I
> >>>>>>>>>> would
> >>>>>>>>>> be interested in other projects take advantage of it.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> It could be either.
> >>>>>>>>>
> >>>>>>>>> 3. Can Kafka maintain this library as new members join and old
> >>>>> members
> >>>>>>>>>> leave? Would this be a piece of code that no one (in Kafka)
> >> in
> >>>> the
> >>>>>>>>>> future
> >>>>>>>>>> spends time improving if the original author left?
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I am not going anywhere in the near term, but if I did, yes,
> >>> this
> >>>>>> would be
> >>>>>>>>> like any other code we have. As with yammer metrics or any
> >> other
> >>>>> code
> >>>>>> at
> >>>>>>>>> that point we would either use it as is or someone would
> >> improve
> >>>> it.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> 4. Does it affect the schedule of producer rewrite? This
> >> needs
> >>>> its
> >>>>>> own
> >>>>>>>>>> stabilization and modification to existing metric dashboards
> >>> if
> >>>>> the
> >>>>>>>>>> format
> >>>>>>>>>> is changed. Many times such cost are not factored in and a
> >>>> project
> >>>>>> loses
> >>>>>>>>>> time before realizing the extra time required to make a
> >>> library
> >>>> as
> >>>>>> this
> >>>>>>>>>> operational.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Probably not. The metrics are going to change regardless of
> >>>> whether
> >>>>>> we use
> >>>>>>>>> the same library or not. If we think this is better I don't
> >> mind
> >>>>>> putting
> >>>>>>>>> in
> >>>>>>>>> a little extra effort to get there.
> >>>>>>>>>
> >>>>>>>>> Irrespective I think this is probably not the right thing to
> >>>>> optimize
> >>>>>> for.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> I am sure we can do better when we write code to a specific
> >>> use
> >>>>>> case (in
> >>>>>>>>>> this case, kafka) rather than building a generic library
> >> that
> >>>>> suits
> >>>>>> all
> >>>>>>>>>> (metrics-core) but I would like us to have answers to the
> >>>>> questions
> >>>>>>>>>> above
> >>>>>>>>>> and be prepared before we proceed to support this with the
> >>>>> producer
> >>>>>>>>>> rewrite.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Naturally we are all considering exactly these things, that is
> >>>>>> exactly the
> >>>>>>>>> reason I started the thread.
> >>>>>>>>>
> >>>>>>>>> -Jay
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On 2/11/14 6:28 PM, "Jun Rao" <jun...@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Thanks for the detailed write-up. It's well thought
> >> through.
> >>> A
> >>>>> few
> >>>>>>>>>>> comments:
> >>>>>>>>>>>
> >>>>>>>>>>> 1. I have a couple of concerns on the percentiles. The
> >> first
> >>>>> issue
> >>>>>> is
> >>>>>>>>>> that
> >>>>>>>>>>> It requires the user to know the value range. Since the
> >> range
> >>>> for
> >>>>>>>>>> things
> >>>>>>>>>>> like message size (in millions) is quite different from
> >> those
> >>>>> like
> >>>>>>>>>> request
> >>>>>>>>>>> time (less than 100), it's going to be hard to pick a good
> >>>> global
> >>>>>>>>>> default
> >>>>>>>>>>> range. Different apps could be dealing with different
> >> message
> >>>>>> size. So
> >>>>>>>>>>> they
> >>>>>>>>>>> probably will have to customize the range. Another issue is
> >>>> that
> >>>>>> it can
> >>>>>>>>>>> only report values at the bucket boundaries. So, if you
> >> have
> >>>> 1000
> >>>>>>>>>> buckets
> >>>>>>>>>>> and a value range of 1 million, you will only see 1000
> >>> possible
> >>>>>> values
> >>>>>>>>>> as
> >>>>>>>>>>> the quantile, which is probably too sparse. The
> >>> implementation
> >>>> of
> >>>>>>>>>>> histogram
> >>>>>>>>>>> in metrics-core keeps a fix size of samples, which avoids
> >>> both
> >>>>>> issues.
> >>>>>>>>>>>
> >>>>>>>>>>> 2. We need to document the 3-part metrics names better
> >> since
> >>>> it's
> >>>>>> not
> >>>>>>>>>>> obvious what the convention is. Also, currently the name of
> >>> the
> >>>>>> sensor
> >>>>>>>>>> and
> >>>>>>>>>>> the metrics defined in it are independent. Would it make
> >>> sense
> >>>> to
> >>>>>> have
> >>>>>>>>>> the
> >>>>>>>>>>> sensor name be a prefix of the metric name?
> >>>>>>>>>>>
> >>>>>>>>>>> Overall, this approach seems to be cleaner than
> >> metrics-core
> >>> by
> >>>>>>>>>> decoupling
> >>>>>>>>>>> measuring and reporting. The main benefit of metrics-core
> >>> seems
> >>>>> to
> >>>>>> be
> >>>>>>>>>> the
> >>>>>>>>>>> existing reporters. Since not that many people voted for
> >>>>>> metrics-core,
> >>>>>>>>>> I
> >>>>>>>>>>> am
> >>>>>>>>>>> ok with going with the new implementation. My only
> >>>> recommendation
> >>>>>> is to
> >>>>>>>>>>> address the concern on percentiles.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>
> >>>>>>>>>>> Jun
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Feb 6, 2014 at 12:51 PM, Jay Kreps <
> >>>> jay.kr...@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hey guys,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I wanted to kick off a quick discussion of metrics with
> >>>> respect
> >>>>>> to
> >>>>>>>>>> the
> >>>>>>>>>>>> new
> >>>>>>>>>>>> producer and consumer (and potentially the server).
> >>>>>>>>>>>>
> >>>>>>>>>>>> At a high level I think there are three approaches we
> >> could
> >>>>> take:
> >>>>>>>>>>>> 1. Plain vanilla JMX
> >>>>>>>>>>>> 2. Use Coda Hale (AKA Yammer) Metrics
> >>>>>>>>>>>> 3. Do our own metrics (with JMX as one output)
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. Has the advantage that JMX is the most commonly used
> >>> java
> >>>>>> thing
> >>>>>>>>>> and
> >>>>>>>>>>>> plugs in reasonably to most metrics systems. JMX is
> >>> included
> >>>> in
> >>>>>> the
> >>>>>>>>>> JDK
> >>>>>>>>>>>> so
> >>>>>>>>>>>> it doesn't impose any additional dependencies on clients.
> >>> It
> >>>>> has
> >>>>>> the
> >>>>>>>>>>>> disadvantage that plain vanilla JMX is a pain to use. We
> >>>> would
> >>>>>> need a
> >>>>>>>>>>>> bunch
> >>>>>>>>>>>> of helper code for maintaining counters to make this
> >>>>> reasonable.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2. Coda Hale metrics is pretty good and broadly used. It
> >>>>>> supports JMX
> >>>>>>>>>>>> output as well as direct output to many other types of
> >>>> systems.
> >>>>>> The
> >>>>>>>>>>>> primary
> >>>>>>>>>>>> downside we have had with Coda Hale has to do with the
> >>>> clients
> >>>>>> and
> >>>>>>>>>>>> library
> >>>>>>>>>>>> incompatibilities. We are currently on an older more
> >>> popular
> >>>>>> version.
> >>>>>>>>>>>> The
> >>>>>>>>>>>> newer version is a rewrite of the APIs and is
> >> incompatible.
> >>>>>>>>>> Originally
> >>>>>>>>>>>> these were totally incompatible and people had to choose
> >>> one
> >>>> or
> >>>>>> the
> >>>>>>>>>>>> other.
> >>>>>>>>>>>> I think that has been improved so now the new version is
> >> a
> >>>>>> totally
> >>>>>>>>>>>> different package. But even in this case you end up with
> >>> both
> >>>>>>>>>> versions
> >>>>>>>>>>>> if
> >>>>>>>>>>>> you use Kafka and we are on a different version than you
> >>>> which
> >>>>> is
> >>>>>>>>>> going
> >>>>>>>>>>>> to
> >>>>>>>>>>>> be pretty inconvenient.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 3. Doing our own has the downside of potentially
> >>> reinventing
> >>>>> the
> >>>>>>>>>> wheel,
> >>>>>>>>>>>> and
> >>>>>>>>>>>> potentially needing to work out any bugs in our code. The
> >>>>> upsides
> >>>>>>>>>> would
> >>>>>>>>>>>> depend on the how good the reinvention was. As it
> >> happens I
> >>>>> did a
> >>>>>>>>>> quick
> >>>>>>>>>>>> (~900 loc) version of a metrics library that is under
> >>>>>>>>>>>> kafka.common.metrics.
> >>>>>>>>>>>> I think it has some advantages over the Yammer metrics
> >>>> package
> >>>>>> for
> >>>>>>>>>> our
> >>>>>>>>>>>> usage beyond just not causing incompatibilities. I will
> >>>>> describe
> >>>>>> this
> >>>>>>>>>>>> code
> >>>>>>>>>>>> so we can discuss the pros and cons. Although I favor
> >> this
> >>>>>> approach I
> >>>>>>>>>>>> have
> >>>>>>>>>>>> no emotional attachment and wouldn't be too sad if I
> >> ended
> >>> up
> >>>>>>>>>> deleting
> >>>>>>>>>>>> it.
> >>>>>>>>>>>> Here are javadocs for this code, though I haven't written
> >>>> much
> >>>>>>>>>>>> documentation yet since I might end up deleting it:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Here is a quick overview of this library.
> >>>>>>>>>>>>
> >>>>>>>>>>>> There are three main public interfaces:
> >>>>>>>>>>>> Metrics - This is a repository of metrics being
> >> tracked.
> >>>>>>>>>>>> Metric - A single, named numerical value being measured
> >>>>> (i.e. a
> >>>>>>>>>>>> counter).
> >>>>>>>>>>>> Sensor - This is a thing that records values and
> >> updates
> >>>> zero
> >>>>>> or
> >>>>>>>>>> more
> >>>>>>>>>>>> metrics
> >>>>>>>>>>>>
> >>>>>>>>>>>> So let's say we want to track three values about message
> >>>> sizes;
> >>>>>>>>>>>> specifically say we want to record the average, the
> >>> maximum,
> >>>>> the
> >>>>>>>>>> total
> >>>>>>>>>>>> rate
> >>>>>>>>>>>> of bytes being sent, and a count of messages. Then we
> >> would
> >>>> do
> >>>>>>>>>> something
> >>>>>>>>>>>> like this:
> >>>>>>>>>>>>
> >>>>>>>>>>>> // setup code
> >>>>>>>>>>>> Metrics metrics = new Metrics(); // this is a global
> >>>>>> "singleton"
> >>>>>>>>>>>> Sensor sensor =
> >>>>>> metrics.sensor("kafka.producer.message.sizes");
> >>>>>>>>>>>> sensor.add("kafka.producer.message-size.avg", new
> >>> Avg());
> >>>>>>>>>>>> sensor.add("kafka.producer.message-size.max", new
> >>> Max());
> >>>>>>>>>>>> sensor.add("kafka.producer.bytes-sent-per-sec", new
> >>>> Rate());
> >>>>>>>>>>>> sensor.add("kafka.producer.message-count", new
> >> Count());
> >>>>>>>>>>>>
> >>>>>>>>>>>> // now when we get a message we do this
> >>>>>>>>>>>> sensor.record(messageSize);
> >>>>>>>>>>>>
> >>>>>>>>>>>> The above code creates the global metrics repository,
> >>>> creates a
> >>>>>>>>>> single
> >>>>>>>>>>>> Sensor, and defines 5 named metrics that are updated by
> >>> that
> >>>>>> Sensor.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Like Yammer Metrics (YM) I allow you to plug in
> >>> "reporters",
> >>>>>>>>>> including a
> >>>>>>>>>>>> JMX reporter. Unlike the Coda Hale JMX reporter the
> >>> reporter
> >>>> I
> >>>>>> have
> >>>>>>>>>> keys
> >>>>>>>>>>>> off the metric names not the Sensor names, which I think
> >> is
> >>>> an
> >>>>>>>>>>>> improvement--I just use the convention that the last
> >>> portion
> >>>> of
> >>>>>> the
> >>>>>>>>>>>> name is
> >>>>>>>>>>>> the attribute name, the second to last is the mbean name,
> >>> and
> >>>>> the
> >>>>>>>>>> rest
> >>>>>>>>>>>> is
> >>>>>>>>>>>> the package. So in the above example there is a producer
> >>>> mbean
> >>>>>> that
> >>>>>>>>>> has
> >>>>>>>>>>>> a
> >>>>>>>>>>>> avg and max attribute and a producer mbean that has a
> >>>>>>>>>> bytes-sent-per-sec
> >>>>>>>>>>>> and message-count attribute. This is nice because you can
> >>>>>> logically
> >>>>>>>>>>>> group
> >>>>>>>>>>>> the values reported irrespective of where in the program
> >>> they
> >>>>> are
> >>>>>>>>>>>> computed--that is an mbean can logically group attributes
> >>>>>> computed
> >>>>>>>>>> off
> >>>>>>>>>>>> different sensors. This means you can report values by
> >>>> logical
> >>>>>>>>>>>> subsystem.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I also allow the concept of hierarchical Sensors which I
> >>>> think
> >>>>>> is a
> >>>>>>>>>> good
> >>>>>>>>>>>> convenience. I have noticed a common pattern in systems
> >>> where
> >>>>> you
> >>>>>>>>>> need
> >>>>>>>>>>>> to
> >>>>>>>>>>>> roll up the same values along different dimensions. An
> >>> simple
> >>>>>>>>>> example is
> >>>>>>>>>>>> metrics about qps, data rate, etc on the broker. These we
> >>>> want
> >>>>> to
> >>>>>>>>>>>> capture
> >>>>>>>>>>>> in aggregate, but also broken down by topic-id. You can
> >> do
> >>>> this
> >>>>>>>>>> purely
> >>>>>>>>>>>> by
> >>>>>>>>>>>> defining the sensor hierarchy:
> >>>>>>>>>>>> Sensor allSizes = metrics.sensor("kafka.producer.sizes");
> >>>>>>>>>>>> Sensor topicSizes = metrics.sensor("kafka.producer." +
> >>> topic
> >>>> +
> >>>>>>>>>>>> ".sizes",
> >>>>>>>>>>>> allSizes);
> >>>>>>>>>>>> Now each actual update will go to the appropriate
> >>> topicSizes
> >>>>>> sensor
> >>>>>>>>>>>> (based
> >>>>>>>>>>>> on the topic name), but allSizes metrics will get updated
> >>>> too.
> >>>>> I
> >>>>>> also
> >>>>>>>>>>>> support multiple parents for each sensor as well as
> >>> multiple
> >>>>>> layers
> >>>>>>>>>> of
> >>>>>>>>>>>> hiearchy, so you can define a more elaborate DAG of
> >>> sensors.
> >>>> An
> >>>>>>>>>> example
> >>>>>>>>>>>> of
> >>>>>>>>>>>> how this would be useful is if you wanted to record your
> >>>>> metrics
> >>>>>>>>>> broken
> >>>>>>>>>>>> down by topic AND client id as well as the global
> >>> aggregate.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Each metric can take a configurable Quota value which
> >>> allows
> >>>> us
> >>>>>> to
> >>>>>>>>>> limit
> >>>>>>>>>>>> the maximum value of that sensor. This is intended for
> >> use
> >>> on
> >>>>> the
> >>>>>>>>>>>> server as
> >>>>>>>>>>>> part of our Quota implementation. The way this works is
> >>> that
> >>>>> you
> >>>>>>>>>> record
> >>>>>>>>>>>> metrics as usual:
> >>>>>>>>>>>> mySensor.record(42.0)
> >>>>>>>>>>>> However if this event occurance causes one of the metrics
> >>> to
> >>>>>> exceed
> >>>>>>>>>> its
> >>>>>>>>>>>> maximum allowable value (the quota) this call will throw
> >> a
> >>>>>>>>>>>> QuotaViolationException. The cool thing about this is
> >> that
> >>> it
> >>>>>> means
> >>>>>>>>>> we
> >>>>>>>>>>>> can
> >>>>>>>>>>>> define quotas on anything we capture metrics for, which I
> >>>> think
> >>>>>> is
> >>>>>>>>>>>> pretty
> >>>>>>>>>>>> cool.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Another question is how to handle windowing of the
> >> values?
> >>>>>> Metrics
> >>>>>>>>>> want
> >>>>>>>>>>>> to
> >>>>>>>>>>>> record the "current" value, but the definition of current
> >>> is
> >>>>>>>>>> inherently
> >>>>>>>>>>>> nebulous. A few of the obvious gotchas are that if you
> >>> define
> >>>>>>>>>> "current"
> >>>>>>>>>>>> to
> >>>>>>>>>>>> be a number of events you can end up measuring an
> >>> arbitrarily
> >>>>>> long
> >>>>>>>>>>>> window
> >>>>>>>>>>>> of time if the event rate is low (e.g. you think you are
> >>>>> getting
> >>>>>> 50
> >>>>>>>>>>>> messages/sec because that was the rate yesterday when all
> >>>>> events
> >>>>>>>>>>>> topped).
> >>>>>>>>>>>>
> >>>>>>>>>>>> Here is how I approach this. All the metrics use the same
> >>>>>> windowing
> >>>>>>>>>>>> approach. We define a single window by a length of time
> >> or
> >>>>>> number of
> >>>>>>>>>>>> values
> >>>>>>>>>>>> (you can use either or both--if both the window ends when
> >>>>>> *either*
> >>>>>>>>>> the
> >>>>>>>>>>>> time
> >>>>>>>>>>>> bound or event bound is hit). The typical problem with
> >> hard
> >>>>>> window
> >>>>>>>>>>>> boundaries is that at the beginning of the window you
> >> have
> >>> no
> >>>>>> data
> >>>>>>>>>> and
> >>>>>>>>>>>> the
> >>>>>>>>>>>> first few samples are too small to be a valid sample.
> >>>> (Consider
> >>>>>> if
> >>>>>>>>>> you
> >>>>>>>>>>>> were
> >>>>>>>>>>>> keeping an avg and the first value in the window happens
> >> to
> >>>> be
> >>>>>> very
> >>>>>>>>>> very
> >>>>>>>>>>>> high, if you check the avg at this exact time you will
> >>>> conclude
> >>>>>> the
> >>>>>>>>>> avg
> >>>>>>>>>>>> is
> >>>>>>>>>>>> very high but on a sample size of one). One simple fix
> >>> would
> >>>> be
> >>>>>> to
> >>>>>>>>>>>> always
> >>>>>>>>>>>> report the last complete window, however this is not
> >>>>> appropriate
> >>>>>> here
> >>>>>>>>>>>> because (1) we want to drive quotas off it so it needs to
> >>> be
> >>>>>> current,
> >>>>>>>>>>>> and
> >>>>>>>>>>>> (2) since this is for monitoring you kind of care more
> >>> about
> >>>>> the
> >>>>>>>>>> current
> >>>>>>>>>>>> state. The ideal solution here would be to define a
> >>> backwards
> >>>>>> looking
> >>>>>>>>>>>> sliding window from the present, but many statistics are
> >>>>> actually
> >>>>>>>>>> very
> >>>>>>>>>>>> hard
> >>>>>>>>>>>> to compute in this model without retaining all the values
> >>>> which
> >>>>>>>>>> would be
> >>>>>>>>>>>> hopelessly inefficient. My solution to this is to keep a
> >>>>>> configurable
> >>>>>>>>>>>> number of windows (default is two) and combine them for
> >> the
> >>>>>> estimate.
> >>>>>>>>>>>> So in
> >>>>>>>>>>>> a two sample case depending on when you ask you have
> >>> between
> >>>>> one
> >>>>>> and
> >>>>>>>>>> two
> >>>>>>>>>>>> complete samples worth of data to base the answer off of.
> >>>>>> Provided
> >>>>>>>>>> the
> >>>>>>>>>>>> sample window is large enough to get a valid result this
> >>>>>> satisfies
> >>>>>>>>>> both
> >>>>>>>>>>>> of
> >>>>>>>>>>>> my criteria of incorporating the most recent data and
> >>> having
> >>>>>>>>>> reasonable
> >>>>>>>>>>>> variance at all times.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Another approach is to use an exponential weighting
> >> scheme
> >>> to
> >>>>>> combine
> >>>>>>>>>>>> all
> >>>>>>>>>>>> history but emphasize the recent past. I have not done
> >> this
> >>>> as
> >>>>> it
> >>>>>>>>>> has a
> >>>>>>>>>>>> lot
> >>>>>>>>>>>> of issues for practical operational metrics. I'd be happy
> >>> to
> >>>>>>>>>> elaborate
> >>>>>>>>>>>> on
> >>>>>>>>>>>> this if anyone cares...
> >>>>>>>>>>>>
> >>>>>>>>>>>> The window size for metrics has a global default which
> >> can
> >>> be
> >>>>>>>>>>>> overridden at
> >>>>>>>>>>>> either the sensor or individual metric level.
> >>>>>>>>>>>>
> >>>>>>>>>>>> In addition to these time series values the user can
> >>> directly
> >>>>>> expose
> >>>>>>>>>>>> some
> >>>>>>>>>>>> method of their choosing JMX-style by implementing the
> >>>>> Measurable
> >>>>>>>>>>>> interface
> >>>>>>>>>>>> and registering that value. E.g.
> >>>>>>>>>>>> metrics.addMetric("my.metric", new Measurable() {
> >>>>>>>>>>>> public double measure(MetricConfg config, long now) {
> >>>>>>>>>>>> return this.calculateValueToExpose();
> >>>>>>>>>>>> }
> >>>>>>>>>>>> });
> >>>>>>>>>>>> This is useful for exposing things like the accumulator
> >>> free
> >>>>>> memory.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The set of metrics is extensible, new metrics can be
> >> added
> >>> by
> >>>>>> just
> >>>>>>>>>>>> implementing the appropriate interfaces and registering
> >>> with
> >>>> a
> >>>>>>>>>> sensor. I
> >>>>>>>>>>>> implement the following metrics:
> >>>>>>>>>>>> total - the sum of all values from the given sensor
> >>>>>>>>>>>> count - a windowed count of values from the sensor
> >>>>>>>>>>>> avg - the sample average within the windows
> >>>>>>>>>>>> max - the max over the windows
> >>>>>>>>>>>> min - the min over the windows
> >>>>>>>>>>>> rate - the rate in the windows (e.g. the total or count
> >>>>>> divided by
> >>>>>>>>>> the
> >>>>>>>>>>>> ellapsed time)
> >>>>>>>>>>>> percentiles - a collection of percentiles computed over
> >>> the
> >>>>>> window
> >>>>>>>>>>>>
> >>>>>>>>>>>> My approach to percentiles is a little different from the
> >>>>> yammer
> >>>>>>>>>> metrics
> >>>>>>>>>>>> package. My complaint about the yammer metrics approach
> >> is
> >>>> that
> >>>>>> it
> >>>>>>>>>> uses
> >>>>>>>>>>>> rather expensive sampling and uses kind of a lot of
> >> memory
> >>> to
> >>>>>> get a
> >>>>>>>>>>>> reasonable sample. This is problematic for per-topic
> >>>>>> measurements.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Instead I use a fixed range for the histogram (e.g. 0.0
> >> to
> >>>>>> 30000.0)
> >>>>>>>>>>>> which
> >>>>>>>>>>>> directly allows you to specify the desired memory use.
> >> Any
> >>>>> value
> >>>>>>>>>> below
> >>>>>>>>>>>> the
> >>>>>>>>>>>> minimum is recorded as -Infinity and any value above the
> >>>>> maximum
> >>>>>> as
> >>>>>>>>>>>> +Infinity. I think this is okay as all metrics have an
> >>>> expected
> >>>>>> range
> >>>>>>>>>>>> except for latency which can be arbitrarily large, but
> >> for
> >>>> very
> >>>>>> high
> >>>>>>>>>>>> latency there is no need to model it exactly (e.g. 30
> >>>> seconds +
> >>>>>>>>>> really
> >>>>>>>>>>>> is
> >>>>>>>>>>>> effectively infinite). Within the range values are
> >> recorded
> >>>> in
> >>>>>>>>>> buckets
> >>>>>>>>>>>> which can be either fixed width or increasing width. The
> >>>>>> increasing
> >>>>>>>>>>>> width
> >>>>>>>>>>>> is analogous to the idea of significant figures, that is
> >> if
> >>>>> your
> >>>>>>>>>> value
> >>>>>>>>>>>> is
> >>>>>>>>>>>> in the range 0-10 you might want to be accurate to within
> >>>> 1ms,
> >>>>>> but if
> >>>>>>>>>>>> it is
> >>>>>>>>>>>> 20000 there is no need to be so accurate. I implemented a
> >>>>> linear
> >>>>>>>>>> bucket
> >>>>>>>>>>>> size where the Nth bucket has width proportional to N. An
> >>>>>> exponential
> >>>>>>>>>>>> bucket size would also be sensible and could likely be
> >>>> derived
> >>>>>>>>>> directly
> >>>>>>>>>>>> from the floating point representation of a the value.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'd like to get some feedback on this metrics code and
> >>> make a
> >>>>>>>>>> decision
> >>>>>>>>>>>> on
> >>>>>>>>>>>> whether we want to use it before I actually go ahead and
> >>> add
> >>>>> all
> >>>>>> the
> >>>>>>>>>>>> instrumentation in the code (otherwise I'll have to redo
> >> it
> >>>> if
> >>>>> we
> >>>>>>>>>> switch
> >>>>>>>>>>>> approaches). So the next topic of discussion will be
> >> which
> >>>>> actual
> >>>>>>>>>>>> metrics
> >>>>>>>>>>>> to add.
> >>>>>>>>>>>>
> >>>>>>>>>>>> -Jay
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
