Hi all, There was consensus at this week's IRC meeting that the best thing to do here is simply to require a release note when metric names change. [0]
I have a patch up to add this to the existing metrics documentation, so feel free to voice any concerns there: https://review.openstack.org/418589 Thanks, Mario [0] http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-3/%23openstack-meeting-3.2017-01-09.log.html#t2017-01-09T17:26:33 On Tue, Jan 3, 2017 at 5:11 PM, Mario Villaplana <mario.villapl...@gmail.com> wrote: > Hi all, > > Recently, Ruby found a patch that modifies the name of a metric > emitted by ironic. [0] After some IRC discussion, we realized that > there is no real deprecation policy regarding changing metric names. > [1] > > For anyone not familiar with this feature, ironic has the capability > to emit various metrics to supported backends using metrics support in > ironic-lib. [2] Currently, the only supported backend is statsd. Most > (all?) metrics currently in ironic are implemented as function > decorators, like the following: > > @METRICS.timer('my.module.MyClass.my_method') > def my_method(self): > ... > > This will send a time series datapoint to statsd which will be stored > with the current epoch timestamp, the name of the metric > ('my.module.MyClass.my_method'), and the amount of time the method > took to finish. > > The primary use case for this that I'm familiar with is generating > graphs with Graphite/Grafana to get a granular look at performance > over time. With Graphite/Grafana, operators can also create graphs > with wildcard matches. For example, a graph that matches on > ironic.conductor.*.* will contain metrics for all methods emitted by > modules in the ironic/conductor subdirectory. Each metric will appear > separately as a line on the same graph by default, if I remember > correctly. > > I did some limited research into the way other OpenStack projects emit > metrics to statsd. I was only able to find one example in a short > amount of time - Swift. [3] Swift seems to document each metric > emitted with a short description of what the metric represents, but it > doesn't guarantee anything at all about the naming or semantics of > metrics. > > I'd like to solicit the opinion of the community, especially operators > who use this feature, for what a good deprecation policy for metric > names should be. > > As a former operator who used a downstream implementation very similar > to the upstream version in production, my recommendation is as > follows: > > 1. Document the metric name as well as what the metric represents in > the deploy docs, for each metric [2] > 2. Guarantee to operators that the docs will be up to date, but don't > guarantee that the metric name won't change without warning between > deploys > 3. Maybe document best practices for using metrics in a stable manner. > Things like using wildcards instead of keying off of specific metric > names, checking documentation for critical changes before deploys, > etc. > > My reasoning for this is that it's hard to guarantee that a function > name won't change (or be completely removed) in between releases. > Since operators can use wildcards to match on metrics, it won't take > too long to notice any changes, even without staying up-to-date on the > documentation. One alternative that was suggested previously - keeping > both prior and new metric names for some deprecation period - won't > solve for the case where the function is removed. Additionally, that > would unnecessarily increase the amount of storage required for > metrics. In my experience, metric storage can be quite expensive. > There's a calculator for storage requirements for Whisper, one of the > storage backends used with Graphite, that can illustrate this. [4] > > I haven't yet scoped the documentation work, but I'm curious about > feedback on this proposal or alternative suggestions from people who > use the feature. > > Thank you! > > Mario > > [0] https://review.openstack.org/#/c/412339 > [1] > http://eavesdrop.openstack.org/irclogs/%23openstack-ironic/%23openstack-ironic.2016-12-19.log.html#t2016-12-19T16:16:11 > [2] http://docs.openstack.org/developer/ironic/deploy/metrics.html > [3] > http://docs.openstack.org/developer/swift/admin_guide.html#reporting-metrics-to-statsd > [4] http://m30m.github.io/whisper-calculator/ __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
