On 01/02/2014 09:44 PM, Clark Boylan wrote:
<snip>
This is great stuff. Out of curiousity is doing the graphing with
pandas and ES vs graphite so that we can graph things in a more ad hoc
fashion?
So, we need to go to ES for the fingerprints anyway (because that's
where we mine them from), which means we need a way to process ES data
into TimeSeries. In order to calculate frequencies we need largely
equivalent TimeSeries that are base lines for # of jobs run of
particular types. Given that we can get that with an ES query, it
prevents the need of having to have a different data transformation
process to get to the same kind of TimeSeries.
It also lets us bulk query. With 1 ~20second ES query we get all states,
of all jobs, across all queues, over the last 7 days (as well as
information on review). And the transform to slice is super easy because
it's 10s of thousands of records that are dictionaries, which makes for
good input. You'd need to do a bunch of unbinning and transforms to
massage the graphite data to pair with what we have in the fingerprint data.
Eventually having tools to do the same thing with graphite is probe ably
a good thing, largely for other analysis people want to do on that (I
think long term having some data kits for our bulk data to let people
play with it is goodness). I'd just put it after a 1.0 as I think it's
not really needed.
Also, for the dashboard, Kibana3 does a lot more stuff than
Kibana2 which we currently use. I have been meaning to get Kibana3
running alongside Kibana2 and I think it may be able to do multi line
queries (I need to double check that but it has a lot more query and
graphing capability). I think Kibana3 is worth looking into as well
before we go too far down the road of custom UI.
Absolutely. There is a reason that's all the way at the bottom of the
list, and honestly, something I almost didn't put in there. But I
figured we needed to understand the implications of multi line matches
with our current UI, and the fact that they will make some things
better, but discovering those matches will be harder with the existing UI.
If Kibana3 solves it, score. One less thing to do. Because I'd really
like to not be in the business of maintaining a custom web UI just for
discovery of fingerprints.
-Sean
--
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
