Yep. I would *never* use mean when it comes to performance to make any sort of decisions. I prefer to graph all the p99 latencies as well as the max.
Some good reading on the topic: https://bravenewgeek.com/everything-you-know-about-latency-is-wrong/ On Thu, May 30, 2019 at 7:35 AM Chris Lohfink <clohfin...@gmail.com> wrote: > For what it is worth, generally I would recommend just using the mean vs > calculating it yourself. It's a lot easier and averages are meaningless for > anything besides trending anyway (which is really what this is useful for, > finding issues on the larger scale), especially with high volume clusters > so the loss in accuracy kinda moot. Your average for local reads/writes > will almost always be sub millisecond but you might end up having 500 > millisecond requests or worse that the mean will hide. > > Chris > > On Thu, May 30, 2019 at 6:30 AM shalom sagges <shalomsag...@gmail.com> > wrote: > >> Thanks for your replies guys. I really appreciate it. >> >> @Alain, I use Graphite for backend on top of Grafana. But the goal is to >> move from Graphite to Prometheus eventually. >> >> I tried to find a direct way of getting a specific Latency metric in >> average and as Chris pointed out, then Mean value isn't that accurate. >> I do not wish to use the percentile metrics either, but a single latency >> metric like the *"Local read latency" *output in nodetool tablestats. >> Looking at the code of nodetool tablestats, it seems that C* also divides >> *ReadTotalLatency.Count* with *ReadLatency.Count *to get the latency >> result. >> >> So I guess I will have no choice but to run the calculation on my own via >> Graphite: >> >> divideSeries(averageSeries(keepLastValue(nonNegativeDerivative($env.path.to.host.$host.org_apache_cassandra_metrics.Table.$ks.$cf.ReadTotalLatency.Count))),averageSeries(keepLastValue(nonNegativeDerivative($env.path.to.host.$host.org_apache_cassandra_metrics.Table.$ks.$cf.ReadLatency.Count)))) >> >> Does this seem right to you? >> >> Thanks! >> >> On Thu, May 30, 2019 at 12:34 AM Paul Chandler <p...@redshots.com> wrote: >> >>> There are various attributes under >>> org.apache.cassandra.metrics.ClientRequest.Latency.Read these measure the >>> latency in milliseconds >>> >>> Thanks >>> >>> Paul >>> www.redshots.com >>> >>> > On 29 May 2019, at 15:31, shalom sagges <shalomsag...@gmail.com> >>> wrote: >>> > >>> > Hi All, >>> > >>> > I'm creating a dashboard that should collect read/write latency >>> metrics on C* 3.x. >>> > In older versions (e.g. 2.0) I used to divide the total read latency >>> in microseconds with the read count. >>> > >>> > Is there a metric attribute that shows read/write latency without the >>> need to do the math, such as in nodetool tablestats "Local read latency" >>> output? >>> > I saw there's a Mean attribute in >>> org.apache.cassandra.metrics.ReadLatency but I'm not sure this is the right >>> one. >>> > >>> > I'd really appreciate your help on this one. >>> > Thanks! >>> > >>> > >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >>> For additional commands, e-mail: user-h...@cassandra.apache.org >>> >>>
