Re: Undiagnosed High FSM Time

2016-01-26 Thread Alex Wolfe
Thanks for your reply. We are. We sort of expected an anomaly in the object size, but there was none. We found the root cause. It was a large number of additions to a single set. It’s not clear to me which metric reveals that problem, but it appears as though object size doesn’t. Alex > On J

Re: Undiagnosed High FSM Time

2016-01-26 Thread Luke Bakken
Hi Alex - Are you monitoring any of Riak's statistics? Specifically object size and sibling count, though all of the stats are useful. -- Luke Bakken Engineer lbak...@basho.com On Tue, Jan 26, 2016 at 11:40 AM, Alex Wolfe wrote: > We have a 5 node Riak cluster running 2.1.1. This morning FSM Ti

Undiagnosed High FSM Time

2016-01-26 Thread Alex Wolfe
We have a 5 node Riak cluster running 2.1.1. This morning FSM Time (99th percentile) went way up. We couldn't find any clear signs of trouble with the cluster and ultimately chose to move the data files and restart the nodes. Once we started with an empty DB, the FSM Time normalized. But now it'