You may want to monitor - node_put_fsm_rejected (and get, too) to if overload protection rejected put or get requests - node_put_fsm_active, node_put_fsm_active_60s (to see how many active put requests are being served now and in the last minute) there is metrics for get, too. A spike of those metrics can show you a hint that something unusual happens.
A big object can cause a lot of put fsms, since big object can cause head-of-line-blocking inside the cluster. So the problem is not with serving the get/put for the big objects, but the puts/gets coming after that. Like during driving pushing a hard brake won't cause you traffic jam, but the others behind you. On Tue, Jan 26, 2016 at 10:43 PM, Alex Wolfe <a...@activeprospect.com> wrote: > Thanks for your reply. > > We are. We sort of expected an anomaly in the object size, but there was > none. We found the root cause. It was a large number of additions to a > single set. It’s not clear to me which metric reveals that problem, but it > appears as though object size doesn’t. > > Alex > > > > On Jan 26, 2016, at 3:40 PM, Luke Bakken <lbak...@basho.com> wrote: > > > > Hi Alex - > > > > Are you monitoring any of Riak's statistics? Specifically object size > > and sibling count, though all of the stats are useful. > > > > -- > > Luke Bakken > > Engineer > > lbak...@basho.com > > > > On Tue, Jan 26, 2016 at 11:40 AM, Alex Wolfe <a...@activeprospect.com> > wrote: > >> We have a 5 node Riak cluster running 2.1.1. This morning FSM Time > (99th percentile) went way up. We couldn't find any clear signs of trouble > with the cluster and ultimately chose to move the data files and restart > the nodes. Once we started with an empty DB, the FSM Time normalized. But > now it's headed back up again. We're stumped on how to trouble shoot this > issue. Any suggestions? > >> _______________________________________________ > >> riak-users mailing list > >> riak-users@lists.basho.com > >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > -- Richard Jonas Erlang Solutions Hungary Kft Address: Riverpark Office K.32 Közraktár street 32. 3/1. 1093 Budapest Hungary Phone/fax: +36-1-7000-654
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
