Re: fsm.get time increase on a less busy system

2013-12-17 Thread kzhang
Hi Sean, To answer your questions -- yeah, the cluster is read heavy, gets:puts ratio is ~80:1. In the past day, node gets averages 1.4k and node puts averages 20. -- View this message in context: http://riak-users.197444.n3.nabble.com/fsm-get-time-increase-on-a-less-busy-system-tp4030107p403

Re: fsm.get time increase on a less busy system

2013-12-17 Thread Luke Bakken
Since you're on 1.4, could you please run "riak-debug" on every node in your cluster? Please mail me directly and we'll come up with a way to get them delivered. Could you please also provide a description of the cluster's typical workload? Any monitoring graphs or other statistics would be helpfu

Re: fsm.get time increase on a less busy system

2013-12-17 Thread kzhang
Thanks for the reply. The version is riak 1.4.0. I am looking at node.get.fsm.time.mean and node.get.fsm.time.median. Both increased. -- View this message in context: http://riak-users.197444.n3.nabble.com/fsm-get-time-increase-on-a-less-busy-system-tp4030107p4030112.html Sent from the Riak

Re: fsm.get time increase on a less busy system

2013-12-17 Thread Luke Bakken
Hello, Could you provide the Riak version you're using? Which of the get times are increasing mostly? 95th, 99th, 100th etc? Thanks. -- Luke Bakken CSE lbak...@basho.com On Tue, Dec 17, 2013 at 5:44 AM, kzhang wrote: > we have a 5 node cluster, in the past few days, all the counters indicate a

Re: fsm.get time increase on a less busy system

2013-12-17 Thread Sean Cribbs
How heavy is the traffic against the cluster? Is your workload mostly gets? Which get_fsm stats are growing, is it all of the percentiles or just the max/100%? On Tue, Dec 17, 2013 at 7:44 AM, kzhang wrote: > we have a 5 node cluster, in the past few days, all the counters indicate a > less bus

fsm.get time increase on a less busy system

2013-12-17 Thread kzhang
we have a 5 node cluster, in the past few days, all the counters indicate a less busy system (lower cpu, more available memory, less puts and gets, less network traffic), however get time increased. I am not sure what's going on. I checked the logs, nothing jumped to me. -- View this message i