Hi Timo. Did you check if there is active anti-entropy activity? [1] That could generate a lot of I/O in the background while building the trees with the data needed for automatic repair or running exchanges to verify replicas are up to date.
[1] http://docs.basho.com/riak/latest/ops/advanced/aae/#Monitoring-AAE On Fri, May 22, 2015 at 10:54 AM, Doug Rohrer <droh...@basho.com> wrote: > Timo: > > The general layout of the information is pretty similar to Top (and > described on the etop docs http://www.erlang.org/doc/man/etop.html and > http://www.erlang.org/doc/apps/observer/etop_ug.html) - the one downside > is that it doesn't profile which function is using the most CPU (the "Reds" > column is Reductions, which is Erlang's measure of work being done by a > process). If you run it for a while longer, with a shorter -interval > setting, you may be able to "catch" what function is being run most > frequently other than the gen_server:loop and gen_event:loop functions > you're seeing, as they are generic OTP functions and not very useful for > diagnostics. > > Doug > > On May 22, 2015, at 9:58 AM, Timo Gatsonides <t...@me.com> wrote: > > Try running `riak-admin top` on the node to see what's going on: > > http://docs.basho.com/riak/1.4.12/ops/running/tools/riak-admin/#top > > This may give you some insight into what that node is doing. > > > Thanks for the quick response. I had already done that, however I don’t > understand the output … > > It is below. > > -Timo > > > > =============================================================================================================================== > '' 13:56:31 > Load: cpu 0 Memory: total 69336 binary > 2717 > procs 664 processes 8439 code > 10720 > runq 0 atom 501 ets > 6227 > > Pid Name or Initial Func Time Reds > Memory MsgQ Current Function > > ------------------------------------------------------------------------------------------------------------------------------- > <6206.105.0> erlang:apply/2 '-' 5091705 > 2600 0 cpu_sup:measurement_server_loop/1 > <6206.181.0> riak_core_vnode_manager '-' 3064188 > 231536 0 gen_server:loop/6 > <6206.580.0> proc_lib:init_p/5 '-' 2805797 > 34472 0 gen_fsm:loop/7 > <6206.715.0> proc_lib:init_p/5 '-' 2485959 > 55144 0 gen_fsm:loop/7 > <6206.709.0> proc_lib:init_p/5 '-' 2445894 > 13896 3 eleveldb:get/3 > <6206.94.0> riak_sysmon_filter '-' 2394053 > 13656 0 gen_server:loop/6 > <6206.714.0> proc_lib:init_p/5 '-' 2177064 > 34472 0 gen_fsm:loop/7 > <6206.710.0> proc_lib:init_p/5 '-' 2153568 > 34472 0 gen_fsm:loop/7 > <6206.3.0> erl_prim_loader '-' 2053533 > 142464 0 erl_prim_loader:loop/3 > <6206.707.0> proc_lib:init_p/5 '-' 2032382 > 34472 0 gen_fsm:loop/7 > > > =============================================================================================================================== > '' 13:56:36 > Load: cpu 7 Memory: total 70450 binary > 2994 > procs 665 processes 9133 code > 10720 > runq 3 atom 501 ets > 6247 > > Pid Name or Initial Func Time Reds > Memory MsgQ Current Function > > ------------------------------------------------------------------------------------------------------------------------------- > <6206.105.0> erlang:apply/2 '-' 8092 > 2600 0 cpu_sup:measurement_server_loop/1 > <6206.580.0> proc_lib:init_p/5 '-' 6710 > 34472 0 eleveldb:write/3 > <6206.94.0> riak_sysmon_filter '-' 6447 > 13656 0 gen_server:loop/6 > <6206.709.0> proc_lib:init_p/5 '-' 6174 > 34472 0 gen_fsm:loop/7 > <6206.28437.0> erlang:apply/2 '-' 3623 > 21440 0 erlang:receive_emd/3 > <6206.707.0> proc_lib:init_p/5 '-' 3089 > 21696 0 gen_fsm:loop/7 > <6206.711.0> proc_lib:init_p/5 '-' 3075 > 34472 0 gen_fsm:loop/7 > <6206.374.0> riak_kv_stat_sj_2 '-' 2679 > 3856 0 gen_server:loop/6 > <6206.714.0> proc_lib:init_p/5 '-' 1955 > 21696 0 gen_fsm:loop/7 > <6206.7.0> application_controller '-' 1779 > 55880 0 gen_server:loop/6 > > > =============================================================================================================================== > '' 13:56:40 > Load: cpu 8 Memory: total 69471 binary > 2827 > procs 664 processes 8469 code > 10720 > runq 0 atom 501 ets > 6251 > > Pid Name or Initial Func Time Reds > Memory MsgQ Current Function > > ------------------------------------------------------------------------------------------------------------------------------- > <6206.105.0> erlang:apply/2 '-' 10113 > 2600 0 cpu_sup:measurement_server_loop/1 > <6206.181.0> riak_core_vnode_manager '-' 9724 > 231536 0 gen_server:loop/6 > <6206.182.0> riak_core_capability '-' 5827 > 54960 0 gen_server:loop/6 > <6206.715.0> proc_lib:init_p/5 '-' 5814 > 34472 0 gen_fsm:loop/7 > <6206.712.0> proc_lib:init_p/5 '-' 5153 > 21696 0 gen_fsm:loop/7 > <6206.94.0> riak_sysmon_filter '-' 4465 > 13656 0 gen_server:loop/6 > <6206.580.0> proc_lib:init_p/5 '-' 3619 > 34472 0 gen_fsm:loop/7 > <6206.376.0> riak_kv_stat_sj_4 '-' 2542 > 3856 0 gen_server:loop/6 > <6206.707.0> proc_lib:init_p/5 '-' 2298 > 34472 0 gen_fsm:loop/7 > <6206.7.0> application_controller '-' 2148 > 55880 0 gen_server:loop/6 > > > =============================================================================================================================== > '' 13:56:45 > Load: cpu 9 Memory: total 69332 binary > 2786 > procs 665 processes 8324 code > 10720 > runq 0 atom 501 ets > 6269 > > Pid Name or Initial Func Time Reds > Memory MsgQ Current Function > > ------------------------------------------------------------------------------------------------------------------------------- > <6206.105.0> erlang:apply/2 '-' 8092 > 2600 0 cpu_sup:measurement_server_loop/1 > <6206.712.0> proc_lib:init_p/5 '-' 5472 > 34472 0 gen_fsm:loop/7 > <6206.94.0> riak_sysmon_filter '-' 4674 > 13656 0 gen_server:loop/6 > <6206.580.0> proc_lib:init_p/5 '-' 3939 > 55144 0 gen_fsm:loop/7 > <6206.161.0> riak_core_ring_manager '-' 3591 > 230096 0 gen_server:loop/6 > <6206.709.0> proc_lib:init_p/5 '-' 2890 > 34472 0 gen_fsm:loop/7 > <6206.376.0> riak_kv_stat_sj_4 '-' 2880 > 5720 0 gen_server:loop/6 > <6206.710.0> proc_lib:init_p/5 '-' 2476 > 34472 0 gen_fsm:loop/7 > <6206.708.0> proc_lib:init_p/5 '-' 2033 > 21696 0 gen_fsm:loop/7 > <6206.639.0> proc_lib:init_p/5 '-' 1997 > 34472 0 gen_fsm:loop/7 > > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
