Hello Chan,
What client library are you using to communicate with Riak, and, are you
using protocol buffers or HTTP?
--
Luke Bakken
CSE
lbak...@basho.com
On Tue, Oct 15, 2013 at 8:31 PM, 성동찬_Chan <c...@kakao.com> wrote:
> Hi~!
>
> I'm testing riak as a cache server with 6 server cluster, 2 hours
> expiration time.
> I gave traffic to cluster about 40,000 quries per second from 12hours ago.
> From start point, there was no problem, but after 8hours fetch method
> returns fail with "null point excpetion".
> I change nothing while test operation.
>
> Below, you can see "riak-admin status" and riak-admin top" result.
> Could show me what cause this and how to solve this problem?
>
> Thank you.
> Chan.
>
> *riak-admin status*
>
> -------------------------------------------------------------------------------------------------------------------------------
> riak_kv_stat_ts : 1381893196
> vnode_gets : 1301374
> vnode_gets_total : 1488219135
> vnode_puts : 650670
> vnode_puts_total : 273164708
> vnode_index_reads : 0
> vnode_index_reads_total : 0
> vnode_index_writes : 650670
> vnode_index_writes_total : 273164708
> vnode_index_writes_postings : 0
> vnode_index_writes_postings_total : 0
> vnode_index_deletes : 0
> vnode_index_deletes_total : 0
> vnode_index_deletes_postings : 0
> vnode_index_deletes_postings_total : 0
> node_gets : 532128
> node_gets_total : 605453222
> node_get_fsm_siblings_mean : 0
> node_get_fsm_siblings_median : 0
> node_get_fsm_siblings_95 : 0
> node_get_fsm_siblings_99 : 0
> node_get_fsm_siblings_100 : 0
> node_get_fsm_objsize_mean : 0
> node_get_fsm_objsize_median : 0
> node_get_fsm_objsize_95 : 0
> node_get_fsm_objsize_99 : 0
> node_get_fsm_objsize_100 : 0
> node_get_fsm_time_mean : 476
> node_get_fsm_time_median : 467
> node_get_fsm_time_95 : 865
> node_get_fsm_time_99 : 1119
> node_get_fsm_time_100 : 3213
> node_puts : 226235
> node_puts_total : 95080556
> node_put_fsm_time_mean : 1179
> node_put_fsm_time_median : 1116
> node_put_fsm_time_95 : 1806
> node_put_fsm_time_99 : 2337
> node_put_fsm_time_100 : 10206
> read_repairs : 0
> read_repairs_total : 2697556
> coord_redirs_total : 58178410
> executing_mappers : 0
> precommit_fail : 0
> postcommit_fail : 0
> index_fsm_create : 0
> index_fsm_create_error : 0
> index_fsm_active : 0
> list_fsm_create : 0
> list_fsm_create_error : 0
> list_fsm_active : 0
> pbc_active : 29
> pbc_connects : 1
> pbc_connects_total : 2284
> node_get_fsm_active : 22
> node_get_fsm_active_60s : 527810
> node_get_fsm_in_rate : 8562
> node_get_fsm_out_rate : 8560
> node_get_fsm_rejected : 0
> node_get_fsm_rejected_60s : 0
> node_get_fsm_rejected_total : 0
> node_put_fsm_active : 5
> node_put_fsm_active_60s : 368997
> node_put_fsm_in_rate : 6130
> node_put_fsm_out_rate : 6128
> node_put_fsm_rejected : 0
> node_put_fsm_rejected_60s : 0
> node_put_fsm_rejected_total : 0
> read_repairs_primary_outofdate_one : 0
> read_repairs_primary_outofdate_count : 126588
> read_repairs_primary_notfound_one : 0
> read_repairs_primary_notfound_count : 4169558
> leveldb_read_block_error : undefined
> riak_pipe_stat_ts : 1381893196
> pipeline_active : 0
> pipeline_create_count : 0
> pipeline_create_one : 0
> pipeline_create_error_count : 0
> pipeline_create_error_one : 0
> cpu_nprocs : 361
> cpu_avg1 : 2068
> cpu_avg5 : 2045
> cpu_avg15 : 2028
> mem_total : 16799096832
> mem_allocated : 4083048448
> disk : [{"/",279713876,2},{"/dev/shm",8202684,0}]
> nodename : 'c...@fox656.kr2.iwilab.com'
> connected_nodes : ['c...@fox136.kr2.iwilab.com','
> c...@fox167.kr2.iwilab.com',
> 'c...@fox655.kr2.iwilab.com','
> c...@fox654.kr2.iwilab.com',
> 'c...@fox168.kr2.iwilab.com']
> sys_driver_version : <<"2.0">>
> sys_global_heaps_size : 0
> sys_heap_type : private
> sys_logical_processors : 8
> sys_otp_release : <<"R15B01">>
> sys_process_count : 566
> sys_smp_support : true
> sys_system_version : <<"Erlang R15B01 (erts-5.9.1) [source] [64-bit]
> [smp:8:8] [async-threads:128] [kernel-poll:true]">>
> sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
> sys_threads_enabled : true
> sys_thread_pool_size : 128
> sys_wordsize : 8
> ring_members : ['c...@fox136.kr2.iwilab.com','c...@fox167.kr2.iwilab.com',
> 'c...@fox168.kr2.iwilab.com','c...@fox654.kr2.iwilab.com',
> 'c...@fox655.kr2.iwilab.com','c...@fox656.kr2.iwilab.com']
> ring_num_partitions : 32
> ring_ownership : <<"[{'c...@fox656.kr2.iwilab.com',5},\n {'
> c...@fox655.kr2.iwilab.com',5},\n {'c...@fox654.kr2.iwilab.com',5},\n {'
> c...@fox168.kr2.iwilab.com',5},\n {'c...@fox167.kr2.iwilab.com',6},\n {'
> c...@fox136.kr2.iwilab.com',6}]">>
> ring_creation_size : 32
> storage_backend : riak_kv_multi_backend
> erlydtl_version : <<"0.7.0">>
> riak_control_version : <<"1.4.2-0-g2f08324">>
> cluster_info_version : <<"1.2.4">>
> riak_search_version : <<"1.4.2-0-ge31368d">>
> merge_index_version : <<"1.3.1">>
> riak_kv_version : <<"1.4.2-0-g61ac9d8">>
> sidejob_version : <<"0.2.0">>
> riak_api_version : <<"1.4.2-0-ga7e00e2">>
> riak_pipe_version : <<"1.4.2-0-g59d1bae">>
> riak_core_version : <<"1.4.2">>
> bitcask_version : <<"1.6.3">>
> basho_stats_version : <<"1.0.3">>
> webmachine_version : <<"1.10.4-0-gfcff795">>
> mochiweb_version : <<"1.5.1p6">>
> inets_version : <<"5.9">>
> erlang_js_version : <<"1.2.2">>
> runtime_tools_version : <<"1.8.8">>
> os_mon_version : <<"2.2.9">>
> riak_sysmon_version : <<"1.1.3">>
> ssl_version : <<"5.0.1">>
> public_key_version : <<"0.15">>
> crypto_version : <<"2.1">>
> sasl_version : <<"2.2.1">>
> lager_version : <<"2.0.0">>
> compiler_version : <<"4.8.1">>
> syntax_tools_version : <<"1.6.8">>
> stdlib_version : <<"1.18.1">>
> kernel_version : <<"2.15.1">>
> memory_total : 333983032
> memory_processes : 10197917
> memory_processes_used : 9925151
> memory_system : 323785115
> memory_atom : 504409
> memory_atom_used : 485838
> memory_binary : 479552
> memory_code : 10516587
> memory_ets : 227826744
>
> *riak-admin top*
>
> -------------------------------------------------------------------------------------------------------------------------------
> Pid Name or Initial Func Time Reds
> Memory MsgQ Current Function
>
> -------------------------------------------------------------------------------------------------------------------------------
> <6210.632.0> proc_lib:init_p/5 '-' 8916284
> 21744 1 riak_core_vnode:continue/2
> <6210.631.0> proc_lib:init_p/5 '-' 8890232
> 21696 0 gen_fsm:loop/7
> <6210.565.0> proc_lib:init_p/5 '-' 8881011
> 8920 0 gen_fsm:loop/7
> <6210.511.0> proc_lib:init_p/5 '-' 8795742
> 21728 1 gen_fsm:loop/7
> <6210.633.0> proc_lib:init_p/5 '-' 8791178
> 21696 0 gen_fsm:loop/7
> <6210.337.0> riak_kv_stat_sj_8 '-' 4292821
> 3856 0 gen_server:loop/6
> <6210.331.0> riak_kv_stat_sj_2 '-' 3583242
> 3856 0 gen_server:loop/6
> <6210.332.0> riak_kv_stat_sj_3 '-' 3374842
> 5720 0 gen_server:loop/6
> <6210.336.0> riak_kv_stat_sj_7 '-' 3362782
> 5720 0 gen_server:loop/6
> <6210.334.0> riak_kv_stat_sj_5 '-' 3328089
> 3856 0 gen_server:loop/6
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
