Looks like the nval is set to a binary <<"3">> rather than an integer. Have you
changed it recently and how?
On Sep 20, 2013, at 8:25 AM, Edgar Veiga <edgarmve...@gmail.com> wrote:
> Hello everyone,
>
> Please lend me a hand here... I'm running a riak cluster of 6 machines
> (version 1.4.1).
>
> Suddenly all the nodes in the cluster went down and they are refusing to go
> up again. It keeps crashing all the the time, this is just a sample of what I
> get when starting a node:
>
> 2013-09-20 15:16:14.016 [info] <0.7.0> Application lager started on node
> 'riak@192.168.20.107'
> 2013-09-20 15:16:14.024 [info] <0.7.0> Application sasl started on node
> 'riak@192.168.20.107'
> 2013-09-20 15:16:14.026 [info] <0.7.0> Application crypto started on node
> 'riak@192.168.20.107'
> 2013-09-20 15:16:14.026 [info] <0.7.0> Application public_key started on node
> 'riak@192.168.20.107'
> 2013-09-20 15:16:14.029 [info] <0.7.0> Application ssl started on node
> 'riak@192.168.20.107'
> 2013-09-20 15:16:14.031 [info] <0.7.0> Application riak_sysmon started on
> node 'riak@192.168.20.107'
> 2013-09-20 15:16:14.036 [info] <0.7.0> Application os_mon started on node
> 'riak@192.168.20.107'
> 2013-09-20 15:16:14.044 [info] <0.7.0> Application runtime_tools started on
> node 'riak@192.168.20.107'
> 2013-09-20 15:16:14.047 [info] <0.7.0> Application erlang_js started on node
> 'riak@192.168.20.107'
> 2013-09-20 15:16:14.058 [info] <0.7.0> Application inets started on node
> 'riak@192.168.20.107'
> 2013-09-20 15:16:14.059 [info] <0.7.0> Application mochiweb started on node
> 'riak@192.168.20.107'
> 2013-09-20 15:16:14.063 [info] <0.7.0> Application webmachine started on node
> 'riak@192.168.20.107'
> 2013-09-20 15:16:14.064 [info] <0.7.0> Application basho_stats started on
> node 'riak@192.168.20.107'
> 2013-09-20 15:16:14.067 [info] <0.7.0> Application bitcask started on node
> 'riak@192.168.20.107'
> 2013-09-20 15:16:14.152 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_core,vnode_routing} = proxy
> 2013-09-20 15:16:14.160 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_core,staged_joins} = true
> 2013-09-20 15:16:14.166 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_core,resizable_ring} = true
> 2013-09-20 15:16:14.172 [info] <0.7.0> Application riak_core started on node
> 'riak@192.168.20.107'
> 2013-09-20 15:16:14.177 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_pipe,trace_format} = ordsets
> 2013-09-20 15:16:14.270 [info] <0.522.0>@riak_kv_env:doc_env:42 Environment
> and OS variables:
> 2013-09-20 15:16:14.446 [notice] <0.41.0>@lager_app:119 Deprecated
> lager_file_backend config detected, please consider updating it
> 2013-09-20 15:16:14.786 [info] <0.573.0>@riak_core:wait_for_service:470
> Waiting for service riak_kv to start (0 seconds)
> 2013-09-20 15:16:14.821 [info] <0.590.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
> (<0.590.0>)
> 2013-09-20 15:16:14.823 [info] <0.591.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
> (<0.591.0>)
> 2013-09-20 15:16:14.825 [info] <0.592.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
> (<0.592.0>)
> 2013-09-20 15:16:14.826 [info] <0.593.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
> (<0.593.0>)
> 2013-09-20 15:16:14.827 [info] <0.594.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
> (<0.594.0>)
> 2013-09-20 15:16:14.829 [info] <0.595.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
> (<0.595.0>)
> 2013-09-20 15:16:14.831 [info] <0.596.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
> (<0.596.0>)
> 2013-09-20 15:16:14.833 [info] <0.597.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
> (<0.597.0>)
> 2013-09-20 15:16:14.837 [info] <0.600.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting
> (<0.600.0>)
> 2013-09-20 15:16:14.841 [info] <0.601.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting
> (<0.601.0>)
> 2013-09-20 15:16:14.845 [info] <0.602.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting
> (<0.602.0>)
> 2013-09-20 15:16:14.848 [info] <0.603.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting
> (<0.603.0>)
> 2013-09-20 15:16:14.850 [info] <0.605.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting
> (<0.605.0>)
> 2013-09-20 15:16:14.852 [info] <0.606.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting
> (<0.606.0>)
> 2013-09-20 15:16:14.853 [info] <0.608.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_hook) host starting
> (<0.608.0>)
> 2013-09-20 15:16:14.855 [info] <0.609.0>@riak_kv_js_vm:init:76 Spidermonkey
> VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_hook) host starting
> (<0.609.0>)
> 2013-09-20 15:16:14.855 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_kv,vnode_vclocks} = true
> 2013-09-20 15:16:14.867 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_kv,legacy_keylisting} = false
> 2013-09-20 15:16:14.873 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_kv,listkeys_backpressure} = true
> 2013-09-20 15:16:14.884 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_kv,index_backpressure} = true
> 2013-09-20 15:16:14.889 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_kv,mapred_system} = pipe
> 2013-09-20 15:16:14.900 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_kv,mapred_2i_pipe} = true
> 2013-09-20 15:16:14.905 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_kv,anti_entropy} = enabled_v1
> 2013-09-20 15:16:14.911 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_kv,handoff_data_encoding} = encode_raw
> 2013-09-20 15:16:14.917 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_kv,object_format} = v1
> 2013-09-20 15:16:14.927 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_kv,secondary_index_version} = v2
> 2013-09-20 15:16:14.943 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_kv,vclock_data_encoding} = encode_zlib
> 2013-09-20 15:16:14.953 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_kv,crdt} = [pncounter]
> 2013-09-20 15:16:15.011 [warning] <0.6.0> lager_error_logger_h dropped 17
> messages in the last second that exceeded the limit of 100 messages/sec
> 2013-09-20 15:16:15.012 [info]
> <0.162.0>@riak_core_capability:process_capability_changes:530 New capability:
> {riak_control,member_info_version} = v1
> 2013-09-20 15:16:15.022 [info] <0.7.0> Application riak_control started on
> node 'riak@192.168.20.107'
> 2013-09-20 15:16:15.022 [info] <0.7.0> Application erlydtl started on node
> 'riak@192.168.20.107'
> 2013-09-20 15:16:16.013 [error] <0.1209.0> gen_fsm <0.1209.0> in state
> started terminated with reason: no function clause matching
> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,...]],...},...]},
> undefined) line 851
> 2013-09-20 15:16:16.013 [error] <0.1209.0> CRASH REPORT Process <0.1209.0>
> with 0 neighbours exited with reason: no function clause matching
> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,...]],...},...]},
> undefined) line 851 in gen_fsm:terminate/7 line 589
> 2013-09-20 15:16:16.014 [error] <0.140.0> Supervisor riak_core_vnode_sup had
> child undefined started with {riak_core_vnode,start_link,undefined} at
> <0.1209.0> exit with reason no function clause matching
> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,...]],...},...]},
> undefined) line 851 in context child_terminated
> 2013-09-20 15:16:16.014 [error] <0.739.0> CRASH REPORT Process
> proxy_riak_kv_vnode_376793390874373408599387495934666716005045108736 with 0
> neighbours exited with reason:
> {{{function_clause,[{riak_kv_vnode,terminate,[{badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,365375409332725729550921208179070754913983135744,359666418561901890026688064301272774368452149248,353957427791078050502454920423474793822921162752,348248437020254210978221776545676813277390176256,342539446249430371453988632667878832731859189760,336830455478606531929755488790080852186328203264,331121464707782692405522344912282871640797216768,...]],...},...]},...],...},...]},...},...}
> in gen_server:call/3 line 188
> 2013-09-20 15:16:16.014 [error] <0.146.0> Supervisor
> riak_core_vnode_proxy_sup had child
> {riak_kv_vnode,376793390874373408599387495934666716005045108736} started with
> riak_core_vnode_proxy:start_link(riak_kv_vnode,
> 376793390874373408599387495934666716005045108736) at <0.739.0> exit with
> reason
> {{{function_clause,[{riak_kv_vnode,terminate,[{badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,365375409332725729550921208179070754913983135744,359666418561901890026688064301272774368452149248,353957427791078050502454920423474793822921162752,348248437020254210978221776545676813277390176256,342539446249430371453988632667878832731859189760,336830455478606531929755488790080852186328203264,331121464707782692405522344912282871640797216768,...]],...},...]},...],...},...]},...},...}
> in context child_terminated
> 2013-09-20 15:16:16.015 [error] <0.138.0> Supervisor riak_core_sup had child
> riak_core_vnode_manager started with riak_core_vnode_manager:start_link() at
> <0.161.0> exit with reason
> {{function_clause,[{riak_kv_vnode,terminate,[{badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,365375409332725729550921208179070754913983135744,359666418561901890026688064301272774368452149248,353957427791078050502454920423474793822921162752,348248437020254210978221776545676813277390176256,342539446249430371453988632667878832731859189760,336830455478606531929755488790080852186328203264,331121464707782692405522344912282871640797216768,...]],...},...]},...],...},...]},...}
> in context child_terminated
> 2013-09-20 15:16:24.147 [error] <0.2034.0> gen_fsm <0.2034.0> in state
> started terminated with reason: no function clause matching
> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[445301280124259482890185222468242482551416946688,439592289353435643365952078590444502005885960192,...]],...},...]},
> undefined) line 851
> 2013-09-20 15:16:24.147 [error] <0.2034.0> CRASH REPORT Process <0.2034.0>
> with 0 neighbours exited with reason: no function clause matching
> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[445301280124259482890185222468242482551416946688,439592289353435643365952078590444502005885960192,...]],...},...]},
> undefined) line 851 in gen_fsm:terminate/7 line 589
> 2013-09-20 15:16:24.147 [error] <0.2019.0> gen_fsm <0.2019.0> in state
> started terminated with reason: no function clause matching
> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[34253944624943037145398863266787883273185918976,28544953854119197621165719388989902727654932480,...]],...},...]},
> undefined) line 851
> 2013-09-20 15:16:24.147 [error] <0.2019.0> CRASH REPORT Process <0.2019.0>
> with 0 neighbours exited with reason: no function clause matching
> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[34253944624943037145398863266787883273185918976,28544953854119197621165719388989902727654932480,...]],...},...]},
> undefined) line 851 in gen_fsm:terminate/7 line 589
> 2013-09-20 15:16:24.153 [error] <0.2029.0> gen_fsm <0.2029.0> in state
> started terminated with reason: no function clause matching
> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[308285501624487334308589769401090949458673270784,302576510853663494784356625523292968913142284288,...]],...},...]},
> undefined) line 851
> 2013-09-20 15:16:24.153 [error] <0.2029.0> CRASH REPORT Process <0.2029.0>
> with 0 neighbours exited with reason: no function clause matching
> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[308285501624487334308589769401090949458673270784,302576510853663494784356625523292968913142284288,...]],...},...]},
> undefined) line 851 in gen_fsm:terminate/7 line 589
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
