Hello everyone,
Please lend me a hand here... I'm running a riak cluster of 6 machines
(version 1.4.1).
Suddenly all the nodes in the cluster went down and they are refusing to go
up again. It keeps crashing all the the time, this is just a sample of what
I get when starting a node:
2013-09-20 15:16:14.016 [info] <0.7.0> Application lager started on node '
riak@192.168.20.107'
2013-09-20 15:16:14.024 [info] <0.7.0> Application sasl started on node '
riak@192.168.20.107'
2013-09-20 15:16:14.026 [info] <0.7.0> Application crypto started on node '
riak@192.168.20.107'
2013-09-20 15:16:14.026 [info] <0.7.0> Application public_key started on
node 'riak@192.168.20.107'
2013-09-20 15:16:14.029 [info] <0.7.0> Application ssl started on node '
riak@192.168.20.107'
2013-09-20 15:16:14.031 [info] <0.7.0> Application riak_sysmon started on
node 'riak@192.168.20.107'
2013-09-20 15:16:14.036 [info] <0.7.0> Application os_mon started on node '
riak@192.168.20.107'
2013-09-20 15:16:14.044 [info] <0.7.0> Application runtime_tools started on
node 'riak@192.168.20.107'
2013-09-20 15:16:14.047 [info] <0.7.0> Application erlang_js started on
node 'riak@192.168.20.107'
2013-09-20 15:16:14.058 [info] <0.7.0> Application inets started on node '
riak@192.168.20.107'
2013-09-20 15:16:14.059 [info] <0.7.0> Application mochiweb started on node
'riak@192.168.20.107'
2013-09-20 15:16:14.063 [info] <0.7.0> Application webmachine started on
node 'riak@192.168.20.107'
2013-09-20 15:16:14.064 [info] <0.7.0> Application basho_stats started on
node 'riak@192.168.20.107'
2013-09-20 15:16:14.067 [info] <0.7.0> Application bitcask started on node '
riak@192.168.20.107'
2013-09-20 15:16:14.152 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_core,vnode_routing} = proxy
2013-09-20 15:16:14.160 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_core,staged_joins} = true
2013-09-20 15:16:14.166 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_core,resizable_ring} = true
2013-09-20 15:16:14.172 [info] <0.7.0> Application riak_core started on
node 'riak@192.168.20.107'
2013-09-20 15:16:14.177 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_pipe,trace_format} = ordsets
2013-09-20 15:16:14.270 [info] <0.522.0>@riak_kv_env:doc_env:42 Environment
and OS variables:
2013-09-20 15:16:14.446 [notice] <0.41.0>@lager_app:119 Deprecated
lager_file_backend config detected, please consider updating it
2013-09-20 15:16:14.786 [info] <0.573.0>@riak_core:wait_for_service:470
Waiting for service riak_kv to start (0 seconds)
2013-09-20 15:16:14.821 [info] <0.590.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
(<0.590.0>)
2013-09-20 15:16:14.823 [info] <0.591.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
(<0.591.0>)
2013-09-20 15:16:14.825 [info] <0.592.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
(<0.592.0>)
2013-09-20 15:16:14.826 [info] <0.593.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
(<0.593.0>)
2013-09-20 15:16:14.827 [info] <0.594.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
(<0.594.0>)
2013-09-20 15:16:14.829 [info] <0.595.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
(<0.595.0>)
2013-09-20 15:16:14.831 [info] <0.596.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
(<0.596.0>)
2013-09-20 15:16:14.833 [info] <0.597.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting
(<0.597.0>)
2013-09-20 15:16:14.837 [info] <0.600.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host
starting (<0.600.0>)
2013-09-20 15:16:14.841 [info] <0.601.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host
starting (<0.601.0>)
2013-09-20 15:16:14.845 [info] <0.602.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host
starting (<0.602.0>)
2013-09-20 15:16:14.848 [info] <0.603.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host
starting (<0.603.0>)
2013-09-20 15:16:14.850 [info] <0.605.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host
starting (<0.605.0>)
2013-09-20 15:16:14.852 [info] <0.606.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host
starting (<0.606.0>)
2013-09-20 15:16:14.853 [info] <0.608.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_hook) host starting
(<0.608.0>)
2013-09-20 15:16:14.855 [info] <0.609.0>@riak_kv_js_vm:init:76 Spidermonkey
VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_hook) host starting
(<0.609.0>)
2013-09-20 15:16:14.855 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,vnode_vclocks} = true
2013-09-20 15:16:14.867 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,legacy_keylisting} = false
2013-09-20 15:16:14.873 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,listkeys_backpressure} = true
2013-09-20 15:16:14.884 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,index_backpressure} = true
2013-09-20 15:16:14.889 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,mapred_system} = pipe
2013-09-20 15:16:14.900 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,mapred_2i_pipe} = true
2013-09-20 15:16:14.905 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,anti_entropy} = enabled_v1
2013-09-20 15:16:14.911 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,handoff_data_encoding} = encode_raw
2013-09-20 15:16:14.917 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,object_format} = v1
2013-09-20 15:16:14.927 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,secondary_index_version} = v2
2013-09-20 15:16:14.943 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,vclock_data_encoding} = encode_zlib
2013-09-20 15:16:14.953 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,crdt} = [pncounter]
2013-09-20 15:16:15.011 [warning] <0.6.0> lager_error_logger_h dropped 17
messages in the last second that exceeded the limit of 100 messages/sec
2013-09-20 15:16:15.012 [info]
<0.162.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_control,member_info_version} = v1
2013-09-20 15:16:15.022 [info] <0.7.0> Application riak_control started on
node 'riak@192.168.20.107'
2013-09-20 15:16:15.022 [info] <0.7.0> Application erlydtl started on node '
riak@192.168.20.107'
2013-09-20 15:16:16.013 [error] <0.1209.0> gen_fsm <0.1209.0> in state
started terminated with reason: no function clause matching
riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,...]],...},...]},
undefined) line 851
2013-09-20 15:16:16.013 [error] <0.1209.0> CRASH REPORT Process <0.1209.0>
with 0 neighbours exited with reason: no function clause matching
riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,...]],...},...]},
undefined) line 851 in gen_fsm:terminate/7 line 589
2013-09-20 15:16:16.014 [error] <0.140.0> Supervisor riak_core_vnode_sup
had child undefined started with {riak_core_vnode,start_link,undefined} at
<0.1209.0> exit with reason no function clause matching
riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,...]],...},...]},
undefined) line 851 in context child_terminated
2013-09-20 15:16:16.014 [error] <0.739.0> CRASH REPORT Process
proxy_riak_kv_vnode_376793390874373408599387495934666716005045108736 with 0
neighbours exited with reason:
{{{function_clause,[{riak_kv_vnode,terminate,[{badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,365375409332725729550921208179070754913983135744,359666418561901890026688064301272774368452149248,353957427791078050502454920423474793822921162752,348248437020254210978221776545676813277390176256,342539446249430371453988632667878832731859189760,336830455478606531929755488790080852186328203264,331121464707782692405522344912282871640797216768,...]],...},...]},...],...},...]},...},...}
in gen_server:call/3 line 188
2013-09-20 15:16:16.014 [error] <0.146.0> Supervisor
riak_core_vnode_proxy_sup had child
{riak_kv_vnode,376793390874373408599387495934666716005045108736} started
with riak_core_vnode_proxy:start_link(riak_kv_vnode,
376793390874373408599387495934666716005045108736) at <0.739.0> exit with
reason
{{{function_clause,[{riak_kv_vnode,terminate,[{badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,365375409332725729550921208179070754913983135744,359666418561901890026688064301272774368452149248,353957427791078050502454920423474793822921162752,348248437020254210978221776545676813277390176256,342539446249430371453988632667878832731859189760,336830455478606531929755488790080852186328203264,331121464707782692405522344912282871640797216768,...]],...},...]},...],...},...]},...},...}
in context child_terminated
2013-09-20 15:16:16.015 [error] <0.138.0> Supervisor riak_core_sup had
child riak_core_vnode_manager started with
riak_core_vnode_manager:start_link() at <0.161.0> exit with reason
{{function_clause,[{riak_kv_vnode,terminate,[{badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,365375409332725729550921208179070754913983135744,359666418561901890026688064301272774368452149248,353957427791078050502454920423474793822921162752,348248437020254210978221776545676813277390176256,342539446249430371453988632667878832731859189760,336830455478606531929755488790080852186328203264,331121464707782692405522344912282871640797216768,...]],...},...]},...],...},...]},...}
in context child_terminated
2013-09-20 15:16:24.147 [error] <0.2034.0> gen_fsm <0.2034.0> in state
started terminated with reason: no function clause matching
riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[445301280124259482890185222468242482551416946688,439592289353435643365952078590444502005885960192,...]],...},...]},
undefined) line 851
2013-09-20 15:16:24.147 [error] <0.2034.0> CRASH REPORT Process <0.2034.0>
with 0 neighbours exited with reason: no function clause matching
riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[445301280124259482890185222468242482551416946688,439592289353435643365952078590444502005885960192,...]],...},...]},
undefined) line 851 in gen_fsm:terminate/7 line 589
2013-09-20 15:16:24.147 [error] <0.2019.0> gen_fsm <0.2019.0> in state
started terminated with reason: no function clause matching
riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[34253944624943037145398863266787883273185918976,28544953854119197621165719388989902727654932480,...]],...},...]},
undefined) line 851
2013-09-20 15:16:24.147 [error] <0.2019.0> CRASH REPORT Process <0.2019.0>
with 0 neighbours exited with reason: no function clause matching
riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[34253944624943037145398863266787883273185918976,28544953854119197621165719388989902727654932480,...]],...},...]},
undefined) line 851 in gen_fsm:terminate/7 line 589
2013-09-20 15:16:24.153 [error] <0.2029.0> gen_fsm <0.2029.0> in state
started terminated with reason: no function clause matching
riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[308285501624487334308589769401090949458673270784,302576510853663494784356625523292968913142284288,...]],...},...]},
undefined) line 851
2013-09-20 15:16:24.153 [error] <0.2029.0> CRASH REPORT Process <0.2029.0>
with 0 neighbours exited with reason: no function clause matching
riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[308285501624487334308589769401090949458673270784,302576510853663494784356625523292968913142284288,...]],...},...]},
undefined) line 851 in gen_fsm:terminate/7 line 589
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
