I was preparing a mail making that suggestion! All the instances in my 6 machines cluster went down because of this :S
Thanks again, keep up the good work! On 20 September 2013 16:30, Jon Meredith <jmered...@basho.com> wrote: > Hi Edgar, > > I'm pleased to hear you've resolved the issue. > https://github.com/basho/riak_kv/issues/666 will track adding some > validation code to protect against similar incidents. > > Jon > > > On Fri, Sep 20, 2013 at 8:59 AM, Edgar Veiga <edgarmve...@gmail.com>wrote: > >> Problem solved. >> >> The n_val = "3" caused the crash! I had a window of time while starting a >> node to send a new PUT command and restore the correct value. >> >> Best regards, thanks Jon >> >> >> On 20 September 2013 15:42, Edgar Veiga <edgarmve...@gmail.com> wrote: >> >>> Yes I did, via CURL command: >>> >>> curl -v -XPUT http://dat7:8098/riak/visitors -H "Content-Type: >>> application/json" -d '{"props":{"n_val":"3"}}' >>> >>> How can I update the n_val of an already existing bucket, while the >>> instance is down? >>> >>> >>> On 20 September 2013 15:32, Jon Meredith <jmered...@basho.com> wrote: >>> >>>> Looks like the nval is set to a binary <<"3">> rather than an integer. >>>> Have you changed it recently and how? >>>> >>>> On Sep 20, 2013, at 8:25 AM, Edgar Veiga <edgarmve...@gmail.com> wrote: >>>> >>>> Hello everyone, >>>> >>>> Please lend me a hand here... I'm running a riak cluster of 6 machines >>>> (version 1.4.1). >>>> >>>> Suddenly all the nodes in the cluster went down and they are refusing >>>> to go up again. It keeps crashing all the the time, this is just a sample >>>> of what I get when starting a node: >>>> >>>> 2013-09-20 15:16:14.016 [info] <0.7.0> Application lager started on >>>> node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:14.024 [info] <0.7.0> Application sasl started on node >>>> 'riak@192.168.20.107' >>>> 2013-09-20 15:16:14.026 [info] <0.7.0> Application crypto started on >>>> node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:14.026 [info] <0.7.0> Application public_key started >>>> on node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:14.029 [info] <0.7.0> Application ssl started on node ' >>>> riak@192.168.20.107' >>>> 2013-09-20 15:16:14.031 [info] <0.7.0> Application riak_sysmon started >>>> on node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:14.036 [info] <0.7.0> Application os_mon started on >>>> node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:14.044 [info] <0.7.0> Application runtime_tools >>>> started on node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:14.047 [info] <0.7.0> Application erlang_js started on >>>> node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:14.058 [info] <0.7.0> Application inets started on >>>> node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:14.059 [info] <0.7.0> Application mochiweb started on >>>> node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:14.063 [info] <0.7.0> Application webmachine started >>>> on node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:14.064 [info] <0.7.0> Application basho_stats started >>>> on node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:14.067 [info] <0.7.0> Application bitcask started on >>>> node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:14.152 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_core,vnode_routing} = proxy >>>> 2013-09-20 15:16:14.160 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_core,staged_joins} = true >>>> 2013-09-20 15:16:14.166 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_core,resizable_ring} = true >>>> 2013-09-20 15:16:14.172 [info] <0.7.0> Application riak_core started on >>>> node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:14.177 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_pipe,trace_format} = ordsets >>>> 2013-09-20 15:16:14.270 [info] <0.522.0>@riak_kv_env:doc_env:42 >>>> Environment and OS variables: >>>> 2013-09-20 15:16:14.446 [notice] <0.41.0>@lager_app:119 Deprecated >>>> lager_file_backend config detected, please consider updating it >>>> 2013-09-20 15:16:14.786 [info] <0.573.0>@riak_core:wait_for_service:470 >>>> Waiting for service riak_kv to start (0 seconds) >>>> 2013-09-20 15:16:14.821 [info] <0.590.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) >>>> host starting (<0.590.0>) >>>> 2013-09-20 15:16:14.823 [info] <0.591.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) >>>> host starting (<0.591.0>) >>>> 2013-09-20 15:16:14.825 [info] <0.592.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) >>>> host starting (<0.592.0>) >>>> 2013-09-20 15:16:14.826 [info] <0.593.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) >>>> host starting (<0.593.0>) >>>> 2013-09-20 15:16:14.827 [info] <0.594.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) >>>> host starting (<0.594.0>) >>>> 2013-09-20 15:16:14.829 [info] <0.595.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) >>>> host starting (<0.595.0>) >>>> 2013-09-20 15:16:14.831 [info] <0.596.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) >>>> host starting (<0.596.0>) >>>> 2013-09-20 15:16:14.833 [info] <0.597.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) >>>> host starting (<0.597.0>) >>>> 2013-09-20 15:16:14.837 [info] <0.600.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: >>>> riak_kv_js_reduce) host starting (<0.600.0>) >>>> 2013-09-20 15:16:14.841 [info] <0.601.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: >>>> riak_kv_js_reduce) host starting (<0.601.0>) >>>> 2013-09-20 15:16:14.845 [info] <0.602.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: >>>> riak_kv_js_reduce) host starting (<0.602.0>) >>>> 2013-09-20 15:16:14.848 [info] <0.603.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: >>>> riak_kv_js_reduce) host starting (<0.603.0>) >>>> 2013-09-20 15:16:14.850 [info] <0.605.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: >>>> riak_kv_js_reduce) host starting (<0.605.0>) >>>> 2013-09-20 15:16:14.852 [info] <0.606.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: >>>> riak_kv_js_reduce) host starting (<0.606.0>) >>>> 2013-09-20 15:16:14.853 [info] <0.608.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_hook) >>>> host starting (<0.608.0>) >>>> 2013-09-20 15:16:14.855 [info] <0.609.0>@riak_kv_js_vm:init:76 >>>> Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_hook) >>>> host starting (<0.609.0>) >>>> 2013-09-20 15:16:14.855 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_kv,vnode_vclocks} = true >>>> 2013-09-20 15:16:14.867 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_kv,legacy_keylisting} = false >>>> 2013-09-20 15:16:14.873 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_kv,listkeys_backpressure} = true >>>> 2013-09-20 15:16:14.884 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_kv,index_backpressure} = true >>>> 2013-09-20 15:16:14.889 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_kv,mapred_system} = pipe >>>> 2013-09-20 15:16:14.900 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_kv,mapred_2i_pipe} = true >>>> 2013-09-20 15:16:14.905 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_kv,anti_entropy} = enabled_v1 >>>> 2013-09-20 15:16:14.911 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_kv,handoff_data_encoding} = encode_raw >>>> 2013-09-20 15:16:14.917 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_kv,object_format} = v1 >>>> 2013-09-20 15:16:14.927 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_kv,secondary_index_version} = v2 >>>> 2013-09-20 15:16:14.943 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_kv,vclock_data_encoding} = encode_zlib >>>> 2013-09-20 15:16:14.953 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_kv,crdt} = [pncounter] >>>> 2013-09-20 15:16:15.011 [warning] <0.6.0> lager_error_logger_h dropped >>>> 17 messages in the last second that exceeded the limit of 100 messages/sec >>>> 2013-09-20 15:16:15.012 [info] >>>> <0.162.0>@riak_core_capability:process_capability_changes:530 New >>>> capability: {riak_control,member_info_version} = v1 >>>> 2013-09-20 15:16:15.022 [info] <0.7.0> Application riak_control started >>>> on node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:15.022 [info] <0.7.0> Application erlydtl started on >>>> node 'riak@192.168.20.107' >>>> 2013-09-20 15:16:16.013 [error] <0.1209.0> gen_fsm <0.1209.0> in state >>>> started terminated with reason: no function clause matching >>>> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,...]],...},...]}, >>>> undefined) line 851 >>>> 2013-09-20 15:16:16.013 [error] <0.1209.0> CRASH REPORT Process >>>> <0.1209.0> with 0 neighbours exited with reason: no function clause >>>> matching >>>> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,...]],...},...]}, >>>> undefined) line 851 in gen_fsm:terminate/7 line 589 >>>> 2013-09-20 15:16:16.014 [error] <0.140.0> Supervisor >>>> riak_core_vnode_sup had child undefined started with >>>> {riak_core_vnode,start_link,undefined} at <0.1209.0> exit with reason no >>>> function clause matching >>>> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,...]],...},...]}, >>>> undefined) line 851 in context child_terminated >>>> 2013-09-20 15:16:16.014 [error] <0.739.0> CRASH REPORT Process >>>> proxy_riak_kv_vnode_376793390874373408599387495934666716005045108736 with 0 >>>> neighbours exited with reason: >>>> {{{function_clause,[{riak_kv_vnode,terminate,[{badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,365375409332725729550921208179070754913983135744,359666418561901890026688064301272774368452149248,353957427791078050502454920423474793822921162752,348248437020254210978221776545676813277390176256,342539446249430371453988632667878832731859189760,336830455478606531929755488790080852186328203264,331121464707782692405522344912282871640797216768,...]],...},...]},...],...},...]},...},...} >>>> in gen_server:call/3 line 188 >>>> 2013-09-20 15:16:16.014 [error] <0.146.0> Supervisor >>>> riak_core_vnode_proxy_sup had child >>>> {riak_kv_vnode,376793390874373408599387495934666716005045108736} started >>>> with riak_core_vnode_proxy:start_link(riak_kv_vnode, >>>> 376793390874373408599387495934666716005045108736) at <0.739.0> exit with >>>> reason >>>> {{{function_clause,[{riak_kv_vnode,terminate,[{badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,365375409332725729550921208179070754913983135744,359666418561901890026688064301272774368452149248,353957427791078050502454920423474793822921162752,348248437020254210978221776545676813277390176256,342539446249430371453988632667878832731859189760,336830455478606531929755488790080852186328203264,331121464707782692405522344912282871640797216768,...]],...},...]},...],...},...]},...},...} >>>> in context child_terminated >>>> 2013-09-20 15:16:16.015 [error] <0.138.0> Supervisor riak_core_sup had >>>> child riak_core_vnode_manager started with >>>> riak_core_vnode_manager:start_link() at <0.161.0> exit with reason >>>> {{function_clause,[{riak_kv_vnode,terminate,[{badarg,[{lists,split,[<<"3">>,[376793390874373408599387495934666716005045108736,371084400103549569075154352056868735459514122240,365375409332725729550921208179070754913983135744,359666418561901890026688064301272774368452149248,353957427791078050502454920423474793822921162752,348248437020254210978221776545676813277390176256,342539446249430371453988632667878832731859189760,336830455478606531929755488790080852186328203264,331121464707782692405522344912282871640797216768,...]],...},...]},...],...},...]},...} >>>> in context child_terminated >>>> 2013-09-20 15:16:24.147 [error] <0.2034.0> gen_fsm <0.2034.0> in state >>>> started terminated with reason: no function clause matching >>>> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[445301280124259482890185222468242482551416946688,439592289353435643365952078590444502005885960192,...]],...},...]}, >>>> undefined) line 851 >>>> 2013-09-20 15:16:24.147 [error] <0.2034.0> CRASH REPORT Process >>>> <0.2034.0> with 0 neighbours exited with reason: no function clause >>>> matching >>>> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[445301280124259482890185222468242482551416946688,439592289353435643365952078590444502005885960192,...]],...},...]}, >>>> undefined) line 851 in gen_fsm:terminate/7 line 589 >>>> 2013-09-20 15:16:24.147 [error] <0.2019.0> gen_fsm <0.2019.0> in state >>>> started terminated with reason: no function clause matching >>>> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[34253944624943037145398863266787883273185918976,28544953854119197621165719388989902727654932480,...]],...},...]}, >>>> undefined) line 851 >>>> 2013-09-20 15:16:24.147 [error] <0.2019.0> CRASH REPORT Process >>>> <0.2019.0> with 0 neighbours exited with reason: no function clause >>>> matching >>>> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[34253944624943037145398863266787883273185918976,28544953854119197621165719388989902727654932480,...]],...},...]}, >>>> undefined) line 851 in gen_fsm:terminate/7 line 589 >>>> 2013-09-20 15:16:24.153 [error] <0.2029.0> gen_fsm <0.2029.0> in state >>>> started terminated with reason: no function clause matching >>>> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[308285501624487334308589769401090949458673270784,302576510853663494784356625523292968913142284288,...]],...},...]}, >>>> undefined) line 851 >>>> 2013-09-20 15:16:24.153 [error] <0.2029.0> CRASH REPORT Process >>>> <0.2029.0> with 0 neighbours exited with reason: no function clause >>>> matching >>>> riak_kv_vnode:terminate({badarg,[{lists,split,[<<"3">>,[308285501624487334308589769401090949458673270784,302576510853663494784356625523292968913142284288,...]],...},...]}, >>>> undefined) line 851 in gen_fsm:terminate/7 line 589 >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>> >> > > > -- > Jon Meredith > VP, Engineering > Basho Technologies, Inc. > jmered...@basho.com >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
