Thanks David - I'll try that on my single-node instance, but I'm working another Riak issue on another thread.
Jim ----- Original Message ----- From: "David Smith" <diz...@basho.com> To: "jim adler" <jim.ad...@comcast.net> Sent: Friday, October 7, 2011 7:02:01 AM Subject: Re: Timeout when storing Hi Jim, Sorry for the slow response -- email is like a running battle at times. :) How many partitions are you running? Also, take down the node and then remove any *.lock files. Thanks, D. On Mon, Oct 3, 2011 at 11:23 AM, <jim.ad...@comcast.net> wrote: > About 90 out of 3000 are zero-bytes. > > Jim > > -----Original Message----- > From: riak-users-boun...@lists.basho.com > [mailto:riak-users-boun...@lists.basho.com] On Behalf Of David Smith > Sent: Monday, October 03, 2011 4:46 AM > To: Jim Adler > Cc: riak-users@lists.basho.com > Subject: Re: Timeout when storing > > Jim, > > If you look at your bitcask directories, do you have a large number of > zero-byte files, perchance? > > D. > > On Sat, Oct 1, 2011 at 1:58 PM, Jim Adler <jad...@intelius.com> wrote: >> After upgrading my single-node instance to 1.0, I'm still seeing the >> "timeout when storing" issue. Here are the changes I made based on >> everyone's suggestions (much appreciated!): >> >> - Ubuntu 11.04 (natty) 32-bit >> - Python client 1.3.0 >> - /etc/riak/vm.args: -env ERL_MAX_PORTS 32768 >> - /etc/default/riak: ulimit -n 32768 >> >> Here's the /var/log/crash.log report: >> >> 2011-10-01 12:31:03 =ERROR REPORT==== >> ** State machine <0.3452.0> terminating >> ** Last event in was >> {'riak_vnode_req_v1',1136089163393944065322395631681798128560666312704 >> ,{fsm,undefined,<0.3451.0>},{'riak_kv_put_req_v1',{<<"nodes">>,<<"user >> _id-17527747-info">>},{r_object,<<"nodes">>,<<"user_id-17527747-info"> >> >,[{r_content,{dict,4,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[], >> [],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[[<<"content-type">>,9 >> 7,112,112,108,105,99,97,116,105,111,110,47,106,115,111,110],[<<"X-Riak >> -VTag">>,49,88,88,75,75,51,90,88,68,117,90,122,85,53,57,85,53,101,107, >> 89,115,110]],[[<<"index">>]],[],[[<<"X-Riak-Last-Modified">>|{1317,497 >> 463,847242}]],[],[]}}},<<"{DATA >> DELETED}">>}],[],{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[], >> [],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[[clean >> |true]],[]}}},undefined},51456853,63484716663,[coord]}} >> >> ** When State == active >> ** Data == >> {state,1136089163393944065322395631681798128560666312704,riak_kv_vnode >> ,{state,1136089163393944065322395631681798128560666312704,false,riak_k >> v_bitcask_backend,{state,#Ref<0.0.0.10359>,"11360891633939440653223956 >> 31681798128560666312704",[{async_folds,true},[{vnode_vclocks,true},{in >> cluded_applications,[]},{add_paths,[]},{allow_strfun,false},{storage_b >> ackend,riak_kv_bitcask_backend},{legacy_keylisting,false},{reduce_js_v >> m_count,6},{js_thread_stack,16},{pb_ip,"0.0.0.0"},{riak_kv_stat,true}, >> {map_js_vm_count,8},{mapred_system,pipe},{js_max_vm_mem,8},{pb_port,80 >> 87},{legacy_stats,true},{mapred_name,"mapred"},{stats_urlpath,"stats"} >> ,{http_url_encoding,on},{hook_js_vm_count,2}],{read_write,true}],11360 >> 89163393944065322395631681798128560666312704,"/var/lib/riak/bitcask"}, >> {dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[] >> },{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<<35,9,254,249, >> 78,135,82,106>>,3000,1000,100,100,true,false},undefined,undefined,none >> ,undefined,<0.3454.0>,60000} >> ** Reason for termination = >> ** {bad_return_value,{error,{write_locked,emfile}}} >> 2011-10-01 12:31:03 =CRASH REPORT==== >> crasher: >> initial call: riak_core_vnode:init/1 >> pid: <0.3452.0> >> registered_name: [] >> exception exit: {bad_return_value,{error,{write_locked,emfile}}} >> in function gen_fsm:terminate/7 >> in call from proc_lib:init_p_do_apply/3 >> ancestors: [riak_core_vnode_sup,riak_core_sup,<0.92.0>] >> messages: [{'EXIT',<0.3454.0>,shutdown}] >> links: [<0.96.0>] >> dictionary: [] >> trap_exit: true >> status: running >> heap_size: 6765 >> stack_size: 24 >> reductions: 160650 >> neighbours: >> 2011-10-01 12:31:03 =SUPERVISOR REPORT==== >> Supervisor: {local,riak_core_vnode_sup} >> Context: child_terminated >> Reason: {bad_return_value,{error,{write_locked,emfile}}} >> Offender: >> [{pid,<0.3452.0>},{name,undefined},{mfargs,{riak_core_vnode,start_link >> ,undefined}},{restart_type,temporary},{shutdown,300000},{child_type,wo >> rker}] >> >> 2011-10-01 12:45:28 =ERROR REPORT==== >> Failed to merge >> > "/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/var/ > lib/riak/bitcask/605153021707326989568713251046585937826284568576/1315770213 > .bitcask.data/var/lib/riak/bitcask/60515302170732698956871325104658593782628 > 4568576/1316329673.bitcask.data/var/lib/riak/bitcask/60515302170732698956871 > 3251046585937826284568576/1316330222.bitcask.data/var/lib/riak/bitcask/60515 > 3021707326989568713251046585937826284568576/1316879145.bitcask.data/var/lib/ > riak/bitcask/605153021707326989568713251046585937826284568576/1316995340.bit > cask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568 > 576/1317493005.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251 > 046585937826284568576/1317495168.bitcask.data": >> {{badmatch,{error,emfile}},[{bitcask,'-merge1/3-lc$^0/1-1-',1},{bitcas >> k,'-merge1/3-lc$^0/1-1-',1},{bitcask,'merge1',3},{bitcask_merge_worker >> ,do_merge,1}]} >> >> >> -----Original Message----- >> From: David Smith [mailto:diz...@basho.com] >> Sent: Fri 9/30/2011 9:56 AM >> To: Jim Adler >> Cc: Sean Cribbs; riak-users@lists.basho.com >> Subject: Re: Timeout when storing >> >> IIRC, {error, emfile} indicates that the max # of ports (in the erlang >> VM) is being exceeded. Try bumping up ERL_MAX_PORTS in vm.args. >> >> D. >> >> On Thu, Sep 29, 2011 at 10:52 PM, Jim Adler <jad...@intelius.com> wrote: >>> Thanks Sean. I added the ulimit -n 10240 to /etc/default/riak, >>> restarted riak, but that didn't work. >>> >>> Fyodor Yarochkin suggested that the bitcask files could be corrupted, >>> but I wasn't sure which bitcask *.data or *.hint file to delete. Any >>> pointers? >>> >>> Here's the /var/log/riak/erlang.log: >>> >>> =ERROR REPORT==== 29-Sep-2011::20:27:42 === >>> ** State machine <0.369.0> terminating >>> ** Last event in was {riak_vnode_req_v1, >>> >>> 941983477185933521498468739836666790012612771840, >>> {fsm,undefined,<0.27704.1>}, >>> {riak_kv_put_req_v1, >>> {<<"nodes">>,<<"screen_name-psych_ic-info">>}, >>> >>> {r_object,<<"nodes">>,<<"screen_name-psych_ic-info">>, >>> [{r_content, >>> {dict,3,16,16,8,80,48, >>> >>> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, >>> {{[],[],[],[],[],[],[],[],[],[], >>> >>> [[<<"content-type">>,97,112,112,108,105,99,97, >>> 116,105,111,110,47,106,115,111,110], >>> >>> [<<"X-Riak-VTag">>,49,90,120,65,84,100,56,99,48, >>> >>> 80,86,99,111,122,71,79,108,90,70,97,53,87]], >>> [],[], >>> [[<<"X-Riak-Last-Modified">>| >>> {1317,353201,695471}]], >>> [],[]}}}, >>> <<"{DELETED DATA}">>}], >>> [{<<2,65,205,48>>,{1,63484572401}}], >>> {dict,1,16,16,8,80,48, >>> >>> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, >>> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[], >>> [[clean|true]], >>> []}}}, >>> undefined}, >>> 1174401,63484572401, >>> [{returnbody,true}]}} >>> ** When State == active >>> ** Data == >>> {state,941983477185933521498468739836666790012612771840, >>> riak_kv_vnode, >>> >>> {state,941983477185933521498468739836666790012612771840, >>> riak_kv_bitcask_backend, >>> {#Ref<0.0.0.3952>, >>> >>> "/var/lib/riak/bitcask/9419834771859335214984687398366667900126127718 >>> 40"}, >>> {dict,0,16,16,8,80,48, >>> >>> {[],[],[],[],[],[],[],[],[],[],[],[],[], >>> [],[],[]}, >>> >>> {{[],[],[],[],[],[],[],[],[],[],[],[],[], >>> [],[],[]}}}, >>> false}, >>> undefined,none,60000} >>> ** Reason for termination = >>> ** {{badmatch,{error,emfile}}, >>> [{bitcask_fileops,create_file_loop,3}, >>> {bitcask,put,3}, >>> {riak_kv_bitcask_backend,put,3}, >>> {riak_kv_vnode,perform_put,3}, >>> {riak_kv_vnode,do_put,7}, >>> {riak_kv_vnode,handle_command,3}, >>> {riak_core_vnode,vnode_command,3}, >>> {gen_fsm,handle_msg,7}]} >>> >>> >>> >>> -----Original Message----- >>> From: Sean Cribbs [mailto:s...@basho.com] >>> Sent: Thu 9/29/2011 3:02 PM >>> To: Jim Adler >>> Cc: riak-users@lists.basho.com >>> Subject: Re: Timeout when storing >>> >>> Your environment has too few file handles. Retry starting riak after >>> setting `ulimit -n 1024` in the shell. Also see our wiki page about >>> this >>> issue: http://wiki.basho.com/Open-Files-Limit.html You may need to >>> set this limit specifically for the 'riak' user. >>> >>> Cheers, >>> >>> -- >>> Sean Cribbs <s...@basho.com> >>> Developer Advocate >>> Basho Technologies, Inc. >>> http://www.basho.com/ >>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >> >> >> >> -- >> Dave Smith >> Director, Engineering >> Basho Technologies, Inc. >> diz...@basho.com >> >> > > > > -- > Dave Smith > Director, Engineering > Basho Technologies, Inc. > diz...@basho.com > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > -- Dave Smith Director, Engineering Basho Technologies, Inc. diz...@basho.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
