Jim,
If you look at your bitcask directories, do you have a large number of
zero-byte files, perchance?
D.
On Sat, Oct 1, 2011 at 1:58 PM, Jim Adler <jad...@intelius.com> wrote:
> After upgrading my single-node instance to 1.0, I'm still seeing the
> "timeout when storing" issue. Here are the changes I made based on
> everyone's suggestions (much appreciated!):
>
> - Ubuntu 11.04 (natty) 32-bit
> - Python client 1.3.0
> - /etc/riak/vm.args: -env ERL_MAX_PORTS 32768
> - /etc/default/riak: ulimit -n 32768
>
> Here's the /var/log/crash.log report:
>
> 2011-10-01 12:31:03 =ERROR REPORT====
> ** State machine <0.3452.0> terminating
> ** Last event in was
> {'riak_vnode_req_v1',1136089163393944065322395631681798128560666312704,{fsm,undefined,<0.3451.0>},{'riak_kv_put_req_v1',{<<"nodes">>,<<"user_id-17527747-info">>},{r_object,<<"nodes">>,<<"user_id-17527747-info">>,[{r_content,{dict,4,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[[<<"content-type">>,97,112,112,108,105,99,97,116,105,111,110,47,106,115,111,110],[<<"X-Riak-VTag">>,49,88,88,75,75,51,90,88,68,117,90,122,85,53,57,85,53,101,107,89,115,110]],[[<<"index">>]],[],[[<<"X-Riak-Last-Modified">>|{1317,497463,847242}]],[],[]}}},<<"{DATA
> DELETED}">>}],[],{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[[clean|true]],[]}}},undefined},51456853,63484716663,[coord]}}
>
> ** When State == active
> ** Data ==
> {state,1136089163393944065322395631681798128560666312704,riak_kv_vnode,{state,1136089163393944065322395631681798128560666312704,false,riak_kv_bitcask_backend,{state,#Ref<0.0.0.10359>,"1136089163393944065322395631681798128560666312704",[{async_folds,true},[{vnode_vclocks,true},{included_applications,[]},{add_paths,[]},{allow_strfun,false},{storage_backend,riak_kv_bitcask_backend},{legacy_keylisting,false},{reduce_js_vm_count,6},{js_thread_stack,16},{pb_ip,"0.0.0.0"},{riak_kv_stat,true},{map_js_vm_count,8},{mapred_system,pipe},{js_max_vm_mem,8},{pb_port,8087},{legacy_stats,true},{mapred_name,"mapred"},{stats_urlpath,"stats"},{http_url_encoding,on},{hook_js_vm_count,2}],{read_write,true}],1136089163393944065322395631681798128560666312704,"/var/lib/riak/bitcask"},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<<35,9,254,249,78,135,82,106>>,3000,1000,100,100,true,false},undefined,undefined,none,undefined,<0.3454.0>,60000}
> ** Reason for termination =
> ** {bad_return_value,{error,{write_locked,emfile}}}
> 2011-10-01 12:31:03 =CRASH REPORT====
> crasher:
> initial call: riak_core_vnode:init/1
> pid: <0.3452.0>
> registered_name: []
> exception exit: {bad_return_value,{error,{write_locked,emfile}}}
> in function gen_fsm:terminate/7
> in call from proc_lib:init_p_do_apply/3
> ancestors: [riak_core_vnode_sup,riak_core_sup,<0.92.0>]
> messages: [{'EXIT',<0.3454.0>,shutdown}]
> links: [<0.96.0>]
> dictionary: []
> trap_exit: true
> status: running
> heap_size: 6765
> stack_size: 24
> reductions: 160650
> neighbours:
> 2011-10-01 12:31:03 =SUPERVISOR REPORT====
> Supervisor: {local,riak_core_vnode_sup}
> Context: child_terminated
> Reason: {bad_return_value,{error,{write_locked,emfile}}}
> Offender:
> [{pid,<0.3452.0>},{name,undefined},{mfargs,{riak_core_vnode,start_link,undefined}},{restart_type,temporary},{shutdown,300000},{child_type,worker}]
>
> 2011-10-01 12:45:28 =ERROR REPORT====
> Failed to merge
> "/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1315770213.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1316329673.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1316330222.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1316879145.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1316995340.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1317493005.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1317495168.bitcask.data":
> {{badmatch,{error,emfile}},[{bitcask,'-merge1/3-lc$^0/1-1-',1},{bitcask,'-merge1/3-lc$^0/1-1-',1},{bitcask,'merge1',3},{bitcask_merge_worker,do_merge,1}]}
>
>
> -----Original Message-----
> From: David Smith [mailto:diz...@basho.com]
> Sent: Fri 9/30/2011 9:56 AM
> To: Jim Adler
> Cc: Sean Cribbs; riak-users@lists.basho.com
> Subject: Re: Timeout when storing
>
> IIRC, {error, emfile} indicates that the max # of ports (in the erlang
> VM) is being exceeded. Try bumping up ERL_MAX_PORTS in vm.args.
>
> D.
>
> On Thu, Sep 29, 2011 at 10:52 PM, Jim Adler <jad...@intelius.com> wrote:
>> Thanks Sean. I added the ulimit -n 10240 to /etc/default/riak, restarted
>> riak, but that didn't work.
>>
>> Fyodor Yarochkin suggested that the bitcask files could be corrupted, but
>> I
>> wasn't sure which bitcask *.data or *.hint file to delete. Any pointers?
>>
>> Here's the /var/log/riak/erlang.log:
>>
>> =ERROR REPORT==== 29-Sep-2011::20:27:42 ===
>> ** State machine <0.369.0> terminating
>> ** Last event in was {riak_vnode_req_v1,
>> 941983477185933521498468739836666790012612771840,
>> {fsm,undefined,<0.27704.1>},
>> {riak_kv_put_req_v1,
>> {<<"nodes">>,<<"screen_name-psych_ic-info">>},
>>
>> {r_object,<<"nodes">>,<<"screen_name-psych_ic-info">>,
>> [{r_content,
>> {dict,3,16,16,8,80,48,
>>
>> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>> {{[],[],[],[],[],[],[],[],[],[],
>>
>> [[<<"content-type">>,97,112,112,108,105,99,97,
>> 116,105,111,110,47,106,115,111,110],
>>
>> [<<"X-Riak-VTag">>,49,90,120,65,84,100,56,99,48,
>>
>> 80,86,99,111,122,71,79,108,90,70,97,53,87]],
>> [],[],
>> [[<<"X-Riak-Last-Modified">>|
>> {1317,353201,695471}]],
>> [],[]}}},
>> <<"{DELETED DATA}">>}],
>> [{<<2,65,205,48>>,{1,63484572401}}],
>> {dict,1,16,16,8,80,48,
>>
>> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>> [[clean|true]],
>> []}}},
>> undefined},
>> 1174401,63484572401,
>> [{returnbody,true}]}}
>> ** When State == active
>> ** Data == {state,941983477185933521498468739836666790012612771840,
>> riak_kv_vnode,
>>
>> {state,941983477185933521498468739836666790012612771840,
>> riak_kv_bitcask_backend,
>> {#Ref<0.0.0.3952>,
>>
>> "/var/lib/riak/bitcask/941983477185933521498468739836666790012612771840"},
>> {dict,0,16,16,8,80,48,
>>
>> {[],[],[],[],[],[],[],[],[],[],[],[],[],
>> [],[],[]},
>>
>> {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>> [],[],[]}}},
>> false},
>> undefined,none,60000}
>> ** Reason for termination =
>> ** {{badmatch,{error,emfile}},
>> [{bitcask_fileops,create_file_loop,3},
>> {bitcask,put,3},
>> {riak_kv_bitcask_backend,put,3},
>> {riak_kv_vnode,perform_put,3},
>> {riak_kv_vnode,do_put,7},
>> {riak_kv_vnode,handle_command,3},
>> {riak_core_vnode,vnode_command,3},
>> {gen_fsm,handle_msg,7}]}
>>
>>
>>
>> -----Original Message-----
>> From: Sean Cribbs [mailto:s...@basho.com]
>> Sent: Thu 9/29/2011 3:02 PM
>> To: Jim Adler
>> Cc: riak-users@lists.basho.com
>> Subject: Re: Timeout when storing
>>
>> Your environment has too few file handles. Retry starting riak after
>> setting `ulimit -n 1024` in the shell. Also see our wiki page about this
>> issue: http://wiki.basho.com/Open-Files-Limit.html You may need to set
>> this
>> limit specifically for the 'riak' user.
>>
>> Cheers,
>>
>> --
>> Sean Cribbs <s...@basho.com>
>> Developer Advocate
>> Basho Technologies, Inc.
>> http://www.basho.com/
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>
>
> --
> Dave Smith
> Director, Engineering
> Basho Technologies, Inc.
> diz...@basho.com
>
>
--
Dave Smith
Director, Engineering
Basho Technologies, Inc.
diz...@basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
