Hi Luke Bakken, The size of the objects stored in Riak varies from 10kb to 500kb. I did see the need to upgrade RIAK due to AAE fixes and will be doing that. However I wanted to understand the root cause and make sure this wouldn't happen again after the upgrade.
Regards, Jerald -----Original Message----- From: Luke Bakken [mailto:lbak...@basho.com] Sent: Wednesday, May 03, 2017 8:01 PM To: Arulappan, Jerald (Jerald) **CTR** <ajer...@avaya.com> Cc: riak-users@lists.basho.com Subject: Re: Riak 2.1.4 crashes with Out of Memory Error Hi Jerald - What is the average size of an object that you are storing in Riak? I'm also seeing a lot of errors in the logs related to AAE, like you note. I know there have been fixes in that part of Riak since 2.1.4 and recommend upgrading. -- Luke Bakken Engineer lbak...@basho.com On Tue, May 2, 2017 at 6:56 AM, Arulappan, Jerald (Jerald) <ajer...@avaya.com> wrote: > Hi, > > I am using a single node riak server 2.1.4 with bitcask as backend for > storing files. > The riak node stops working after every week. (Looks like when the > active anti-entropy process recreates the hash tree) The sylog shows > Out of memory Error. But the console.log shows "sst: No such file or > directory" > Syslog Error: > > Apr 26 17:39:37 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 > (beam.smp) score 824 or sacrifice child Apr 26 17:39:37 TLCCBAPRO2 > kernel: Killed process 16987 (sh) total-vm:106168kB, anon-rss:116kB, > file-rss:0kB Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill > process 16685 > (beam.smp) score 824 or sacrifice child Apr 26 17:39:41 TLCCBAPRO2 > kernel: Killed process 30374 (memsup) total-vm:4112kB, anon-rss:80kB, > file-rss:0kB Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill > process 16685 > (beam.smp) score 824 or sacrifice child Apr 26 17:39:41 TLCCBAPRO2 > kernel: Killed process 14351 (cpu_sup) total-vm:4112kB, anon-rss:68kB, > file-rss:0kB Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill > process 16685 > (beam.smp) score 824 or sacrifice child Apr 26 17:39:41 TLCCBAPRO2 > kernel: Killed process 30385 (sh) total-vm:106164kB, anon-rss:136kB, > file-rss:416kB Apr 26 17:44:48 TLCCBAPRO2 run_erl[16682]: Erlang > closed the connection. > > Console.log: > > 2017-04-26 17:37:03.493 [info] > <0.625.0>@riak_kv_vnode:maybe_create_hashtrees:227 > riak_kv/91343852333181432387730302044767688728495783936: unable to > start > index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: > ./data/anti_entropy/91343852333181432387730302044767688728495783936/sst_0/001954.sst: > No such file or > directory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl" > },{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]} > ,{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hasht > ree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}] > },{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hasht > ree.erl"},{line,474}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak > _kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"ge > n_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_l > ib.erl"},{line,239}]}]}} > 2017-04-26 17:37:03.515 [error] <0.30178.2881> CRASH REPORT Process > <0.30178.2881> with 0 neighbours exited with reason: no match of right > hand value {error,{db_open,"IO error: > ./data/anti_entropy/936274486415109681974235595958868809467081785344/000037.sst: > No such file or directory"}} in hashtree:new_segment_store/2 line 675 > in > gen_server:init_it/6 line 328 > 2017-04-26 17:37:03.515 [info] > <0.623.0>@riak_kv_vnode:maybe_create_hashtrees:227 > riak_kv/45671926166590716193865151022383844364247891968: unable to > start > index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: > ./data/anti_entropy/45671926166590716193865151022383844364247891968/sst_0/002239.sst: > No such file or > directory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl" > },{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]} > ,{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hasht > ree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}] > },{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hasht > ree.erl"},{line,474}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak > _kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"ge > n_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_l > ib.erl"},{line,239}]}]}} > 2017-04-26 17:37:03.516 [error] <0.30207.2881> CRASH REPORT Process > <0.30207.2881> with 0 neighbours exited with reason: no match of right > hand value {error,{db_open,"IO error: > ./data/anti_entropy/45671926166590716193865151022383844364247891968/sst_0/002239.sst: > No such file or directory"}} in hashtree:new_segment_store/2 line 675 > in > gen_server:init_it/6 line 328 > > > > The complete logs are in the attached zip file. Any thoughts on the > root cause and possible solution to overcome this is much appreciated. > > > > Regards, > > Jerald > > > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.basho.com_ma > ilman_listinfo_riak-2Dusers-5Flists.basho.com&d=DwIFaQ&c=BFpWQw8bsuKpl > 1SgiZH64Q&r=zOPuaocGAfHxJUtnOg9ZgYW14EJAnNldsbCwB84w4zE&m=7t76DY3mCrip > DH3glx_DHTdBaJGT_aqr1EtXkWhPYhI&s=rVfw97EO99wQM0eCfFhWyIPzTLyS2jk_3qZw > XcvWcpE&e= > _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
