Travis, I cannot address the crash error you see in the logs. Someone else will have to review that problem.
I want to point you to this wiki article: https://github.com/basho/leveldb/wiki/riak-tuning-2 The article details how you can potentially increase Riak's throughput by restricting the number of schedulers (CPUs) assigned solely to Erlang. Heavy bulk loading scenarios demonstrated 20 to 30% throughput gain. Per the wiki article, you would create an (or add to your existing) advanced.config with the following based upon your 16 core machines: [ {vm_args, [{"+S", "14:14"}]} ]. Matthew > On Aug 31, 2016, at 11:41 AM, Travis Kirstine > <[email protected]> wrote: > > Magnus > > Thanks for your reply. We’re are using the riack C client library for riak > (https://github.com/trifork/riack <https://github.com/trifork/riack>) which > is used within an application called MapCache to store 256x256 px images with > a corresponding key within riak. Currently we have 75 million images to > transfer from disk into riak which is being done concurrently. Periodically > this transfer process will crash > > Riak is setup using n=3 on 5 nodes with a leveldb backend. Each server has > 45GB of memory and 16 cores with standard hard drives. We made no > significant modification to the riak.conf except upping the > leveldb.maximum_memory.percent to 70 and tweeking the sysctl.conf as follows > > vm.swappiness = 0 > net.ipv4.tcp_max_syn_backlog = 40000 > net.core.somaxconn = 40000 > net.core.wmem_default = 8388608 > net.core.rmem_default = 8388608 > net.ipv4.tcp_sack = 1 > net.ipv4.tcp_window_scaling = 1 > net.ipv4.tcp_fin_timeout = 15 > net.ipv4.tcp_keepalive_intvl = 30 > net.ipv4.tcp_tw_reuse = 1 > net.ipv4.tcp_moderate_rcvbuf = 1 > # Increase the open file limit > # fs.file-max = 65536 # current setting > > I have seen this error in the logs > 2016-08-30 22:26:07.180 [error] <0.20777.512> CRASH REPORT Process > <0.20777.512> with 0 neighbours crashed with reason: no function clause > matching webmachine_request:peer_from_peername({error,enotconn}, > {webmachine_request,{wm_reqstate,#Port<0.2817336>,[],undefined,undefined,undefined,{wm_reqdata,'GET',...},...}}) > line 150 > > Regards > > From: Magnus Kessler [mailto:[email protected] <mailto:[email protected]>] > Sent: August-31-16 4:08 AM > To: Travis Kirstine <[email protected] > <mailto:[email protected]>> > Cc: [email protected] <mailto:[email protected]> > Subject: Re: speeding up bulk loading > > On 26 August 2016 at 22:20, Travis Kirstine <[email protected] > <mailto:[email protected]>> wrote: > Is there any way to speed up bulk loading? I wondering if I should be > tweeking the erlang, aae or other config options? > > > > Hi Travis, > > Excuse the late reply; your message had been stuck in the moderation queue. > Please consider subscribing to this list. > > Without knowing more about how you perform bulk uploads, it's difficult to > recommend any changes. Are you using the HTTP REST API or one of the client > libraries, which use protocol buffers by default? What concerns do you have > about the upload performance? Please let us know a bit more about your setup. > > Kind Regards, > > Magnus > > > -- > Magnus Kessler > Client Services Engineer > Basho Technologies Limited > > Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431 > _______________________________________________ > riak-users mailing list > [email protected] <mailto:[email protected]> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
