Magnus
Thanks for your reply. We’re are using the riack C client library for riak
(https://github.com/trifork/riack) which is used within an application called
MapCache to store 256x256 px images with a corresponding key within riak.
Currently we have 75 million images to transfer from disk into riak which is
being done concurrently. Periodically this transfer process will crash
Riak is setup using n=3 on 5 nodes with a leveldb backend. Each server has
45GB of memory and 16 cores with standard hard drives. We made no significant
modification to the riak.conf except upping the leveldb.maximum_memory.percent
to 70 and tweeking the sysctl.conf as follows
vm.swappiness = 0
net.ipv4.tcp_max_syn_backlog = 40000
net.core.somaxconn = 40000
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_moderate_rcvbuf = 1
# Increase the open file limit
# fs.file-max = 65536 # current setting
I have seen this error in the logs
2016-08-30 22:26:07.180 [error] <0.20777.512> CRASH REPORT Process
<0.20777.512> with 0 neighbours crashed with reason: no function clause
matching webmachine_request:peer_from_peername({error,enotconn},
{webmachine_request,{wm_reqstate,#Port<0.2817336>,[],undefined,undefined,undefined,{wm_reqdata,'GET',...},...}})
line 150
Regards
From: Magnus Kessler [mailto:[email protected]]
Sent: August-31-16 4:08 AM
To: Travis Kirstine <[email protected]>
Cc: [email protected]
Subject: Re: speeding up bulk loading
On 26 August 2016 at 22:20, Travis Kirstine
<[email protected]<mailto:[email protected]>>
wrote:
Is there any way to speed up bulk loading? I wondering if I should be tweeking
the erlang, aae or other config options?
Hi Travis,
Excuse the late reply; your message had been stuck in the moderation queue.
Please consider subscribing to this list.
Without knowing more about how you perform bulk uploads, it's difficult to
recommend any changes. Are you using the HTTP REST API or one of the client
libraries, which use protocol buffers by default? What concerns do you have
about the upload performance? Please let us know a bit more about your setup.
Kind Regards,
Magnus
--
Magnus Kessler
Client Services Engineer
Basho Technologies Limited
Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
