Hi Toby - Are you using the stock bitcask configuration for merging?
-- Luke Bakken Engineer lbak...@basho.com On Tue, Jan 20, 2015 at 5:07 PM, Toby Corkindale <t...@dryft.net> wrote: > Hi Kota, > I had a bit of an off-list chat about this a while ago, plus continued > to investigate locally, and eventually achieved some faster speeds, > around 15MByte/sec writes. > Things that were changed: > * Adjusted Riak CS GC to be spread out over the cluster much more. > * Tweaked up the put buffers and concurrency further > * Moved most of the files out of CS and into Amazon S3+Glacier > * Switched from nginx to haproxy > * simplified firewalling for internal clients > > Each one of those changes made a small to modest improvement, but > overall combined to make a quite noticeable improvement. > > I did notice something odd though -- despite moving most of the data > out of the cluster, the disk-space-in-use by Riak is still very large > compared to the amount stored. I mean, we moved more than 90% of the > data out of the cluster, yet the actual disk space used only halved. > For every gigabyte of file stored in CS, dozens of gigabytes are > actually on disk! > > Either the garbage collection algorithm is very, very lazy, or somehow > something has gone a bit wrong in the past, which might have explained > part of the performance problems. > > We're going to look at redeploying a new, fresh cluster based on Riak > 2 in the not too distant future, once Riak CS looks like it's approved > for use there, and maybe that'll clear all of this up. > > Toby _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
