Re: RiakCS poor s3 upload speeds 2MB/s

2015-01-28 Thread Toby Corkindale
I turned the triggers and thresholds down, to: {max_file_size,805306368}, %% 768 MB {dead_bytes_merge_trigger, 134217728}, %% dead bytes > 128 MB {dead_bytes_threshold, 33554432} %% dead bytes > 32 MB And restarted nodes; however after 24 h

Re: RiakCS poor s3 upload speeds 2MB/s

2015-01-21 Thread Kota Uenishi
Toby, Glad to hear you gained some. For disk usage gaps, I'd recommend smaller leeway_seconds, like an hour or even less, depending on the average object size and read concurrency. leeway seconds is just for assuring long-running download not being interrupted by concurrent overwrite or deletion

Re: RiakCS poor s3 upload speeds 2MB/s

2015-01-21 Thread Luke Bakken
You should be able to help with disk usage by "turning down" the trigger and threshold values described here: http://docs.basho.com/riak/latest/ops/advanced/backends/bitcask/ Your cluster will merge more data which should help with disk usage. If your typical use is to create and delete objects f

Re: RiakCS poor s3 upload speeds 2MB/s

2015-01-21 Thread Toby Corkindale
On 21 January 2015 at 15:22, Luke Bakken wrote: > Hi Toby - > > Are you using the stock bitcask configuration for merging? Hi Luke, Yes, pretty much. > On Tue, Jan 20, 2015 at 5:07 PM, Toby Corkindale wrote: >> Hi Kota, >> I had a bit of an off-list chat about this a while ago, plus continued >

Re: RiakCS poor s3 upload speeds 2MB/s

2015-01-20 Thread Luke Bakken
Hi Toby - Are you using the stock bitcask configuration for merging? -- Luke Bakken Engineer lbak...@basho.com On Tue, Jan 20, 2015 at 5:07 PM, Toby Corkindale wrote: > Hi Kota, > I had a bit of an off-list chat about this a while ago, plus continued > to investigate locally, and eventually ach

Re: RiakCS poor s3 upload speeds 2MB/s

2015-01-20 Thread Toby Corkindale
Hi Kota, I had a bit of an off-list chat about this a while ago, plus continued to investigate locally, and eventually achieved some faster speeds, around 15MByte/sec writes. Things that were changed: * Adjusted Riak CS GC to be spread out over the cluster much more. * Tweaked up the put buffers

Re: RiakCS poor s3 upload speeds 2MB/s

2015-01-20 Thread Kota Uenishi
Toby and David, Thank you for working on Riak CS and I apologize for being late responder. I believe the reason of being slow down is different between Toby's and David's cluster. Toby's reason looks like that's just because of the data increasing. How much data per vnode do you have in your clu

Re: RiakCS poor s3 upload speeds 2MB/s

2014-11-26 Thread Heinz Nikolaus Gies
I’ve run into a few issues along the line, never lost data so. It requires more love and care then riak and things happen less automatic. everything involving riak gives me a warm and happy feeling that I can trust what goes on, Leo is not just there yet but for me the order of magnitude in spee

Re: RiakCS poor s3 upload speeds 2MB/s

2014-11-26 Thread Toby Corkindale
Thanks, that's interesting to hear. How have you been finding the stability and reliability to be with leofs, over time? I still wish I could just get our Riak CS cluster performing better; it just seems so unreasonably slow at the moment, that I suspect there's *something* holding it back. I can

Re: RiakCS poor s3 upload speeds 2MB/s

2014-11-25 Thread Heinz Nikolaus Gies
If you’re evaluating RiakCS vs. Ceph you might want to toss LeoFS[1] in the mix and give it a run. Just as RiakCS it is a dynamo inspired system build in Erlang and comes with the same advantages and disadvantages. But unlike RiakCS it is pretty much exclusive a Object Store so can take a few di

Re: RiakCS poor s3 upload speeds 2MB/s

2014-11-24 Thread Toby Corkindale
Hi, I wondered if you managed to significantly improve your Riak CS performance, or not? I just ask as we've been getting not-dissimilar performance out of Riak CS too (4-5 mbyte/sec max per client, on bare metal hardware), for quite a long time. (I swear it was faster originally, when there was a

Re: RiakCS poor s3 upload speeds 2MB/s

2014-11-10 Thread Toby Corkindale
I found that adjusting these RiakCS settings improved write speeds a little for us: {put_concurrency, 4}, {put_buffer_factor, 8}, As well as making sure the Riak vm.args included a high +zdbbl setting. But even so, Riak CS has been really quite slow compared to other

Re: RiakCS poor s3 upload speeds 2MB/s

2014-11-10 Thread Toby Corkindale
I found that adjusting these RiakCS settings improved write speeds a little for us: {put_concurrency, 4}, {put_buffer_factor, 8}, As well as making sure the Riak vm.args included a high +zdbbl setting. But even so, Riak CS has been really quite slow compared to other

Re: RiakCS poor s3 upload speeds 2MB/s

2014-11-10 Thread Luke Bakken
Physical hardware we setup a 4 node cluster > within our cloudstack environment and attained the same speeds. It feels > like something is throttling somewhere. > > > > Thanks for your help > > > > Dave > > *From:* Luke Bakken [mailto:lbak...@basho.com] > *Sen

RE: RiakCS poor s3 upload speeds 2MB/s

2014-11-10 Thread David Meekin
: Luke Bakken [mailto:lbak...@basho.com] Sent: 08 November 2014 15:33 To: David Meekin Cc: riak-users@lists.basho.com Subject: Re: RiakCS poor s3 upload speeds 2MB/s Hi David, Could you please provide a bit more information? * The version of Riak and Riak CS you are using * A description of the

Re: RiakCS poor s3 upload speeds 2MB/s

2014-11-08 Thread Luke Bakken
Hi David, Could you please provide a bit more information? * The version of Riak and Riak CS you are using * A description of the files you are uploading. What size are they and are you using multipart uploads? Thanks -- Luke Bakken Engineer / CSE lbak...@basho.com On Fri, Nov 7, 2014 at 6:41 A