Re: Riak stalls, leveldb backend

2015-02-19 Thread Andy Pellett
Hi Luke, Your prediction was spot on. Today, about 2 weeks after setting the parameters from my earlier email, we saw riak nodes start to run out of memory and fail. I have to say, it was a great 2 weeks though. I've adjusted the parameters again, this time paying attention the linked spreadshe

Re: Riak stalls, leveldb backend

2015-02-11 Thread Luke Bakken
Hi Andy, I'd like to strongly recommend using the spreadsheet linked to by this document: http://docs.basho.com/riak/1.4.12/ops/advanced/backends/leveldb/ Spreadsheet: https://docs.google.com/spreadsheet/ccc?key=0AnW_U8Qe8NdYdGk2V3Qza0VRNkxyRFNGUVVCV0c3V3c&usp=sharing#gid=0 The reason I bring

Re: Riak stalls, leveldb backend

2015-02-11 Thread Andy Pellett
Hi All, I ran across this page: http://docs.basho.com/riak/1.0.0/tutorials/choosing-a-backend/LevelDB/ and ended up tuning the following parameters: max_open_files 16384 write_buffer_size_min 1gb write_buffer_size_max 2gb cache_size 15gb block_size 262144 Since making these changes about a wee

Re: Riak stalls, leveldb backend

2015-01-06 Thread Drew Pirrone-Brusse
Hey all, So, the long and short of it is that Riak doesn't handle objects larger than 2Mb very well at all. We try to be pretty up-front about this in the documentation (for example[1]) be we're also aware that sometimes you need to store big objects. Riak CS was actually created to deal with thes

Re: Riak stalls, leveldb backend

2015-01-06 Thread Michael Lewis
Hi, I'd be interested in the answer to this too, isn't it inevitable that in the underlying storage you will get compaction delays, you see the same in any key value store, eg if you were pushing lots of small packets of data into HBase/Cassandra etc... Eventually the system has to perform the

Re: Riak stalls, leveldb backend

2015-01-06 Thread Sorin Manole
Hey Andy, We had the same issue few days ago. We were getting timeouts when trying to read a key from riak. Also we were seeing in the logs a warning about reading/writing a large object. In our case that object was first read from riak and after written back. It was that big (17Mb) because of so

Riak stalls, leveldb backend

2015-01-06 Thread Andy Pellett
Hi all, I've been experiencing stalls where riak won't return any data (queries time out) with my riak cluster. Here are some basic details: - 8 nodes - riak 1.4.10 (upgraded from 1.4.6 -> 1.4.8 -> 1.4.10) - leveldb backend - n_val is 2 - allow_mult is false - ec2 i2.2xlarge boxes (8 cores, 61gb