Hey all, So, the long and short of it is that Riak doesn't handle objects larger than 2Mb very well at all. We try to be pretty up-front about this in the documentation (for example[1]) be we're also aware that sometimes you need to store big objects. Riak CS was actually created to deal with these exact performance issues, so I'd highly recommend you take a look at it, if your usecase requires large objects.
When talking specifically about LevelDB, we have some useful implementation details[2] in our docs. Take special note of the file sizes and total size limits for Level-0 and Level-1---each file is limited to 2Mb, and Level-1 will only store up to 10Mb. The disk I/O required to flush Level-0 and Level-1 completely so that you can write -- and then compact -- a >10Mb file (as is your case, Andy) adds up pretty quickly. I hope that helps. Best, -Drew [1] - http://docs.basho.com/riak/latest/dev/using/application-guide/#What-Kind-of-Data-Are-You-Storing- [2] - http://docs.basho.com/riak/latest/ops/advanced/backends/leveldb/#Implementation-Details On Tue, Jan 6, 2015 at 6:01 AM, Sorin Manole <sorin.man...@trustev.com> wrote: > > Hey Andy, > > We had the same issue few days ago. We were getting timeouts when trying to > read a key from riak. > > Also we were seeing in the logs a warning about reading/writing a large > object. In our case that object was first read from riak and after written > back. It was that big (17Mb) because of some load testing that we did. > > We deleted the object and we didn't experience timeouts anymore. > > We tried to repro the issue, so we can be sure it was this and we posted a > 20Mb json to the same key and the timeouts came back again, deleted the 20Mb > object and everything worked fine again. > > I don't know why this is happening but we can say that this was the issue for > our timeouts. > > Maybe someone has a better understanding on this.. I'd like to be in the loop > of this conversation if possible. > > Thanks! > Sorin. > > On 5 January 2015 at 18:07, Andy Pellett <a...@embed.ly> wrote: >> >> Hi all, >> >> I've been experiencing stalls where riak won't return any data (queries time >> out) with my riak cluster. Here are some basic details: >> >> - 8 nodes >> - riak 1.4.10 (upgraded from 1.4.6 -> 1.4.8 -> 1.4.10) >> - leveldb backend >> - n_val is 2 >> - allow_mult is false >> - ec2 i2.2xlarge boxes (8 cores, 61gb ram, 800gb disk space) >> - about 33% disk space utilization per node >> >> The riak cluster will stall for as long as a few minutes at a time, but will >> otherwise work as expected for hours. There doesn't seem to be an obvious >> pattern as to when the stalls happen. >> >> My first thought was that the stalls may be related to AAE, but I've >> disabled that via 'riak attach' and the settings file. Sidenote, I still see >> messages like: >> >> 2015-01-05 12:24:04.666 [info] >> <0.574.0>@riak_kv_entropy_manager:perhaps_log_throttle_change:826 Changing >> AAE throttle from 0 -> 10 msec/key, based on maximum vnode mailbox size 209 >> from 'riak-user@riak-host' >> >> which makes me question whether AAE is actually turned off. >> >> Now I'm leaning towards leveldb compactions being the issue. What can I do >> to verify this is the issue, and how can I fix it? >> >> I see log messages about large objects: >> >> 2015-01-05 16:11:28.046 [warning] >> <0.6398.0>@riak_kv_vnode:encode_and_put_no_sib_check:1830 Writing very large >> object (11307735 bytes) to <<"BucketName">>/<<"keys_1420466400">> >> >> Could these be causing longer-running compactions, or more frequent >> compactions? >> >> Thanks for reading, >> Andy >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > > > > -- > Sorin Manole > Senior Software Engineer, Trustev > m:+353 86 051 2658 | e:sorin.man...@trustev.com | w:www.trustev.com| a: > Trustev Ltd, 2100 Airport Business Park, Cork, Ireland. > > This message is for the named person's use only. If you received this message > in error, please immediately delete it and all copies and notify the sender. > You must not, directly or indirectly, use, disclose, distribute, print, or > copy any part of this message if you are not the intended recipient. Any > views expressed in this message are those of the individual sender and not > Trustev Ltd. Trustev is registered in Ireland No. 516425 and trades from 2100 > Cork Airport Business Park, Cork, Ireland. > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
