I did riakc_pb_socket:list_keys/2 which completes quickly, then fetched each
object individually by the key using riakc_pb_socket:get/3 which was slow.
The bucket uses the default settings (see below), allow_mult=false, using a
single bitcask backend.
I came across this problem because one of the Riak nodes was reporting via
Nagios to be short of disk space. Investigating I found that
/var/lib/riak/bitcask was taking up about 24GB of disk. This much disk usage
was crazy big, since there's probably less than 100MB of data I'm storing in
Riak. Since my original email I've deleted all item in the misbehaving
bucket (it took several tries as the read slowness/CPU usage
also appeared on delete operations). The bitcask folder is now 44MB in size
(down from 24GB!). I've repopulated the data in the bucket now storing 157
items. Things seem to be back to normal for this bucket.
Perhaps I inadvertently pushed something large into the bucket, but each of
the items I put in there was only about 3K in size - this doesn't seem
likely to me.
Bucket Properties:
{"props":{"name":"suite_diff","n_val":3,"allow_mult":false,"last_write_wins":false,"precommit":[],"postcommit":[],"chash_keyfun":{"mod":"riak_core_util","fun":"chash_std_keyfun"},"linkfun":{"mod":"riak_kv_wm_link_walker","fun":"mapreduce_linkfun"},"old_vclock":86400,"young_vclock":20,"big_vclock":50,"small_vclock":10,"r":"quorum","w":"quorum","dw":"quorum","rw":"quorum"}}
- Jeremy
On Mon, Sep 12, 2011 at 1:43 PM, Daniel Reverri <d...@basho.com> wrote:
> Hi Jeremy
>
> Are you doing a full bucket query or retrieving individual objects?
>
> Is your bucket configured to allow siblings (allow_mult=true)? Do the
> objects being retrieved have many siblings?
>
> Are you using multiple backends?
>
> Thanks
> Dan
>
> Sent from my iPhone
>
> On Sep 12, 2011, at 10:13 AM, Jeremy Raymond <jeraym...@gmail.com> wrote:
>
> > Hello,
> >
> > I have a bucket with about 70 items in it the Keys look like this
> <<"1294549200_UbEu1topckAFu2UYy3spr8AG5lc">> and the values are about 2.8K
> of JSON. For some reason operations on data in this bucket are extremely
> slow. It takes between 5 - 10 seconds to read a value of of the bucket by
> it's key using the Erlang protobuf client. When I do the reads the CPU of
> the node I'm connecting to spikes using up 100% usage on the Riak beam
> process.
> >
> > I can read data out of any of the other 9 buckets in the system some of
> which have a lot more items in them (one as > 16k items) and the operations
> are super fast.
> >
> > There are no errors in the sasl-error.log files and everything looks
> normal in the erlang.log files. Any ideas on what may cause a bucket to
> churn like this?
> >
> > - Jeremy
> > _______________________________________________
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
