Hi Luke,
I'm reviving this thread from March where we discussed a new backend
configuration for our riak cluster. We have had a chance to test out the
new recommended configuration, and so far we have not been successful in
limiting the RAM usage of leveldb with multi_backend. We have tried various
configurations to limit memory usage without success.
First try (default config).
riak.conf: leveldb.maximum_memory.percent = 70
Second try.
riak.conf: leveldb.maximum_memory.percent = 40
Third try
riak.conf: #leveldb.maximum_memory.percent = 40 (commented out)
advanced.config: [{eleveldb, [{total_leveldb_mem_percent, 30}]}, ...
In all cases (under load) riak consumes all available RAM and eventually
becomes unresponsive, presumably due to OOM conditions. Is there a way to
limit the amount of RAM consumed by riak with the new multi_backend
configuration? For example, do we need to consider ring size or other
configuration parameters when calculating the value of
total_leveldb_mem_percent?
Notably, the old (storage_backend = leveldb in riak.conf, empty
advanced.config) clusters have had very good RAM and disk usage
characteristics. Is there any way we can make riak or riak cs avoid the
rare occasions where it overwrites the manifest file while using this
(non-multi) backend?
Thank you,
Daniel Miller
On Tue, Mar 7, 2017 at 3:58 PM, Luke Bakken <lbak...@basho.com> wrote:
> Hi Daniel,
>
> Thanks for providing all of that information.
>
> You are missing important configuration for riak_kv that can only be
> provided in an /etc/riak/advanced.config file. Please see the following
> document, especially the section to which I link here:
>
> http://docs.basho.com/riak/cs/2.1.1/cookbooks/configuration/
> riak-for-cs/#setting-up-the-proper-riak-backend
>
> [
> {riak_kv, [
> *% NOTE: double-check this path for your environment:*
> {add_paths, ["/usr/lib/riak-cs/lib/riak_cs-2.1.1/ebin"]},
> {storage_backend, riak_cs_kv_multi_backend},
> {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
> {multi_backend_default, be_default},
> {multi_backend, [
> {be_default, riak_kv_eleveldb_backend, [
> {data_root, "/opt/data/ecryptfs/riak"}
> ]},
> {be_blocks, riak_kv_eleveldb_backend, [
> {data_root, "/opt/data/ecryptfs/riak_blocks"}
> ]}
> ]}
> ]}
> ].
>
> Your configuration will look like the above. The contents of this file are
> merged with the contents of /etc/riak/riak.conf to produce the
> configuration that Riak uses.
>
> Notice that I chose riak_kv_eleveldb_backend twice because of the
> discussion you had previously about RAM usage and bitcask (
> http://lists.basho.com/pipermail/riak-users_lists.basho.
> com/2016-November/018801.html)
>
> In your current configuration, you are not using the expected prefix for
> the block data. My guess is that on very rare occasions your data happens
> to overwrite the manifest for a file. You may also have corrupted files at
> this point without noticing it at all.
>
> *IMPORTANT:* you can't switch from your current configuration to this new
> one without re-saving all of your data.
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
