Daniel,
Something is wrong. All instances of leveldb within a node share the total
memory configuration. The memory is equally divided between all active vnodes.
It is possible to create an OOM situation if total RAM is low and vnodes count
per node is high relative to RAM size.
The best next step would be for you to execute the riak-debug program on one of
the nodes known to experience OOM. Send the resulting .tar.gz file directly to
me (no need to share that with the mailing list). I will review the memory
situation and suggest options.
Matthew
> On Apr 28, 2017, at 8:22 AM, Daniel Miller <dmil...@dimagi.com> wrote:
>
> Hi Luke,
>
> I'm reviving this thread from March where we discussed a new backend
> configuration for our riak cluster. We have had a chance to test out the new
> recommended configuration, and so far we have not been successful in limiting
> the RAM usage of leveldb with multi_backend. We have tried various
> configurations to limit memory usage without success.
>
> First try (default config).
> riak.conf: leveldb.maximum_memory.percent = 70
>
> Second try.
> riak.conf: leveldb.maximum_memory.percent = 40
>
> Third try
> riak.conf: #leveldb.maximum_memory.percent = 40 (commented out)
> advanced.config: [{eleveldb, [{total_leveldb_mem_percent, 30}]}, ...
>
> In all cases (under load) riak consumes all available RAM and eventually
> becomes unresponsive, presumably due to OOM conditions. Is there a way to
> limit the amount of RAM consumed by riak with the new multi_backend
> configuration? For example, do we need to consider ring size or other
> configuration parameters when calculating the value of
> total_leveldb_mem_percent?
>
> Notably, the old (storage_backend = leveldb in riak.conf, empty
> advanced.config) clusters have had very good RAM and disk usage
> characteristics. Is there any way we can make riak or riak cs avoid the rare
> occasions where it overwrites the manifest file while using this (non-multi)
> backend?
>
> Thank you,
> Daniel Miller
>
>
> On Tue, Mar 7, 2017 at 3:58 PM, Luke Bakken <lbak...@basho.com
> <mailto:lbak...@basho.com>> wrote:
> Hi Daniel,
>
> Thanks for providing all of that information.
>
> You are missing important configuration for riak_kv that can only be provided
> in an /etc/riak/advanced.config file. Please see the following document,
> especially the section to which I link here:
>
> http://docs.basho.com/riak/cs/2.1.1/cookbooks/configuration/riak-for-cs/#setting-up-the-proper-riak-backend
>
> <http://docs.basho.com/riak/cs/2.1.1/cookbooks/configuration/riak-for-cs/#setting-up-the-proper-riak-backend>
>
> [
> {riak_kv, [
> % NOTE: double-check this path for your environment:
> {add_paths, ["/usr/lib/riak-cs/lib/riak_cs-2.1.1/ebin"]},
> {storage_backend, riak_cs_kv_multi_backend},
> {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
> {multi_backend_default, be_default},
> {multi_backend, [
> {be_default, riak_kv_eleveldb_backend, [
> {data_root, "/opt/data/ecryptfs/riak"}
> ]},
> {be_blocks, riak_kv_eleveldb_backend, [
> {data_root, "/opt/data/ecryptfs/riak_blocks"}
> ]}
> ]}
> ]}
> ].
>
> Your configuration will look like the above. The contents of this file are
> merged with the contents of /etc/riak/riak.conf to produce the configuration
> that Riak uses.
>
> Notice that I chose riak_kv_eleveldb_backend twice because of the discussion
> you had previously about RAM usage and bitcask
> (http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018801.html
>
> <http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018801.html>)
>
> In your current configuration, you are not using the expected prefix for the
> block data. My guess is that on very rare occasions your data happens to
> overwrite the manifest for a file. You may also have corrupted files at this
> point without noticing it at all.
>
> IMPORTANT: you can't switch from your current configuration to this new one
> without re-saving all of your data.
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
