Hi Luke,
Thanks for the response.
I played around with the various config options - and in the end I had to
include the /etc/defaults/riak file, as well as the
/etc/security/limits.conf config in order to get Riak to stay up. That must
be relatively new - our existing v1.4.0 cluster is operating fine without
it.
Cheers
Matt
On 10 September 2013 01:32, Luke Bakken <lbak...@basho.com> wrote:
> Hi Matt -
>
> Can you attach to a node in your cluster with "riak attach" and run
> this erlang snippet (before the node errors out)?
>
> os:cmd("ulimit -n").
>
> This will show the actual value for ulimit for that running riak node.
> I suspect that this value hasn't been updated for the running process,
> even though you've modified limits.conf.
>
> Depending on your OS there may be more configuration necessary for the
> open files limit to "stick":
> http://docs.basho.com/riak/latest/ops/tuning/open-files-limit/#Linux
>
> On Sun, Sep 8, 2013 at 9:15 PM, Matt Black <matt.bl...@jbadigital.com>
> wrote:
> > Hey list,
> >
> > We're migrating from our currently operational cluster to a new one in a
> > different EC2 region. As part of this migration, we'd like to move from
> > Bitcask to eLevelDB - mostly for the benefits provided by secondary
> indexes.
> >
> > We also use Multi-backend configuration, in order to split our various
> > client's data into entirely separate spaces on disk (this is useful for
> > legal reasons). I made the simple change from "riak_kv_bitcask_backend"
> to
> > "riak_kv_eleveldb_backend" in our config, and did some calculations for
> > "max_open_files", and then started a new node.
> >
> > It fails with the following error in short order:
> >
> >> cat error.log
> > 2013-09-09 03:49:51.391 [error] <0.715.0>@riak_kv_vnode:init:375 Failed
> to
> > start riak_kv_multi_backend Reason:
> [{riak_kv_eleveldb_backend,{db_open,"IO
> > error:
> >
> /mnt/riak/eleveldb/client1/365375409332725729550921208179070754913983135744/000015.dbtmp:
> > Too many open files"}}]
> > 2013-09-09 03:49:51.392 [error] <0.940.0>@riak_kv_vnode:init:375 Failed
> to
> > start riak_kv_multi_backend Reason:
> [{riak_kv_eleveldb_backend,{db_open,"IO
> > error:
> >
> /mnt/riak/eleveldb/client2/570899077082383952423314387779798054553098649600/CURRENT:
> > Too many open files"}},{riak_kv_eleveldb_backend,{db_open,"IO error:
> >
> /mnt/riak/eleveldb/client3/570899077082383952423314387779798054553098649600/CURRENT:
> > Too many open files"}},{riak_kv_eleveldb_backend,{db_open,"IO error:
> >
> /mnt/riak/eleveldb/client4/570899077082383952423314387779798054553098649600/CURRENT:
> > Too many open files"}},{riak_kv_eleveldb_backend,{db_open,"IO error:
> >
> /mnt/riak/eleveldb/client5/570899077082383952423314387779798054553098649600/CURRENT:
> > Too many open files"}},{riak_kv_eleveldb_backend,{db_open,"IO error:
> >
> /mnt/riak/eleveldb/client6/570899077082383952423314387779798054553098649600/CURRENT:
> > Too many open files"}}]
> >
> > And some relevant app.config:
> >
> > {multi_backend_default, <<"default">>},
> > {multi_backend, [
> > %% Default fallback: this is unused
> > {<<"default">>, riak_kv_eleveldb_backend, [
> > {data_root, "/mnt/riak/eleveldb/default"}
> > ]},
> > {<<"eleveldb_client1">>, riak_kv_eleveldb_backend, [
> > {data_root, "/mnt/riak/eleveldb/client1"}
> > ]},
> > {<<"eleveldb_client2">>, riak_kv_eleveldb_backend, [
> > {data_root, "/mnt/riak/eleveldb/client2"}
> > ]},
> > {<<"eleveldb_client3">>, riak_kv_eleveldb_backend, [
> > {data_root, "/mnt/riak/eleveldb/client3"}
> > ]},
> > {<<"eleveldb_client4">>, riak_kv_eleveldb_backend, [
> > {data_root, "/mnt/riak/eleveldb/client4"}
> > ]},
> > {<<"eleveldb_client5">>, riak_kv_eleveldb_backend, [
> > {data_root, "/mnt/riak/eleveldb/client5"}
> > ]},
> > {<<"eleveldb_client6">>, riak_kv_eleveldb_backend, [
> > {data_root, "/mnt/riak/eleveldb/client6"}
> > ]}
> > ]},
> >
> > .. snip ..
> >
> > %% Default cache size of 8MB
> > {cache_size, 8388608},
> > %% Maximum number of files open at once per partition
> > {max_open_files, 50}
> >
> > I've set the "riak soft/hard nofile 65536" in /etc/security/limits.conf,
> so
> > presumably this "Too many open files" error is referring to the
> > "max_open_files" option as part of eLevelDB config. The RAM in each
> machine
> > is 3.5GB free, so I calculated on a 5 node cluster this 50 max_open_files
> > limit.
> >
> > - Is there something about Multi-backend I haven't taken into account?
> > - Do I need a larger max_open_files? (And thus more RAM? :)
> >
> > Cheers!
> > Matt
> >
> > _______________________________________________
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
>
>
> --
> Luke Bakken
> CSE
> lbak...@basho.com
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
