On Tue, Aug 30, 2011 at 5:31 PM, David Koblas <da...@koblas.com> wrote:
>
> The one issue that I see with LevelDB in a big insert job like this is file
> descriptors, I've currently got about 9,000 files open on the machine, I'm
> guessing that I'll run out of file descriptors before the load is done. Is
> there a configuration parameter to change the size of the SST files before
> they're rolled?
No, you can't control SST rollover (that I've seen), but you can
control how many file descriptors are in use, per-vnode. In the
section for eleveldb in app.config add the directive:
{max_open_files, N}
Note that this will restrict number of open handles _per vnode_, so
you'll still need to ensure that your ulimit -n is set higher than #
of vnodes * max_open_files. Also note that this may affect performance
if set too low, since it will require the system to open/close files
rapidly.
Ultimately, the reason innostore was slow was likely due to the fact
that if you were not inserting in sorted order, it was a worst case
scenario for updating the b-tree. LevelDB gets around this by
discarding the b-tree idea altogether in exchange for slightly higher
whole-tree traversal and less cache-locality in that scenario. Seems
like a reasonable tradeoff in all the testing I've done. :)
D.
--
Dave Smith
Director, Engineering
Basho Technologies, Inc.
diz...@basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
