Hi Justin, Your comment for issue 480 reads: Implement a separate bitcask backend (riak_kv_bitcask_bucket_backend?) that uses a separate bitcask per-bucket per-partition. What is a partition here? A vnode or a physical host or something else? I appreciate the difficulty of item 1. And would second your solution to item 2. Item 2 is a big concern for those of use looking to take advantage of riak as a primary datastore for realtime or near realtime systems. Perhaps piggy backing on some of the multibackend code you have already put in place. Perhaps having multiple bitcask instances per bucket per partition could also help with memory management in that if one bucket were used primarily as a WOL all the keys wouldnt need to be memory resident all the time.
Thank you, Alexander On Jul 21, 2010, at 9:31 AM, Justin Sheehy wrote: > I think that we are all (myself included) getting two different issues > a bit mixed up in this discussion: > > 1: storing an implicit index of keys in the Riak key/value store > > 2: making buckets separate in that a per-bucket operation's > performance would not be affected by the content of other buckets > > The thread started out with a request for #2, but included a > suggestion to do #1. These are actually two different topics. > > The first issue, implicitly storing a big index of keys, is > impractical in a distributed key/value storage system that has Riak's > availability goals. We are very unlikely to implement this as > described in the near future. However, we very much recognize that > there are many different ways that people would like to find their > data. In that light, we are working on multiple different efforts > that will use the Riak core to provide data storage with more than > just "simple" key/value access. > > The second issue, of isolating buckets, is a much simpler design > choice and is also a per-backend implementation detail. We can create > and provide an alternative bitcask adapter that does this. It will be > a real tradeoff: in exchange for buckets not impacting each other as > much, the system will consume more filehandles, be a bit less > efficient at rebalancing, and will generally make buckets no longer > "free". This is a reasonable tradeoff in either direction for various > applications, and I support making it available as a choice. I have > created a bugzilla entry to track it: > https://issues.basho.com/show_bug.cgi?id=480 > > I hope that this helps to clarify the issue. > > -Justin > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
