Justin, I think we could address both 1) and 2) in another way. The "real world" need seems to be restricting the scope of costly operations like walking a huge list of keys. Either having distinct buckets or reliable lists of keys could solve the problem.
But simply looking up the (Dynamo) heritage for Riak shows that this is a solved problem in other platforms. For instance Amazon's S3 service does actual separation of keys per bucket (which is what improving Bitcask in the way you described will achieve) but also has a more subtle, and complementary solution: prefixes. When you search inside a bucket you can specify a key prefix that will serve as a filter. While I have no idea if their internal implementation of prefixes is efficient, I see no reason why we can't have the same functionality in Riak and guarantee it is efficient even in distributed cases. If keys are limited to some manageable length (say 255 char strings) they could be indexed into a distributed (only eventually consistent of course) ordered list or btree that could be efficiently queried for ranges. Then to the application writer it's just a matter of providing sane keys that will allows for scoping searchs in several ways (e.g. per year, per year+month, per lot, etc.) The suggestion above would not solve all indexing needs but would surely allow for the current ones to be refactored into cases that would be efficiently solvable. -jd 2010/7/21 Justin Sheehy <jus...@basho.com> > I think that we are all (myself included) getting two different issues > a bit mixed up in this discussion: > > 1: storing an implicit index of keys in the Riak key/value store > > 2: making buckets separate in that a per-bucket operation's > performance would not be affected by the content of other buckets > > The thread started out with a request for #2, but included a > suggestion to do #1. These are actually two different topics. > > The first issue, implicitly storing a big index of keys, is > impractical in a distributed key/value storage system that has Riak's > availability goals. We are very unlikely to implement this as > described in the near future. However, we very much recognize that > there are many different ways that people would like to find their > data. In that light, we are working on multiple different efforts > that will use the Riak core to provide data storage with more than > just "simple" key/value access. > > The second issue, of isolating buckets, is a much simpler design > choice and is also a per-backend implementation detail. We can create > and provide an alternative bitcask adapter that does this. It will be > a real tradeoff: in exchange for buckets not impacting each other as > much, the system will consume more filehandles, be a bit less > efficient at rebalancing, and will generally make buckets no longer > "free". This is a reasonable tradeoff in either direction for various > applications, and I support making it available as a choice. I have > created a bugzilla entry to track it: > https://issues.basho.com/show_bug.cgi?id=480 > > I hope that this helps to clarify the issue. > > -Justin > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
