Re: Expected vs Actual Bucket Behavior

2010-07-21 Thread Justin Sheehy
I think that we are all (myself included) getting two different issues a bit mixed up in this discussion: 1: storing an implicit index of keys in the Riak key/value store 2: making buckets separate in that a per-bucket operation's performance would not be affected by the content of other buckets

Re: Expected vs Actual Bucket Behavior

2010-07-21 Thread John D. Rowell
Justin, I think we could address both 1) and 2) in another way. The "real world" need seems to be restricting the scope of costly operations like walking a huge list of keys. Either having distinct buckets or reliable lists of keys could solve the problem. But simply looking up the (Dynamo) herita

Re: Expected vs Actual Bucket Behavior

2010-07-21 Thread Alexander Sicular
Hi Justin, Your comment for issue 480 reads: Implement a separate bitcask backend (riak_kv_bitcask_bucket_backend?) that uses a separate bitcask per-bucket per-partition. What is a partition here? A vnode or a physical host or something else? I appreciate the difficulty of item 1. And would se

Re: Expected vs Actual Bucket Behavior

2010-07-21 Thread Justin Sheehy
Hi, Alexander. On Wed, Jul 21, 2010 at 1:36 PM, Alexander Sicular wrote: > uses a separate bitcask per-bucket per-partition. What is a partition here? A > vnode or a physical host or something else? My apologies. Given that it was in our bugzilla I let myself use some Riak-internals jargon wit

Re: Best way to back-up riak

2010-07-21 Thread Alan McConnell
I'm curious about this as well. Say I have a ten node cluster. Could I just schedule a midnight copy of each bitcask data directory every night, then restore to another ten node cluster by dropping one of each data directories on each new node? How close does the timing needs to be? What if the

Re: Expected vs Actual Bucket Behavior

2010-07-21 Thread Curtis Caravone
Regarding #2, I think bitcask could be modified to support an efficient list keys by bucket fairly easily, without sacrificing free buckets: The current bitcask stores record locators (key, file_id, file_offset) in memory in a big hash table by key (the bitcask key, in Riak's case, is the Riak {bu

Riak Recap for 7/19 - 7/20

2010-07-21 Thread Mark Phillips
Afternoon, Evening, Morning to all, Greetings yet again from OSCON. For today's Recap: A few Riak-related jobs, capacity planning, Riak in Oregon and much more. Enjoy, Mark Community Manager Basho Technologies wiki.basho.com twitter.com/pharkmillups - Riak Recap for 7/19 - 7/20 1) A few

Re: Expected vs Actual Bucket Behavior

2010-07-21 Thread Jason J. W. Williams
Hmm...just created a new account to track this in the Basho bugzilla...seems not to recognize new accounts... -J On Wed, Jul 21, 2010 at 7:31 AM, Justin Sheehy wrote: > I think that we are all (myself included) getting two different issues > a bit mixed up in this discussion: > > 1: storing an i