Re: Getting all the Keys

Wow. You really went deep in the way back machine for that gem. @siculars http://siculars.posterous.com Sent from my iRotaryPhone On Apr 25, 2013, at 13:32, n6mac41717 wrote: > Thanks for the quick reply Dmitri! > > Given your insight below, I'm wondering in Riak v2.x, whether it would be >

Re: Getting all the Keys

The feature to automatically index all of the objects in a bucket already exists. If you choose the LevelDB backend and enable Secondary Indexes in the config file, you automatically get the '$bucket' index (see the 'Special Fields' section of http://docs.basho.com/riak/latest/cookbooks/Secondary-

Re: Getting all the Keys

Thanks for the quick reply Dmitri! Given your insight below, I'm wondering in Riak v2.x, whether it would be possible to include a feature that automatically creates the index for you behind the scenes so that indeed GET url/bucket(s) would return the keys. Just a thought... Dmitri Zagidulin wr

Re: Getting all the Keys

In addition, to reiterate what Alexander said in the email thread above, keep in mind that doing a 'list keys' on a bucket forces Riak to iterate through ALL of the keys in a cluster, not just those belonging to a bucket. Meaning, if your cluster has 100 million keys, but a particular bucket has o

Re: Getting all the Keys

Hi Chuck, So there is not currently support for listing keys by just issuing a GET to /buckets/bucketname/. Part of the reason for that is - there's many operations to be performed on the bucket resource -- list keys, get bucket properties, etc. That's why you have several URLs to specify what you

Re: Getting all the Keys

I know it's been over two years since this post, and I'm wondering if the latest version of Riak has made improvements to list keys--I tried the query with "keys=true" and I didn't seem to have TSA/octomom-related wait times. I was originally hoping that I could get a list of keys via the RESTful

Re: Getting all the Keys

Another thing to keep in mind is how you design your keys in Riak. In an RDBMS, it's easy to use an arbitrary surrogate key, like a UUID or identity column and then create secondary indexes on "useful" attributes in order to provide natural lookups. Let's say you have a user. The easiest way to ke

Re: Getting all the Keys

Those are some ways, yes. Generally there are one or more keys that are "known" -- usernames, device identifiers, etc -- which then have references or in some other way inform the application about how to find other keys. This is why links and link-walking were some of the earliest available f

Re: Getting all the Keys

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I'm playing with riak to see what it's good for, so I need to ask: Sean Cribbs wrote: > To understand why listing keys is difficult, you have to understand Riak's > (and Dynamo's) original design motivations: > > * To be basically available at all

Re: Getting all the Keys

I wouldn't do a linked list using Riak links unless it was a small list. With a bytes you can transverse to the node the has a key closest to the left side of your range and then do a in order tranversal of the tree until you reach the right side of your range. It's a lot like a binary search tre

Re: Getting all the Keys

After thinking about it. A b-tree in a bucket wouldn't provide any functionality that I couldn't get from riak search... so that's probably a bad idea. On Jan 22, 2011 3:28 PM, "Eric Moritz" wrote: > I have a pipe dream of doing a distibuted b-tree with a bucket who's value > is a node in the tree

Re: Getting all the Keys

I'd think that you could use a post-commit hook, detect the operation, and then modify the "left" and "right" records to maintain the doubly linked list. The downside would be that you're incurring multiple hits to disk, but you're already incurring multiple hits to disk with an RDBMS when you modi

Re: Getting all the Keys

On 1/22/11 7:46 PM, Eric Moritz wrote: Unlike the web, with Riak, we have complete control over what links to what. Unfortunately it adds complexity to applications that otherwise would be simple in an ACID DB. The benefit to this added complexity is AP. It is usually easy to build a forward-l

Re: Getting all the Keys

phase. Has anyone done something as crazy as a bucket for a b-tree nodes with links for doing inexact range searches? Eric. -- Forwarded message -- From: "Alexander Sicular" Date: Jan 22, 2011 11:31 AM Subject: Re: Getting all the Keys To: "Thomas Burdick" Cc:

Re: Getting all the Keys

As a point of nit picky correctness, the one thing to keep in mind is that the "relational" in "relational database" refers to a relationor relation variable. Which is a table and not the relationship between the tables. The relationship between tab

Re: Getting all the Keys

I have a pipe dream of doing a distibuted b-tree with a bucket who's value is a node in the tree containing a list of keys to another bucket and left and right links to children nodes. It feels right in my head, though it is probably horribly flawed in some way. I'm certain more clever data nerds

Re: Getting all the Keys

This is the best way for me to understand how to model data in Riak. Think about the web. You always have a starting point. The starting point is an URL. An URL is analogous to a key in Riak. A URL gets you a document on the web, a key gets you a document in Riak. Now on the web page addressed by

Re: Getting all the Keys

If you ever want to think about putting indexes in Riak, I played a little thought game and wrote it out on my blog: http://facility9.com/2010/12/16/secondary-indexes-how-would-you-do-it Otherwise - reverse indexes/roll you own b-tree. As an aside, thanks for asking your questions, it prompted m

Re: Getting all the Keys

I mistakenly didn't send a reply to the whole list, but given what everyone is saying I think I "get it" now and the reasoning. Given all of that it seems pretty clear that if I wanted to do what I'm talking about purely in the context of riak using links might work or a bucket containing keys and

Re: Getting all the Keys

On Jan 22, 2011, at 4:15 PM, Thomas Burdick wrote: > * Why is key listing so slow? It is slow because, even if the keys are in RAM, you have to scan roughly all of the keys in the cluster to get a listing for a single bucket. As a certain person is fond of saying, "full table scan is full tabl

Re: Getting all the Keys

I think it's worth mentioning that riak is based on Amazon Dynamo and if you read the paper you'll see Dynamo's use case is lookup by primary key. -Ryan [Sent from my iPhone] On Jan 22, 2011, at 5:43 PM, Neville Burnell wrote: > >As of Riak 0.14 your m/r can filter on key name. I would highly

Re: Getting all the Keys

>As of Riak 0.14 your m/r can filter on key name. I would highly recommend that your data architecture take this into account by using keys that have meaningful names. >>This will allow you to not scan every key in your cluster. Is this part true? I understood that key filtering just means yo

Re: Getting all the Keys

No one seems to have really answered either of my questions in any great detail other than "don't do that" or "use redis" which to me just adds another layer of complexity and potential bugginess to my end application or fails to really describe what the problem is. So really my questions can be b

Re: Getting all the Keys

On 1/22/11 1:45 PM, Gary William Flake wrote: Riak is bad at enumerating keys. If the key isn't something that you can use to retrieve the items you want, what's the point of having it? -- Les Mikesell lesmikes...@gmail.com ___ riak-users ma

Re: Getting all the Keys

On Sat, Jan 22, 2011 at 3:18 PM, Alexander Sicular wrote: > I'll drop a phat tangent and just mention that I watched @rk's talk at Qcon > SF 2010 the other day and am kinda crushing on how they implemented > distributed counters in cassandra (mainlined in 0.7.1 me thinks) which, > imho, is so cho

Re: Getting all the Keys

I don't think it is a flaw at all. Rather I am of the opinion that riak was never meant to do the things we are all talking about in this thread. When I need to do these things I specifically use redis because, as noted, it has tremendous support for specific data structures. When I need

Re: Getting all the Keys

This is a really big pain point for me as well and -- at the risk of prematurely being overly critical of Riak's overall design -- I think it points to a major flaw of Riak in its current state. Let me explain Riak is bad at enumerating keys. We know that. I am happy to manage a list of keys

Re: Getting all the Keys

On Sat, Jan 22, 2011 at 19:34, Alexander Staubo wrote: > On Sat, Jan 22, 2011 at 18:23, Thomas Burdick > wrote: >> So really whats the solution to just having a list of like 50k keys that can >> quickly be appended to without taking seconds to then retrieve later on. Or >> is this just not a vali

Re: Getting all the Keys

On Sat, Jan 22, 2011 at 18:23, Thomas Burdick wrote: > So really whats the solution to just having a list of like 50k keys that can > quickly be appended to without taking seconds to then retrieve later on. Or > is this just not a valid use case for riak at all? That would suck cause > again, I re

Re: Getting all the Keys

If you're looking for a fast, in memory, store that has support for ordered lists you should probably give Redis a look-see. It's an in memory key-value store but it has support for lists as a native data type: http://redis.io/commands#list You could do the same thing in Riak, but you'd be storing

Re: Getting all the Keys

I guess I'm left even more baffled now, if the keys are all in memory and I only have 1 real node in my cluster, why would it take half a second to obtain all the keys from a completely empty database? If it takes half a second to just list the keys out like that how could a map/reduce ever take le

Re: Getting all the Keys

I was going to respond, but I think Alex answered it well with much more humor than I can muster on a good day. All I can add is: - Make sure you're on Riak 0.14. - Take a look at the filter documentation and see how you can clean up your queries

Re: Getting all the Keys

Hi Thomas, This is a topic that has come up many times. Lemme just hit a couple of high notes in no particular order: - If you must do a list keys op on a bucket, you must must must use "?keys=stream". True will block on the coordinating node until all nodes return their keys. Stream will star