On Jan 22, 2011, at 4:15 PM, Thomas Burdick wrote: > * Why is key listing so slow?
It is slow because, even if the keys are in RAM, you have to scan roughly all of the keys in the cluster to get a listing for a single bucket. As a certain person is fond of saying, "full table scan is full table scan". There are ways to improve this, but without single-arbiters of state (and points of failure) it is very costly. > * What do people do in the context of purely using riak to do what I want, > have a big set of keys to iterate over? As others have said so eloquently, they don't, they use something else. Or they try to minimize how frequently they do it. Part of the current revolution in data storage is about realizing that no one tool is going to completely fit your needs, and that that's good and right. Anyone who tells you otherwise is selling you a bill of goods. To understand why listing keys is difficult, you have to understand Riak's (and Dynamo's) original design motivations: * To be basically available at all times for reads and writes, which in turn means to be tolerant of machine and network failures. * To provide low-latency random access to large data sets. (Note I didn't say an entire data set.) * To scale linearly with minimal operational complexity. Everything has tradeoffs - these are the ones we chose with Riak. Now, we (Basho) are actively trying to create ways to make discovering your data easier (key-filters are one of them, as Justin mentioned we're discussing counters and indices), but the majority of people who use Riak have ways of discovering or knowing keys ahead of time. If that's not your case, you should look into other solutions; some good ones have been mentioned in this thread. That said, we hear your pain and are working hard to improve usability while maintaining the properties discussed above. Cheers, Sean Cribbs <s...@basho.com> Developer Advocate Basho Technologies, Inc. http://basho.com/ _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
