I've fooled around with paginating large lists of keys in Riak. A
technique that I have found is to emulate a set using 2i with
Lexicographically sorted keys. You can then emulate a db cursor using
a ranged 2i query to get chunks of values. You'll do repeated queries
to get chunks of data to build up the page much like a B-Tree. You'll
want to make sure your key ranges include enough items so that you're
not doing too many queries but not too large that you are needlessly
fetching too many items.
The easiest example is explain is to use date boxes for the index
value. This is problematic because your chunks are likely not to be
uniform but for the sake of demonstration I'll use dates.
To date box your objects, simply chop off the time part of the
object's ISO8601 creation date:
stories/slug1 = {"creation_date": "2012-07-09T06:00:00Z"}
x-riak-index-date_bucket_bin: 2012-07-09
stories/slug2 = {"creation_date": "2012-07-09T06:00:00Z"}
x-riak-index-date_bucket_bin: 2012-07-09
stories/slug3 = {"creation_date": "2012-07-08T23:00:00Z"}
x-riak-index-date_bucket_bin: 2012-07-08
stories/slug4 = {"creation_date": "2012-07-08T23:30:00Z"}
x-riak-index-date_bucket_bin: 2012-07-08
Then you can put a bound on the query:
"inputs":{
"bucket":"stories",
"index":"date_bucket_bin",
"start":"2012-07-09T00:00:00Z",
"end":"2012-07-09T25:00:00Z"
},
That will get you all the objects for the day of the 9th. If your
page isn't full, you can fetch the objects for the 8th, etc. You'll
want to design things so that your pages are the size of your page or
larger so to prevent round trips to Riak. For instance getting a
1,000 keys and then sorting and limiting them to 20 items in the
reduce phase is nicer than fetching two or three chunks of 20 items so
that you can merge sort and limit them client side.
For a previous page you'll want to start with the last item on the
first page to get the next 20 items, etc.
Eric.
On Thu, Jul 19, 2012 at 10:33 AM, Alex Thompson <godfo...@acis.ufl.edu> wrote:
> I agree, but sometimes you make sacrifices. ;)
>
> Alternatively, you could use something like redis or memcache to hold
> keylists, either populated dynamically with key lists, or loaded with data on
> startup. You might even be able to get away with just some reasonably complex
> send-current-and-refresh style page caching and a script to warm your caches
> from a cold start.
>
> Riak is a wonderful, scalable database, but unless you're holding lots data
> in ram _somewhere_, your service is never really going to run at interactive
> speeds. This is true of all data systems, not just riak, consider highly
> random access to a traditional RDBMS with data > ram and no indexes.
>
> - Alex
>
> ----- Original Message -----
> From: "Martin Stabenfeldt" <mar...@stabenfeldt.net>
> To: riak-users@lists.basho.com
> Sent: Thursday, July 19, 2012 3:15:25 AM
> Subject: Re: Pagination with Ripple
>
>
>
> Hi,
>
>
> We´re also in the hundreds of thousands scale. So loading everything is
> unfortunately not possible.
> Would be nice if one could settle for one database, instead of mixing
> different types. Nice to only care about scaling and providing redundancy for
> Riak, instead of Riak and PostgreSQL :-)
>
>
> If anyone has implemented pagination using Ripple or other solutions with
> Ruby, I´d love to see it!
>
>
>
>
> Cheers,
>
>
> Martin Stabenfeldt
>
>
>
> On Wednesday, 18 July 2012 at 16:24, Alex Thompson wrote:
>
>
>
>
>
> My 2c:
>
>
> 1000s of subscribers per list isn't a problem. Just have a data object which
> represents the subscriber list, store a list of subscriber ids in the
> subscriber list object. Load the whole list every time, and sort/paginate in
> your app code.
>
>
> Unless you're planning on shuffling the lists a lot, that will be the most
> direct way to implement it.
>
>
> Caveats:
> Unless you're planning on running into the object size limit with subscriber
> ids or planning on being able to handle thousands of simultaneous
> subscriptions to the same list, but don't want to handle conflict resolution.
>
>
> My solution:
> I ran into this same problem with my app, but with hundreds of thousands to
> millions of uuids per list. I tried a couple of Riak based solutions with
> different data formats, but I wasn't really satisfied with any of them. I
> ended up just having a uuid to uuid lookup table in a postgres database to
> handle internal item to item relations using the uuid datatype for efficient
> indexing.
>
>
> I'm not sure if there are any good support libraries for handling polyglot
> consistency in ruby, or if you could blend database ActiveRecord and ripple
> models in the same app (I mean, you probably can, right? You'd just have
> objects inherit from different base classes? There are probably some
> unhandled relation-chaining edge cases though... although ActiveRecord can be
> magic sometimes.)
>
>
> - Alex
>
>
> ----- Original Message -----
> From: "Martin Stabenfeldt" < mar...@stabenfeldt.net >
> To: riak-users@lists.basho.com
> Sent: Wednesday, July 18, 2012 2:54:35 AM
> Subject: Pagination with Ripple
>
>
>
>
> Dear List,
>
>
>
>
> I´m planning to use Riak as my primary DB for my Rails app.
>
>
>
>
> The first issue I´ve encountered is pagination.
>
>
>
>
> I got a SubscriberList which may contain 1000´s of Subscribers. Any
> suggestion on how I can implement pagination? My plan was to use
> will_paginate .
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> class Subscriber
> include Ripple :: Document
>
>
>
>
> # Belongs to a SubscriberList
> one :subscriber_list
> property :subscriber_list_key , String
> property :email , String
> property :referer , String
> timestamps!
>
>
>
>
> validates_presence_of :email
> end
>
>
>
>
>
>
>
>
>
>
>
>
> class SubscriberList
> include Ripple :: Document
> one :user
> property :user_key , String
>
>
>
>
> # Has_many Subscribers
> many :subscribers , :using => :reference
> # Used by User to find which SubscriberList he has
> property :subscriber_key , String
> property :title , String
> property :deleted , Boolean
> timestamps!
>
>
>
>
> validates_presence_of :user
> end
>
>
>
>
> --
> Martin Stabenfeldt
> Tlf: +47 93441707
>
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
