I for one would love to see a CRDT-like interface in Riak... Cheers,
Mark Steele, CISSP, CSM Bering Media Inc. On Sat, Jan 14, 2012 at 1:03 PM, Ryan Zezeski <rzeze...@basho.com> wrote: > Marek, > > You're definitely butting up against the limits of Riak's data model here. > It sounds like you are looking for something like Redis? E.g. add element > E to set S. You could probably use statebox [1] and pull some tricks in > the precommit hook to determine if the incoming client object is the > initial object or a delta but you would still need to perform a read in the > hook to get the box, apply the delta, and then return that object from the > hook. As you said this happens on the coordinator and there is no easy way > to guarantee this op applies local to the data off the top of my head. At > least a precommit hook would prevent streaming the object to/from the > client. > > I think what you really want is native support for different data models > on top of Riak. Internally, we've played around with a CRDT [2] interface > on top of Riak where instead of sending objects you send operations. This > is analogous to how Redis works (although the underlying implementation is > very different given Riak's distributed nature). The trick with a new data > model is making sure it scales and making sure we understand its > consistency model. > > Riak's pedigree is the key-value model. I'm sure you could hack something > together on top of Riak's KV model with precommit hooks but you will be > swimming upstream. That said, we're always looking at new models that > would compliment our existing key-value model. > > -Ryan > > [1]: https://github.com/mochi/statebox > > [2]: http://hal.archives-ouvertes.fr/inria-00555588/ > > On Wed, Jan 4, 2012 at 10:11 AM, Marek Zawirski <marek.zawir...@lip6.fr>wrote: > >> Hi, >> >> we are trying use Riak as a storage layer for experimental >> higher-level data types updated by clients, using a set of >> well-defined operations. To this end, each data type instance is >> stored under a single key. One problem with this approach is that >> after client modifies even a small piece of the data structure, it >> needs to write (and transfer) the whole data structure back to Riak. >> We are looking for a way to reduce this overhead by sending just a >> delta operation, preferably without partitioning the data structure to >> several keys. >> >> One approach we thought about is to perform operations on Riak-side >> using pre-commit hooks or similar technique. I.e. reconstruct the new >> value on Riak using original old value + delta send by client. The >> operations (deltas) we are talking about have necessary properties to >> ensure convergence. Still, it seems there a couple technical issues >> involved, we are looking how to solve them: >> 1) pre-commit API seems to only offer access to the object value >> passed by client during write and not the old value; I wonder - am I >> able to read the value from the store in pre-commit hook? In >> particular, the value previously read by the client writing, >> identified by version vector? >> 2) pre-commit hooks are executed on the coordinator node; is there an >> easy way in Riak to apply operations at data nodes instead? >> >> Thanks for any info that can help addressing these issues. >> >> Regards, >> Marek Zawirski >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
