Hi Ryan, Marc,
thanks for your answer with a bunch of useful info. I should have
introduced ourselves better given your replies. In fact we authored the
paper on CRDTs that Ryan mentioned and we continue to work in this area.
I am putting some relevant people in CC. What we try now is to
experiment with CRDTs on top of Riak.
More recently, we came up with a class of CRDT data types that mix
state-based and log-based propagation of updates. This is why I asked
about the feasibility of "delta-updates" implementation.
Ryan, I got the idea you confirmed - sending delta as object value and
recomputing the real value using (hackish) pre-commit hook at the
coordinator node. It seems though the overhead of shipping (and
logging?) big objects to each data node can render the trick not worth
itself. Given your knowledge on Riak internals, would it be difficult to
modify Riak to support hooks at data nodes? I understand there is going
to be some subtle ordering cases of versions and it is probably not a
common feature everybody are looking for, rather a hack. It could help
us though, at least for sake of the experiments.
Marek
On 16.01.2012 15:30, Mark Steele wrote:
I for one would love to see a CRDT-like interface in Riak...
Cheers,
Mark Steele, CISSP, CSM
Bering Media Inc.
On Sat, Jan 14, 2012 at 1:03 PM, Ryan Zezeski <rzeze...@basho.com
<mailto:rzeze...@basho.com>> wrote:
Marek,
You're definitely butting up against the limits of Riak's data model
here. It sounds like you are looking for something like Redis?
E.g. add element E to set S. You could probably use statebox [1]
and pull some tricks in the precommit hook to determine if the
incoming client object is the initial object or a delta but you
would still need to perform a read in the hook to get the box, apply
the delta, and then return that object from the hook. As you said
this happens on the coordinator and there is no easy way to
guarantee this op applies local to the data off the top of my head.
At least a precommit hook would prevent streaming the object
to/from the client.
I think what you really want is native support for different data
models on top of Riak. Internally, we've played around with a CRDT
[2] interface on top of Riak where instead of sending objects you
send operations. This is analogous to how Redis works (although the
underlying implementation is very different given Riak's distributed
nature). The trick with a new data model is making sure it scales
and making sure we understand its consistency model.
Riak's pedigree is the key-value model. I'm sure you could hack
something together on top of Riak's KV model with precommit hooks
but you will be swimming upstream. That said, we're always looking
at new models that would compliment our existing key-value model.
-Ryan
[1]: https://github.com/mochi/statebox
[2]: http://hal.archives-ouvertes.fr/inria-00555588/
On Wed, Jan 4, 2012 at 10:11 AM, Marek Zawirski
<marek.zawir...@lip6.fr <mailto:marek.zawir...@lip6.fr>> wrote:
Hi,
we are trying use Riak as a storage layer for experimental
higher-level data types updated by clients, using a set of
well-defined operations. To this end, each data type instance is
stored under a single key. One problem with this approach is that
after client modifies even a small piece of the data structure, it
needs to write (and transfer) the whole data structure back to Riak.
We are looking for a way to reduce this overhead by sending just a
delta operation, preferably without partitioning the data
structure to
several keys.
One approach we thought about is to perform operations on Riak-side
using pre-commit hooks or similar technique. I.e. reconstruct
the new
value on Riak using original old value + delta send by client. The
operations (deltas) we are talking about have necessary
properties to
ensure convergence. Still, it seems there a couple technical issues
involved, we are looking how to solve them:
1) pre-commit API seems to only offer access to the object value
passed by client during write and not the old value; I wonder - am I
able to read the value from the store in pre-commit hook? In
particular, the value previously read by the client writing,
identified by version vector?
2) pre-commit hooks are executed on the coordinator node; is
there an
easy way in Riak to apply operations at data nodes instead?
Thanks for any info that can help addressing these issues.
Regards,
Marek Zawirski
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
