You can already use Riak KV as persistence for Hadoop (although it can be a bit intensive on IO), I know of one production setup already doing this. Riak Pipe on the other hand, could be used to build jobs like you would in Hadoop, with greater independence and flexibility than Riak KV's standard MapReduce gives you. It's still very new, so we're not sure of all the ins-and-outs of supporting it, but we encourage you to give it a try.
Sean Cribbs <s...@basho.com> Developer Advocate Basho Technologies, Inc. http://basho.com/ On Jun 13, 2011, at 9:26 PM, Antonio Rohman Fernandez wrote: > Can Riak Pipe be used with Hadoop? That would be wonderful! > > Rohman > > Sent from my iPad > > On Jun 14, 2011, at 12:20 AM, Bryan Fink <br...@basho.com> wrote: > >> Hello again, Community. >> >> I'm excited to announce the opening of a new beta-status Basho project >> today: Riak Pipe. >> >> http://github.com/basho/riak_pipe >> >> Riak Pipe is a new way to distribute work around a Riak cluster. >> >> The README explains much more than I can here, but essentially Riak >> Pipe allows you to specify work in the form of a chain of function >> pairs. One function of that pair describes how to produce output from >> input, and the other describes where in the cluster an input should be >> processed. Riak Pipe handles the details of ferrying data between >> workers by building atop Riak Core's distribution power. >> >> At this point in time Riak Pipe is BETA-status software. We'd like >> anyone who is interested in it to take a look and send us feedback. >> Please do not put it into production. We will be continuing to >> improve Riak Pipe toward a future release date. >> >> We have two plans for Riak Pipe. The first is to power Riak's >> MapReduce system with it. We think Riak Pipe provides a cleaner, more >> manageable subsystem that will provide much easier monitoring, >> debugging, and general use of MapReduce in Riak. You can see our work >> toward that goal in the "pipe" branch of Riak KV (start at >> src/riak_kv_mrc_pipe.erl): >> >> https://github.com/basho/riak_kv/tree/pipe >> >> Our second plan for Riak Pipe is to expand Riak's MapReduce system >> with more abilities (imagine a keyed-reduce phase, or additional >> processing languages), possibly to the extent of providing an entirely >> separate interface (new query syntax? offline/asynchronous >> processing?). But for this part, we need your help. >> >> We have some ideas about what external client interfaces might look >> like. We also have some ideas about what an external processing >> interface might look like. We're still in the early phases of >> creating these, though, so if exploring the riak_pipe repository gives >> you ideas, please don't hesitate to get in touch. >> >> And, again, Riak Pipe is BETA software. Basho does not support >> running it in production at this time. >> >> Cheers, >> >> Bryan Fink >> Senior Software Engineer >> Basho Technologies >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
