Re: Tokyo Cabinet backend

2010-07-23 Thread Jebu Ittiachen
I would love to go with bitcask but its memory usage bloats over time as the number of keys goes up. My use case involves keys which keep on increasing over time. Bitcask seems to be maintaining an in mem structure of key locations. So the quest for a backend which will have a steady memory footpr

Riak Recap for 7/21-7/22

2010-07-23 Thread Mark Phillips
Thanks, guys! Site here ---> http://riakjs.org/ And it also made a bit of splash on Hacker News ---> http://img.skitch.com/20100723-quiq1k9e8yjqybkq734c32ei36.jpg ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Best way to bulk load 1M rows

2010-07-23 Thread golsombe
All, I'm trying to find an optimized method for loading 1Million trending tweets into a single Riak node for map-reduce analysis. I've played with the erlang protobuffer client as well as Ruby Riak-Client, both are too slow. Any ideas? Robert Hall aka Golsombe Sent on the Sprint® Now Network fro

Re: Best way to bulk load 1M rows

2010-07-23 Thread John D. Rowell
Hi Robert, I get about 200 inserts/s with the Ruby client on really low-end hardware (athlon xp2000). This means that in just over 1h you can get 1M tweets up, which is way less time than you'll have to wait for a response from the list. Also you can insert in parallel on different nodes, so if yo

Reduce phase only on one node?

2010-07-23 Thread John Butler
Hello, I went through the Riak Fast Track and overall I'm excited about the possibilities of Riak. However, I did see one thing that concerns me in regard to MapReduce jobs.  From the Riak docs and this page here: http://seancribbs.com/tech/2010/02/06/why-riak-should-power-your-next-rails-app/

Re: Reduce phase only on one node?

2010-07-23 Thread John D. Rowell
+1 to this, my understanding is that you can use the same reduce funcion to re-reduce a stream of data and still get the same results. Is this what actually happens in Riak internally (i.e. the coordinating node only re-reduces each node's reduce) or does the reduce function only run on the coordin

Re: Reduce phase only on one node?

2010-07-23 Thread Dan Reverri
Currently, reduce functions run on the coordinating node. The coordinating node runs 2 processes per reduce phase to achieve some parallelism. There are two features requests open to improve the implementation: Allow users to toggle the number of processes used during reduce https://issues.basho.c

basho bench 64 bit

2010-07-23 Thread Michael Colussi
Has anyone tried compiling basho bench on 64 bit linux? Debian in particular? I get this error when I run make: ./rebar get-deps ==> protobuffs (get-deps) ==> riakc (get-deps) ==> casbench (get-deps) ==> ibrowse (get-deps) ==> stats (get-deps) ==> basho_bench (get-deps) ./rebar compile test escr

Re: basho bench 64 bit

2010-07-23 Thread Grant Schofield
Several of us have run basho_bench on 64bit Linux including Debian and Ubuntu. It looks like the Erlang being used is from your home directory (/home/mcolussi/gateway/cirrus/sdk/cirrus-sdk-linux/lib/erlang), what version of Erlang is this? Have you tried to compile basho_bench with a compiled v

Re: basho bench 64 bit

2010-07-23 Thread Michael Colussi
It was a 32 bit version of R14B04, built from source. The issue was caused by trying to install a 64 bit version of basho bench on top of it. Anyway, I realized that this was silly, and then installed a 64 bit R14B04 build of erlang. Unfortunately now when I run make, basho bench complains that