I was able to achieve similar results. I wrote a Ruby process that would keep at most n (I think n = 10) things at once and reached 2,500ish req/s on my macbook pro.
I loaded data to a cluster of six Riak nodes by running several of these processes at once and attaching each to a different Riak node, and I hit 18,000 req/s. I'm not sure whether loading different nodes affected the speed or not, now that I think of it. 2011/11/14 Russell Brown <russel...@basho.com> > > On 14 Nov 2011, at 11:47, Nitish Sharma wrote: > > > Hi, > > This is more sort of a discussion than a question. I am just trying to > see the trend in how users import their data to Riak. > > For the data I am using, I was able to achieve almost 150 records/second > with PHP library, and 400 records/second with node.js (fairly new with > node; was hitting memory wall when trying to import 1 million records). > > What are some hacks/tricks/tweaks to import large amount of data to Riak? > > New keys, new data, straight in for the first time, no fetch before store? > I've had reasonable results creating a *number* of threads and using the > Java Raw PB client to write. > > For example, maybe have a 1 or a couple of threads that reads data (from > Oracle, a file, what-have-you) and puts it on a queue, and have a bunch of > threads that pull data off the queue, create a riak object and store it. > From my laptop I've got up to 2500 writes a second like this, and it was > just ad hoc, throw away code with 4 threads against a small 3 node cluster > (running on desktops.) > > I imagine others on the list have more direct, real world examples? > > Cheers > > Russell > > > > > Cheers > > Nitish > > _______________________________________________ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
