On 14 Nov 2011, at 11:47, Nitish Sharma wrote: > Hi, > This is more sort of a discussion than a question. I am just trying to see > the trend in how users import their data to Riak. > For the data I am using, I was able to achieve almost 150 records/second with > PHP library, and 400 records/second with node.js (fairly new with node; was > hitting memory wall when trying to import 1 million records). > What are some hacks/tricks/tweaks to import large amount of data to Riak?
New keys, new data, straight in for the first time, no fetch before store? I've had reasonable results creating a *number* of threads and using the Java Raw PB client to write. For example, maybe have a 1 or a couple of threads that reads data (from Oracle, a file, what-have-you) and puts it on a queue, and have a bunch of threads that pull data off the queue, create a riak object and store it. From my laptop I've got up to 2500 writes a second like this, and it was just ad hoc, throw away code with 4 threads against a small 3 node cluster (running on desktops.) I imagine others on the list have more direct, real world examples? Cheers Russell > > Cheers > Nitish > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
