There are a few things you can do to load data into Riak faster. I blogged about it a while back: http://www.brentozar.com/go/riak-writes/
Basically, set w = 0, dw = 0, and return-body = false. This will effectively throw data into your Riak cluster as fast as possible and not care if the write succeeds. If you actually care about the data, you could set w to 1. --- Jeremiah Peschka Founder, Brent Ozar PLF, LLC On Aug 1, 2011, at 2:13 PM, gtuhl wrote: > I am currently load testing Riak using riak_0.14.2-1_amd64.deb with > fs.file-max set to 503840 for all users. > > I have a reasonably large set of data (hundreds of millions of documents, > many terabytes in size) that is currently stored in a combination of > PostgreSQL+Redis and Disco/DDFS. The first for key/value and the second for > map/reduce to satisfy the full set of user requirements. > > I am trying to consolidate these data sources so trying out a variety of > different data stores with the potential of satisfying both usage types. > > With Riak, my main challenge is getting this data loaded. Using the PHP > library I am able to push 100-200 documents/sec. Is there a recommended > approach to bulk loading data? At that pace it would take a couple months > to load everything. That is not necessarily a deal breaker, but wanted to > sniff around for better options. > > Related to this, I did attempt to break up my records and load them with a > bunch of concurrently running loaders. This actually seems to work fairly > well with not much of a penalty in terms of documents/sec on any single > loader process. But, once I reach 4-5 loaders running concurrently I > consistently get the "Could not contact Riak Server" error and all of my > loader processes die simultaneously. If I wait a few seconds the Riak > server does begin to respond again. > > Any idea for approaching this differently? Is attempting to run many > loaders concurrently a bad idea with Riak? > > I am running a single server right now while I test with bucket nval set to > 1. > > -- > View this message in context: > http://riak-users.197444.n3.nabble.com/Bulk-loading-data-and-Could-not-contact-Riak-Server-error-tp3217091p3217091.html > Sent from the Riak Users mailing list archive at Nabble.com. > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
