Awesome.. I'll quickly hack out what I need for the quick import, and hope you can release your re-write.
Gui Pinto Software Engineer at Chitika On Fri, Apr 8, 2011 at 12:26 PM, Mark Steele <mste...@beringmedia.com>wrote: > I've re-written the php library to use keep alives (and various other > tweaks). Let me see what I can do about releasing the code. > > The current php library simply instantiates a new curl instance for each > request, making it less than optimal. > > Mark Steele > Bering Media Inc. > > > On Fri, Apr 8, 2011 at 12:23 PM, Gui Pinto <gpi...@chitika.com> wrote: > >> Hey Everyone, thanks for all of the recommendations. >> >> I've tried importing using the example load_data >> script<http://wiki.basho.com/Loading-Data-and-Running-MapReduce-Queries.html>available >> on the Fast Track, and have last tried the PHP library. >> >> Both of these execute a straight-foward CURL -X PUT request.. which makes >> me think Mark might have just guessed it.. >> Keep-alive not being used definitely explains the 200-writes/second cap. >> >> I'm going to take a look into the PHP library and test this theory. >> >> Gui Pinto >> Software Engineer at Chitika >> >> >> >> On Fri, Apr 8, 2011 at 10:01 AM, Mark Steele <mste...@beringmedia.com>wrote: >> >>> If using HTTP, make sure you're using keep-alives. That will be a >>> gigantic speed boost. >>> >>> The protocol buffer API is much faster if you're client language supports >>> it. >>> >>> >>> Mark Steele >>> Bering Media Inc. >>> >>> >>> >>> On Thu, Apr 7, 2011 at 10:58 PM, matthew hawthorne <mhawtho...@gmail.com >>> > wrote: >>> >>>> Hi Gui, >>>> >>>> I recently pushed 70 million records of size 1K each into a 5-node >>>> Riak cluster (which was replicating to another 5-node cluster) at >>>> around 1000 writes/second using basho_bench and the REST interface. I >>>> probably could have pushed it further, but I wanted to confirm that it >>>> could maintain the load for the entire data set, which it did. >>>> >>>> My point being that your speed-limit of 200 writes/second is likely >>>> specific to your configuration. >>>> >>>> I wonder: >>>> 1) what's your average write latency? >>>> 2) how big is your connection pool? >>>> >>>> Because it's possible that you don't have enough connections available >>>> to handle your desired load. >>>> >>>> -matt >>>> >>>> >>>> On Thu, Apr 7, 2011 at 6:01 PM, Gui Pinto <gpi...@chitika.com> wrote: >>>> > Hey guys, >>>> > I'm attempting to importing 300M+ objects into a Riak cluster, but >>>> have >>>> > quickly reached the REST API's speed-limit at 200-store()'s per >>>> second.. >>>> > At the rate of 200/s, I'm looking at 20-days to import this data set! >>>> That >>>> > can't be the fastest method to do this.. >>>> > >>>> > Any recommendations? >>>> > >>>> > Thanks! >>>> > Gui Pinto >>>> > >>>> > _______________________________________________ >>>> > riak-users mailing list >>>> > riak-users@lists.basho.com >>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> > >>>> > >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>> >>> >> >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
