I haven't tested net/http, I couldn't get it to work when passing a block to keys. I am pretty sure this is the same issue I mentioned on 5/4 and you reported as a bug. It only seems to happen with large numbers of keys. Riak will return partial JSON objects and they are rejected in ripple because they are invalid.
Thanks, Adam On May 12, 2010, at 11:17 PM, Sean Cribbs wrote: > If the problem is with net/http then we need to fix that backend, not apply > hacks to the generic portions. You can verify that Riak will return valid > JSON objects in each chunk by adding --raw to the curl command. Here's what > I get after loading the Fast Track stock data: http://gist.github.com/399450 > > Sean Cribbs <s...@basho.com> > Developer AdvocateI > Basho Technologies, Inc. > http://basho.com/ > > On May 12, 2010, at 10:55 PM, Adam Hunter wrote: > >> This is a confirmed bug and will be fixed soon. You can check out >> http://github.com/adamhunter/ripple/blob/associations/lib/riak/bucket.rb : >> 65 for a temporary hack. >> >> Thanks, >> >> Adam >> >> On May 12, 2010, at 10:17 PM, Andrew Harvey wrote: >> >>> I've just run into this one as well. >>> >>> >>> Not sure if it's a curb thing or what, but it's making me a sad panda. We >>> have 500k+ keys in our bucket, and I'm being returned chunks of invalid >>> json so nothing is being yielded to my block. Needless to say, the task I'm >>> doing here I'd prefer to do in a MapReduce, but unfortunately I need to get >>> everything out into ruby-land for the moment. >>> >>> Andrew >>> >>> >>> On 04/05/2010, at 11:46 PM, Sean Cribbs wrote: >>> >>>> This looks like a bug. I'll add an issue to the github tracker. >>>> >>>> Sean Cribbs <s...@basho.com> >>>> Developer Advocate >>>> Basho Technologies, Inc. >>>> http://basho.com/ >>>> >>>> On May 4, 2010, at 9:29 AM, Adam Hunter wrote: >>>> >>>>> Just curb, I couldn't get net/http to work when doing chunked (something >>>>> about read_body being called twice). >>>>> >>>>> >>>>> On May 4, 2010, at 9:28 AM, Sean Cribbs wrote: >>>>> >>>>>> Are you using curb or net/http? >>>>>> >>>>>> Sean Cribbs <s...@basho.com> >>>>>> Developer Advocate >>>>>> Basho Technologies, Inc. >>>>>> http://basho.com/ >>>>>> >>>>>> On May 4, 2010, at 9:26 AM, Adam Hunter wrote: >>>>>> >>>>>>> The problem I was running into was some of the chunks were only part of >>>>>>> the json object. Decoded it would be something like this: >>>>>>> >>>>>>> chunk 1: {'keys':[1,2,3,4 >>>>>>> chunk 2: 5,6,7,8,9]} >>>>>>> >>>>>>> I was running this on a bucket with about 11k keys. Using ripple I >>>>>>> could only get about 150 keys back this way because all the other >>>>>>> chunks were not valid json. I hacked around this for now by checking >>>>>>> to see if the chunk ends in ]}, otherwise I accumulate chunks until I >>>>>>> have a full json object. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Adam >>>>>>> >>>>>>> >>>>>>> On May 4, 2010, at 9:22 AM, Sean Cribbs wrote: >>>>>>> >>>>>>>> This behavior is unchanged since its implementation in 0.8 or 0.9. To >>>>>>>> better demonstrate what that statement means, here's the joined output >>>>>>>> of a streamed keys response: >>>>>>>> >>>>>>>> {"props":{"name":"foo","n_val":3,"allow_mult":false,"precommit":[],"postcommit":[],"chash_keyfun":{"mod":"riak_core_util","fun":"chash_std_keyfun"},"linkfun":{"mod":"riak_kv_wm_link_walker","fun":"mapreduce_linkfun"},"old_vclock":86400,"young_vclock":20,"big_vclock":50,"small_vclock":10}}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]} >>>>>>>> >>>>>>>> If your client library supports per-chunk processing, each of those >>>>>>>> {"keys":[]} objects will be an encoded chunk. The Ruby client and >>>>>>>> most of the other clients support this. >>>>>>>> >>>>>>>> Sean Cribbs <s...@basho.com> >>>>>>>> Developer Advocate >>>>>>>> Basho Technologies, Inc. >>>>>>>> http://basho.com/ >>>>>>>> >>>>>>>> On May 3, 2010, at 9:28 PM, Adam Hunter wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> I apologize if this has been covered already. >>>>>>>>> >>>>>>>>> In the wiki it says "If keys=stream, the response will be transferred >>>>>>>>> using chunked-encoding, where each chunk is a JSON object" (on >>>>>>>>> http://wiki.basho.com/display/RIAK/REST+API) >>>>>>>>> >>>>>>>>> This doesn't seem to be the case, sometimes a json object is broken >>>>>>>>> into several chunks. Is this behavior correct? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Adam >>>>>>>>> _______________________________________________ >>>>>>>>> riak-users mailing list >>>>>>>>> riak-users@lists.basho.com >>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>>> >>>> Click here to report this email as spam. >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >>> Andrew Harvey / Developer >>> lexer >>> >>> m/ >>> t/ +61 2 9019 6379 >>> w/ http://lexer.com.au >>> >>> Help put an end to whaling. Visit www.givewhalesavoice.com.au >>> >>> >>> >>> Please consider the environment before printing this email >>> This email transmission is confidential and intended solely for the person >>> or organisation to whom it is addressed. If you are not the intended >>> recipient, you must not copy, distribute or disseminate the information, or >>> take any action in relation to it and please delete this e-mail. Any views >>> expressed in this message are those of the individual sender, except where >>> the send specifically states them to be the views of any organisation or >>> employer. If you have received this message in error, do not open any >>> attachment but please notify the sender (above). This message has been >>> checked for all known viruses powered by McAfee. >>> >>> For further information visit >>> http://www.mcafee.com/us/threat_center/default.asp >>> Please rely on your own virus check as no responsibility is taken by the >>> sender for any damage rising out of any virus infection this communication >>> may contain. >>> >>> >>> This message has been scanned for malware by Websense. www.websense.com >>> >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
