The chunk size-splitting makes sense, although it is irritating to hack around it. How many partitions are you using in your cluster?
Sean Cribbs <s...@basho.com> Developer Advocate Basho Technologies, Inc. http://basho.com/ On May 12, 2010, at 11:26 PM, Andrew Harvey wrote: > I believe it's a curb bug. > > > curl --raw gives me expected output, but the chunks yielded by the > @client.http.get are not the same chunks shown by curl – they're fragmented > as shown earlier in this thread., that said the chunks returned are pretty > huge (1.2 MiB), which might explain why curb is cutting them up. > > Andrew > > > On 13/05/2010, at 1:17 PM, Sean Cribbs wrote: > >> If the problem is with net/http then we need to fix that backend, not apply >> hacks to the generic portions. You can verify that Riak will return valid >> JSON objects in each chunk by adding --raw to the curl command. Here's what >> I get after loading the Fast Track stock data: http://gist.github.com/399450 >> >> Sean Cribbs <s...@basho.com> >> Developer Advocate >> Basho Technologies, Inc. >> http://basho.com/ >> >> On May 12, 2010, at 10:55 PM, Adam Hunter wrote: >> >>> This is a confirmed bug and will be fixed soon. You can check out >>> http://github.com/adamhunter/ripple/blob/associations/lib/riak/bucket.rb : >>> 65 for a temporary hack. >>> >>> Thanks, >>> >>> Adam >>> >>> On May 12, 2010, at 10:17 PM, Andrew Harvey wrote: >>> >>>> I've just run into this one as well. >>>> >>>> >>>> Not sure if it's a curb thing or what, but it's making me a sad panda. We >>>> have 500k+ keys in our bucket, and I'm being returned chunks of invalid >>>> json so nothing is being yielded to my block. Needless to say, the task >>>> I'm doing here I'd prefer to do in a MapReduce, but unfortunately I need >>>> to get everything out into ruby-land for the moment. >>>> >>>> Andrew >>>> >>>> >>>> On 04/05/2010, at 11:46 PM, Sean Cribbs wrote: >>>> >>>>> This looks like a bug. I'll add an issue to the github tracker. >>>>> >>>>> Sean Cribbs <s...@basho.com> >>>>> Developer Advocate >>>>> Basho Technologies, Inc. >>>>> http://basho.com/ >>>>> >>>>> On May 4, 2010, at 9:29 AM, Adam Hunter wrote: >>>>> >>>>>> Just curb, I couldn't get net/http to work when doing chunked (something >>>>>> about read_body being called twice). >>>>>> >>>>>> >>>>>> On May 4, 2010, at 9:28 AM, Sean Cribbs wrote: >>>>>> >>>>>>> Are you using curb or net/http? >>>>>>> >>>>>>> Sean Cribbs <s...@basho.com> >>>>>>> Developer Advocate >>>>>>> Basho Technologies, Inc. >>>>>>> http://basho.com/ >>>>>>> >>>>>>> On May 4, 2010, at 9:26 AM, Adam Hunter wrote: >>>>>>> >>>>>>>> The problem I was running into was some of the chunks were only part >>>>>>>> of the json object. Decoded it would be something like this: >>>>>>>> >>>>>>>> chunk 1: {'keys':[1,2,3,4 >>>>>>>> chunk 2: 5,6,7,8,9]} >>>>>>>> >>>>>>>> I was running this on a bucket with about 11k keys. Using ripple I >>>>>>>> could only get about 150 keys back this way because all the other >>>>>>>> chunks were not valid json. I hacked around this for now by checking >>>>>>>> to see if the chunk ends in ]}, otherwise I accumulate chunks until I >>>>>>>> have a full json object. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Adam >>>>>>>> >>>>>>>> >>>>>>>> On May 4, 2010, at 9:22 AM, Sean Cribbs wrote: >>>>>>>> >>>>>>>>> This behavior is unchanged since its implementation in 0.8 or 0.9. >>>>>>>>> To better demonstrate what that statement means, here's the joined >>>>>>>>> output of a streamed keys response: >>>>>>>>> >>>>>>>>> {"props":{"name":"foo","n_val":3,"allow_mult":false,"precommit":[],"postcommit":[],"chash_keyfun":{"mod":"riak_core_util","fun":"chash_std_keyfun"},"linkfun":{"mod":"riak_kv_wm_link_walker","fun":"mapreduce_linkfun"},"old_vclock":86400,"young_vclock":20,"big_vclock":50,"small_vclock":10}}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]} >>>>>>>>> >>>>>>>>> If your client library supports per-chunk processing, each of those >>>>>>>>> {"keys":[]} objects will be an encoded chunk. The Ruby client and >>>>>>>>> most of the other clients support this. >>>>>>>>> >>>>>>>>> Sean Cribbs <s...@basho.com> >>>>>>>>> Developer Advocate >>>>>>>>> Basho Technologies, Inc. >>>>>>>>> http://basho.com/ >>>>>>>>> >>>>>>>>> On May 3, 2010, at 9:28 PM, Adam Hunter wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> I apologize if this has been covered already. >>>>>>>>>> >>>>>>>>>> In the wiki it says "If keys=stream, the response will be >>>>>>>>>> transferred using chunked-encoding, where each chunk is a JSON >>>>>>>>>> object" (on http://wiki.basho.com/display/RIAK/REST+API) >>>>>>>>>> >>>>>>>>>> This doesn't seem to be the case, sometimes a json object is broken >>>>>>>>>> into several chunks. Is this behavior correct? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Adam >>>>>>>>>> _______________________________________________ >>>>>>>>>> riak-users mailing list >>>>>>>>>> riak-users@lists.basho.com >>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> Click here to report this email as spam. >>>>> >>>>> _______________________________________________ >>>>> riak-users mailing list >>>>> riak-users@lists.basho.com >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>>> Andrew Harvey / Developer >>>> lexer >>>> >>>> m/ >>>> t/ +61 2 9019 6379 >>>> w/ http://lexer.com.au >>>> >>>> Help put an end to whaling. Visit www.givewhalesavoice.com.au >>>> >>>> >>>> >>>> Please consider the environment before printing this email >>>> This email transmission is confidential and intended solely for the person >>>> or organisation to whom it is addressed. If you are not the intended >>>> recipient, you must not copy, distribute or disseminate the information, >>>> or take any action in relation to it and please delete this e-mail. Any >>>> views expressed in this message are those of the individual sender, except >>>> where the send specifically states them to be the views of any >>>> organisation or employer. If you have received this message in error, do >>>> not open any attachment but please notify the sender (above). This message >>>> has been checked for all known viruses powered by McAfee. >>>> >>>> For further information visit >>>> http://www.mcafee.com/us/threat_center/default.asp >>>> Please rely on your own virus check as no responsibility is taken by the >>>> sender for any damage rising out of any virus infection this communication >>>> may contain. >>>> >>>> >>>> This message has been scanned for malware by Websense. www.websense.com >>>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > > > Andrew Harvey / Developer > lexer > > m/ > t/ +61 2 9019 6379 > w/ http://lexer.com.au > > Help put an end to whaling. Visit www.givewhalesavoice.com.au > > > > Please consider the environment before printing this email > This email transmission is confidential and intended solely for the person or > organisation to whom it is addressed. If you are not the intended recipient, > you must not copy, distribute or disseminate the information, or take any > action in relation to it and please delete this e-mail. Any views expressed > in this message are those of the individual sender, except where the send > specifically states them to be the views of any organisation or employer. If > you have received this message in error, do not open any attachment but > please notify the sender (above). This message has been checked for all known > viruses powered by McAfee. > > For further information visit > http://www.mcafee.com/us/threat_center/default.asp > Please rely on your own virus check as no responsibility is taken by the > sender for any damage rising out of any virus infection this communication > may contain.
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
