I've not messed with partition numbers, so I'm going to take a stab and say 64 across 3 hosts.
Andrew On 13/05/2010, at 1:34 PM, Sean Cribbs wrote: The chunk size-splitting makes sense, although it is irritating to hack around it. How many partitions are you using in your cluster? Sean Cribbs <s...@basho.com<mailto:s...@basho.com>> Developer Advocate Basho Technologies, Inc. http://basho.com/ On May 12, 2010, at 11:26 PM, Andrew Harvey wrote: I believe it's a curb bug. curl --raw gives me expected output, but the chunks yielded by the @client.http.get are not the same chunks shown by curl – they're fragmented as shown earlier in this thread., that said the chunks returned are pretty huge (1.2 MiB), which might explain why curb is cutting them up. Andrew On 13/05/2010, at 1:17 PM, Sean Cribbs wrote: If the problem is with net/http then we need to fix that backend, not apply hacks to the generic portions. You can verify that Riak will return valid JSON objects in each chunk by adding --raw to the curl command. Here's what I get after loading the Fast Track stock data: http://gist.github.com/399450 Sean Cribbs <s...@basho.com<mailto:s...@basho.com>> Developer Advocate Basho Technologies, Inc. http://basho.com/ On May 12, 2010, at 10:55 PM, Adam Hunter wrote: This is a confirmed bug and will be fixed soon. You can check out http://github.com/adamhunter/ripple/blob/associations/lib/riak/bucket.rb : 65 for a temporary hack. Thanks, Adam On May 12, 2010, at 10:17 PM, Andrew Harvey wrote: I've just run into this one as well. Not sure if it's a curb thing or what, but it's making me a sad panda. We have 500k+ keys in our bucket, and I'm being returned chunks of invalid json so nothing is being yielded to my block. Needless to say, the task I'm doing here I'd prefer to do in a MapReduce, but unfortunately I need to get everything out into ruby-land for the moment. Andrew On 04/05/2010, at 11:46 PM, Sean Cribbs wrote: This looks like a bug. I'll add an issue to the github tracker. Sean Cribbs <s...@basho.com<mailto:s...@basho.com>> Developer Advocate Basho Technologies, Inc. http://basho.com/ On May 4, 2010, at 9:29 AM, Adam Hunter wrote: Just curb, I couldn't get net/http to work when doing chunked (something about read_body being called twice). On May 4, 2010, at 9:28 AM, Sean Cribbs wrote: Are you using curb or net/http? Sean Cribbs <s...@basho.com<mailto:s...@basho.com>> Developer Advocate Basho Technologies, Inc. http://basho.com/ On May 4, 2010, at 9:26 AM, Adam Hunter wrote: The problem I was running into was some of the chunks were only part of the json object. Decoded it would be something like this: chunk 1: {'keys':[1,2,3,4 chunk 2: 5,6,7,8,9]} I was running this on a bucket with about 11k keys. Using ripple I could only get about 150 keys back this way because all the other chunks were not valid json. I hacked around this for now by checking to see if the chunk ends in ]}, otherwise I accumulate chunks until I have a full json object. Thanks, Adam On May 4, 2010, at 9:22 AM, Sean Cribbs wrote: This behavior is unchanged since its implementation in 0.8 or 0.9. To better demonstrate what that statement means, here's the joined output of a streamed keys response: {"props":{"name":"foo","n_val":3,"allow_mult":false,"precommit":[],"postcommit":[],"chash_keyfun":{"mod":"riak_core_util","fun":"chash_std_keyfun"},"linkfun":{"mod":"riak_kv_wm_link_walker","fun":"mapreduce_linkfun"},"old_vclock":86400,"young_vclock":20,"big_vclock":50,"small_vclock":10}}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]} If your client library supports per-chunk processing, each of those {"keys":[]} objects will be an encoded chunk. The Ruby client and most of the other clients support this. Sean Cribbs <s...@basho.com<mailto:s...@basho.com>> Developer Advocate Basho Technologies, Inc. http://basho.com/ On May 3, 2010, at 9:28 PM, Adam Hunter wrote: Hi All, I apologize if this has been covered already. In the wiki it says "If keys=stream, the response will be transferred using chunked-encoding, where each chunk is a JSON object" (on http://wiki.basho.com/display/RIAK/REST+API) This doesn't seem to be the case, sometimes a json object is broken into several chunks. Is this behavior correct? Thanks, Adam _______________________________________________ riak-users mailing list riak-users@lists.basho.com<mailto:riak-users@lists.basho.com> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com Andrew Harvey / Developer lexer m/ t/ +61 2 9019 6379 w/ http://lexer.com.au Help put an end to whaling. Visit www.givewhalesavoice.com.au<http://www.givewhalesavoice.com.au/> ________________________________ Please consider the environment before printing this email This email transmission is confidential and intended solely for the person or organisation to whom it is addressed. If you are not the intended recipient, you must not copy, distribute or disseminate the information, or take any action in relation to it and please delete this e-mail. Any views expressed in this message are those of the individual sender, except where the send specifically states them to be the views of any organisation or employer. If you have received this message in error, do not open any attachment but please notify the sender (above). This message has been checked for all known viruses powered by McAfee. For further information visit http://www.mcafee.com/us/threat_center/default.asp Please rely on your own virus check as no responsibility is taken by the sender for any damage rising out of any virus infection this communication may contain. This message has been scanned for malware by Websense. www.websense.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
