Re: keys=stream

2010-05-12 Thread Andrew Harvey
I've not messed with partition numbers, so I'm going to take a stab and say 64 across 3 hosts. Andrew On 13/05/2010, at 1:34 PM, Sean Cribbs wrote: The chunk size-splitting makes sense, although it is irritating to hack around it. How many partitions are you using in your cluster? Sean Crib

Re: keys=stream

2010-05-12 Thread Sean Cribbs
The chunk size-splitting makes sense, although it is irritating to hack around it. How many partitions are you using in your cluster? Sean Cribbs Developer Advocate Basho Technologies, Inc. http://basho.com/ On May 12, 2010, at 11:26 PM, Andrew Harvey wrote: > I believe it's a curb bug. > >

Re: keys=stream

2010-05-12 Thread Andrew Harvey
I believe it's a curb bug. curl --raw gives me expected output, but the chunks yielded by the @client.http.get are not the same chunks shown by curl – they're fragmented as shown earlier in this thread., that said the chunks returned are pretty huge (1.2 MiB), which might explain why curb is cu

Re: keys=stream

2010-05-12 Thread Adam Hunter
I haven't tested net/http, I couldn't get it to work when passing a block to keys. I am pretty sure this is the same issue I mentioned on 5/4 and you reported as a bug. It only seems to happen with large numbers of keys. Riak will return partial JSON objects and they are rejected in ripple be

Re: keys=stream

2010-05-12 Thread Sean Cribbs
If the problem is with net/http then we need to fix that backend, not apply hacks to the generic portions. You can verify that Riak will return valid JSON objects in each chunk by adding --raw to the curl command. Here's what I get after loading the Fast Track stock data: http://gist.github.co

Re: keys=stream

2010-05-12 Thread Adam Hunter
This is a confirmed bug and will be fixed soon. You can check out http://github.com/adamhunter/ripple/blob/associations/lib/riak/bucket.rb : 65 for a temporary hack. Thanks, Adam On May 12, 2010, at 10:17 PM, Andrew Harvey wrote: > I've just run into this one as well. > > > Not sure if it'

Re: keys=stream

2010-05-12 Thread Andrew Harvey
I've just run into this one as well. Not sure if it's a curb thing or what, but it's making me a sad panda. We have 500k+ keys in our bucket, and I'm being returned chunks of invalid json so nothing is being yielded to my block. Needless to say, the task I'm doing here I'd prefer to do in a Map

Re: keys=stream

2010-05-04 Thread Sean Cribbs
This looks like a bug. I'll add an issue to the github tracker. Sean Cribbs Developer Advocate Basho Technologies, Inc. http://basho.com/ On May 4, 2010, at 9:29 AM, Adam Hunter wrote: > Just curb, I couldn't get net/http to work when doing chunked (something > about read_body being called twi

Re: keys=stream

2010-05-04 Thread Adam Hunter
Just curb, I couldn't get net/http to work when doing chunked (something about read_body being called twice). On May 4, 2010, at 9:28 AM, Sean Cribbs wrote: > Are you using curb or net/http? > > Sean Cribbs > Developer Advocate > Basho Technologies, Inc. > http://basho.com/ > > On May 4, 201

Re: keys=stream

2010-05-04 Thread Sean Cribbs
Are you using curb or net/http? Sean Cribbs Developer Advocate Basho Technologies, Inc. http://basho.com/ On May 4, 2010, at 9:26 AM, Adam Hunter wrote: > The problem I was running into was some of the chunks were only part of the > json object. Decoded it would be something like this: > > c

Re: keys=stream

2010-05-04 Thread Adam Hunter
The problem I was running into was some of the chunks were only part of the json object. Decoded it would be something like this: chunk 1: {'keys':[1,2,3,4 chunk 2: 5,6,7,8,9]} I was running this on a bucket with about 11k keys. Using ripple I could only get about 150 keys back this way becau

Re: keys=stream

2010-05-04 Thread Sean Cribbs
This behavior is unchanged since its implementation in 0.8 or 0.9. To better demonstrate what that statement means, here's the joined output of a streamed keys response: {"props":{"name":"foo","n_val":3,"allow_mult":false,"precommit":[],"postcommit":[],"chash_keyfun":{"mod":"riak_core_util","fu