Hi Ryan,
I appreciate the response and look forward to seeing 2i rock harder
than ever before with all the changes you guys are pushing in.
I did make a mistake in my benchmark, and in reality it was closer to
70 req/s that I was getting across my concurrent erlang benchmark
workers.
HTTP got me in to the 350 - 400 req/s range which is significantly
better, using ab -n 1000 -c 10
Looking forward to seeing 2i be very fast!
-Tom
On Mon, Jun 18, 2012 at 9:26 AM, Ryan Zezeski <rzeze...@basho.com> wrote:
> Hi Tom, response inline.
>
> On Thu, Jun 14, 2012 at 3:29 PM, Tom Burdick <thomas.burd...@gmail.com> wrote:
>
>>
>> retrieve_by_client(Db, Key) when is_binary(Key) ->
>> {ok, BKeys} = riakc_pb_socket:get_index(Db, ?bucket,
>> <<"client_id_bin">>, Key),
>> lists:map(fun([_Bucket, Key0]) ->
>> Key0
>> end, BKeys).
>>
>> What I've noticed while benchmarking the put is that that is actually
>> quite fast and I can do that
>> at a pretty high rate even on my devrel setup, like 1000 puts/sec.
>>
>> However when doing retrieve_by_client at most I've seen 20
>> get_indexes/sec with latencies all over the place from a few
>> milliseconds up to a few seconds.
>
> I notice you're using the Erlang PB client. This is the client we
> recommend but a current shortfall of it re 2i is that it unnecessarily
> uses map/reduce to collate the results.
>
> Bryan Find recently made some changes to Pipe, the system behind
> map/reduce, that should improve latencies for queries that match
> "larger" result sets. How large I'm not sure, but the larger the
> result set the more of an improvement you'll see is what I'm told.
>
> Sean Cribbs recently made some change to our protobuffs server that
> will allow direct querying of 2i without the use of map/reduce. IMO
> this should greatly reduce your latencies. However, the various
> clients still need to be modified to take advantage of this. I defer
> to Sean on when that might be done.
>
> In the meantime you could try changing your query code to use the HTTP
> interface and see if that brings you better latency.
>
> Oh, I just remember, Matthew Von-Maszewski has made some amazing
> strides in our branch of leveldb in terms of
> latency/throughput/robustness and that should result in trickle-down
> improvements as well.
>
> All these changes, with the exception of the client modifications,
> should be coming with our next release, 1.2.
>
>>
>> I'm tempted to fire up fprof on Client:get_index just to see whats up
>> but I'm a bit scared at what I might see there :-)
>
> Feel free to fprof, my guess is most of the time will be spent in
> map/reduce related code. The other place 2i spends a majority of it's
> time is sext decoding.
>
> -Z
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
