2014-04-03 1:36 GMT+06:00 Seth Thomas <stho...@basho.com>: > Could you also include your riak app.config and vm.args. It seems like > you're load balancing Riak CS but I'm curious how the underlying Riak > topology looks as well since that will likely be where the performance > bottlenecks are uncovered.
Config templates attached.
> On Wed, Apr 2, 2014 at 6:38 AM, Luke Bakken <lbak...@basho.com> wrote:
>>
>> Hi Stanislav,
>>
>> In your Riak /etc/riak/app.config files, please use the following value:
>>
>> {pb_backlog, 256},
>>
>> After changing this, you will have to restart Riak in a rolling fashion.
>>
>> Could you please run riak-debug on one node in your cluster and make the
>> generated archive available? (dropbox, for instance). Also, could you run
>> tar -czf /tmp/riak-cs-$(hostname).tgz and make the archive available?
>>
>> Thanks
>> --
>> Luke Bakken
>> CSE
>> lbak...@basho.com
>>
>>
>> On Tue, Apr 1, 2014 at 9:32 PM, Stanislav Vlasov <stanislav....@gmail.com>
>> wrote:
>>>
>>> Hello!
>>>
>>> I have 8x cluster of riak+riak-cs on debian. Config templates attached
>>> Versions:
>>> ii riak 1.4.8-1
>>> amd64 Riak is a distributed data store
>>> ii riak-cs 1.4.5-1
>>> amd64 Riak CS
>>>
>>> Every riak-cs connect to local node. Between clients and riak-cs exist
>>> frontend (Tengine version: Tengine/1.5.1 (nginx/1.2.9)), config
>>> attached
>>> Clients - s3cmd + some numbers of php (read-only)
>>>
>>> When 1-3 clients wants write to riak-cs, write speed is near 3-4MB/sec.
>>> If 30-40 clients wants write, write speed slow down to lower than
>>> 100kB/sec.
>>>
>>> In riak-cs crash.log:
>>>
>>> 2014-04-02 03:52:11 =ERROR REPORT====
>>> webmachine error:
>>> path="/buckets/test/objects/win.img/uploads/PuqEyz0BRCCk6rDxtH7tRQ=="
>>>
>>> {error,{error,{badmatch,{error,closed}},[{webmachine_request,recv_unchunked_body,3,[{file,"src/webmachine_request.erl"},{line,474}]},{webmachine_request,call,2,[{file,"src/webmachine_request.erl"},{line,193}]},{wrq,stream_req_body,2,[{file,"src/wrq.erl"},{line,121}]},{riak_cs_wm_object_upload_part,accept_body,2,[{file,"src/riak_cs_wm_object_upload_part.erl"},{line,235}]},{riak_cs_wm_common,accept_body,2,[{file,"src/riak_cs_wm_common.erl"},{line,337}]},{webmachine_resource,resource_call,3,[{file,"src/webmachine_resource.erl"},{line,186}]},{webmachine_resource,do,3,[{file,"src/webmachine_resource.erl"},{line,142}]},{webmachine_decision_core,resource_call,1,[{file,"src/webmachine_decision_core.erl"},{line,48}]}]}}
>>>
>>> [{webmachine_request,recv_unchunked_body,3,[{file,"src/webmachine_request.erl"},{line,474}]},{webmachine_request,call,2,[{file,"src/webmachine_request.erl"},{line,193}]},{wrq,stream_req_body,2,[{file,"src/wrq.erl"},{line,121}]},{riak_cs_wm_object_upload_part,accept_body,2,[{file,"src/riak_cs_wm_object_upload_part.erl"},{line,235}]},{riak_cs_wm_common,accept_body,2,[{file,"src/riak_cs_wm_common.erl"},{line,337}]},{webmachine_resource,resource_call,3,[{file,"src/webmachine_resource.erl"},{line,186}]},{webmachine_resource,do,3,[{file,"src/webmachine_resource.erl"},{line,142}]},{webmachine_decision_core,resource_call,1,[{file,"src/webmachine_decision_core.erl"},{line,48}]}]
>>>
>>> After this event s3cmd makes throttling to slower speed:
>>>
>>> $ s3cmd put win.img s3://test/
>>> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
>>> 184320 of 15728640 1% in 0s 2.16 MB/s failed
>>> WARNING: Upload failed:
>>> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
>>> Connection reset by peer)
>>> WARNING: Retrying on lower speed (throttle=0.00)
>>> WARNING: Waiting 3 sec...
>>> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
>>> 13799424 of 15728640 87% in 2s 5.18 MB/s failed
>>> WARNING: Upload failed:
>>> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
>>> Connection reset by peer)
>>> WARNING: Retrying on lower speed (throttle=0.01)
>>> WARNING: Waiting 6 sec...
>>> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
>>> 167936 of 15728640 1% in 0s 249.46 kB/s failed
>>> WARNING: Upload failed:
>>> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
>>> Connection reset by peer)
>>> WARNING: Retrying on lower speed (throttle=0.05)
>>> WARNING: Waiting 9 sec...
>>> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
>>> 6225920 of 15728640 39% in 76s 79.51 kB/s failed
>>> WARNING: Upload failed:
>>> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
>>> Connection reset by peer)
>>> WARNING: Retrying on lower speed (throttle=0.25)
>>> WARNING: Waiting 12 sec...
>>> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
>>> 15728640 of 15728640 100% in 962s 15.96 kB/s done
>>>
>>> I think, even on 1Gbit network betwen nodes, write speed should be
>>> higher, but i don't understand where the bottleneck.
>>>
>>> --
>>> Stanislav
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
--
Stanislav
app.config.template
Description: Binary data
vm.args.template
Description: Binary data
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
