2014-04-02 19:38 GMT+06:00 Luke Bakken <lbak...@basho.com>:
> In your Riak /etc/riak/app.config files, please use the following value:
>
> {pb_backlog, 256},
I try even {pb_backlog, 512} - no changes.
> After changing this, you will have to restart Riak in a rolling fashion.
> Could you please run riak-debug on one node in your cluster and make the
> generated archive available? (dropbox, for instance).
# riak-debug
............................................./usr/sbin/riak-debug:
538: [: =: argument expected
Unable to locate be_default LevelDB data directory. Aborting.
Using riak_kv_eleveldb_backend data_root:
Generated data in archive: http://ovh.to/QmPhSAy
> Also, could you run
> tar -czf /tmp/riak-cs-$(hostname).tgz and make the archive available?
What data you need in archive if not debug info?
> Thanks
> --
> Luke Bakken
> CSE
> lbak...@basho.com
>
>
> On Tue, Apr 1, 2014 at 9:32 PM, Stanislav Vlasov <stanislav....@gmail.com>
> wrote:
>>
>> Hello!
>>
>> I have 8x cluster of riak+riak-cs on debian. Config templates attached
>> Versions:
>> ii riak 1.4.8-1
>> amd64 Riak is a distributed data store
>> ii riak-cs 1.4.5-1
>> amd64 Riak CS
>>
>> Every riak-cs connect to local node. Between clients and riak-cs exist
>> frontend (Tengine version: Tengine/1.5.1 (nginx/1.2.9)), config
>> attached
>> Clients - s3cmd + some numbers of php (read-only)
>>
>> When 1-3 clients wants write to riak-cs, write speed is near 3-4MB/sec.
>> If 30-40 clients wants write, write speed slow down to lower than
>> 100kB/sec.
>>
>> In riak-cs crash.log:
>>
>> 2014-04-02 03:52:11 =ERROR REPORT====
>> webmachine error:
>> path="/buckets/test/objects/win.img/uploads/PuqEyz0BRCCk6rDxtH7tRQ=="
>>
>> {error,{error,{badmatch,{error,closed}},[{webmachine_request,recv_unchunked_body,3,[{file,"src/webmachine_request.erl"},{line,474}]},{webmachine_request,call,2,[{file,"src/webmachine_request.erl"},{line,193}]},{wrq,stream_req_body,2,[{file,"src/wrq.erl"},{line,121}]},{riak_cs_wm_object_upload_part,accept_body,2,[{file,"src/riak_cs_wm_object_upload_part.erl"},{line,235}]},{riak_cs_wm_common,accept_body,2,[{file,"src/riak_cs_wm_common.erl"},{line,337}]},{webmachine_resource,resource_call,3,[{file,"src/webmachine_resource.erl"},{line,186}]},{webmachine_resource,do,3,[{file,"src/webmachine_resource.erl"},{line,142}]},{webmachine_decision_core,resource_call,1,[{file,"src/webmachine_decision_core.erl"},{line,48}]}]}}
>>
>> [{webmachine_request,recv_unchunked_body,3,[{file,"src/webmachine_request.erl"},{line,474}]},{webmachine_request,call,2,[{file,"src/webmachine_request.erl"},{line,193}]},{wrq,stream_req_body,2,[{file,"src/wrq.erl"},{line,121}]},{riak_cs_wm_object_upload_part,accept_body,2,[{file,"src/riak_cs_wm_object_upload_part.erl"},{line,235}]},{riak_cs_wm_common,accept_body,2,[{file,"src/riak_cs_wm_common.erl"},{line,337}]},{webmachine_resource,resource_call,3,[{file,"src/webmachine_resource.erl"},{line,186}]},{webmachine_resource,do,3,[{file,"src/webmachine_resource.erl"},{line,142}]},{webmachine_decision_core,resource_call,1,[{file,"src/webmachine_decision_core.erl"},{line,48}]}]
>>
>> After this event s3cmd makes throttling to slower speed:
>>
>> $ s3cmd put win.img s3://test/
>> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
>> 184320 of 15728640 1% in 0s 2.16 MB/s failed
>> WARNING: Upload failed:
>> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
>> Connection reset by peer)
>> WARNING: Retrying on lower speed (throttle=0.00)
>> WARNING: Waiting 3 sec...
>> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
>> 13799424 of 15728640 87% in 2s 5.18 MB/s failed
>> WARNING: Upload failed:
>> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
>> Connection reset by peer)
>> WARNING: Retrying on lower speed (throttle=0.01)
>> WARNING: Waiting 6 sec...
>> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
>> 167936 of 15728640 1% in 0s 249.46 kB/s failed
>> WARNING: Upload failed:
>> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
>> Connection reset by peer)
>> WARNING: Retrying on lower speed (throttle=0.05)
>> WARNING: Waiting 9 sec...
>> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
>> 6225920 of 15728640 39% in 76s 79.51 kB/s failed
>> WARNING: Upload failed:
>> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
>> Connection reset by peer)
>> WARNING: Retrying on lower speed (throttle=0.25)
>> WARNING: Waiting 12 sec...
>> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
>> 15728640 of 15728640 100% in 962s 15.96 kB/s done
>>
>> I think, even on 1Gbit network betwen nodes, write speed should be
>> higher, but i don't understand where the bottleneck.
>>
>> --
>> Stanislav
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
--
Stanislav
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
