Thanks again for the insight, I did lower the frequency and expiration dates a bit. I'm sure that will resolve itself when actual data is in use. I have made further development with the proxying issue, as far as narrowing it down to client_body_buffer_size:
http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size 2013/11/15 00:13:03 [warn] 2696#0: *20 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000004, client: 192.168.1.1, server: cs.domain.com, request: "PUT /huge/Windows7Ultimate.iso?partNumber=2&uploadId=L9kmj1rNTomi_ZRnHZapvw== HTTP/1.1", host: "big.cs.domain.com" Checking out the nginx mailing list for any available solutions before I consider HAProxy. http://mailman.nginx.org/pipermail/nginx/2013-November/041104.html Thanks again and I appreciate the help. On Thu, Nov 14, 2013 at 12:25 PM, Kelly McLaughlin <ke...@basho.com> wrote: > Andrew, > > There is no tuning necessary for read-repairs. They happen automatically, > but only when you read the object. So if your access pattern involves > frequent read access to your data the repairs will just happen. > > The other option is active anti-entropy. This is basically a way for Riak > to repair data in the background without relying on access pattern. It is > configured in the Riak app.config and you can find more info about the > configuration options under the 'riak_kv settings' section here: > http://docs.basho.com/riak/latest/ops/advanced/configs/configuration-files/ > . > > Hope that helps. > > Kelly > > On November 14, 2013 at 9:57:12 AM, Andrew Tynefield > (atynefi...@gmail.com<//atynefi...@gmail.com>) > wrote: > > Thanks again, Kelly, for your response. > > I continued playing with things last night and > updated fold_objects_for_list_keys across the nodes to be true and > performed a full roll of the stack, that seems to have sped up the replies. > But, the lol bucket was still returning differing results. > > I added the second node to my nginx config and have it load balancing > between the two nodes now, same results. However, I just performed 5-10 > recursive get's on the lol bucket and it seems that did the trick. Getting > consistent values in response to the ls: > > (10:51:14) [andrew/desktop] ~ $ s3cmd ls s3://lol > DIR s3://lol/kitties/ > 2013-11-13 14:59 93107 s3://lol/_1 > 2013-11-13 22:20 84513 s3://lol/kitty.jpg > (10:51:39) [andrew/desktop] ~ $ s3cmd ls s3://lol > DIR s3://lol/kitties/ > 2013-11-13 14:59 93107 s3://lol/_1 > 2013-11-13 22:20 84513 s3://lol/kitty.jpg > (10:51:40) [andrew/desktop] ~ $ s3cmd ls s3://lol > DIR s3://lol/kitties/ > 2013-11-13 14:59 93107 s3://lol/_1 > 2013-11-13 22:20 84513 s3://lol/kitty.jpg > (10:51:41) [andrew/desktop] ~ $ s3cmd ls s3://lol > DIR s3://lol/kitties/ > 2013-11-13 14:59 93107 s3://lol/_1 > 2013-11-13 22:20 84513 s3://lol/kitty.jpg > (10:51:42) [andrew/desktop] ~ $ s3cmd ls s3://lol > DIR s3://lol/kitties/ > 2013-11-13 14:59 93107 s3://lol/_1 > 2013-11-13 22:20 84513 s3://lol/kitty.jpg > (10:51:43) [andrew/desktop] ~ $ s3cmd ls s3://lol > DIR s3://lol/kitties/ > 2013-11-13 14:59 93107 s3://lol/_1 > 2013-11-13 22:20 84513 s3://lol/kitty.jpg > > I was wondering how I can tune against issues like this, but keep the same > performance that's currently in place. I'm not very familiar with > performing read-repairs. Look forward to any assistance with this. > > Thanks, > Andrew > > > On Thu, Nov 14, 2013 at 10:37 AM, Kelly McLaughlin <ke...@basho.com>wrote: > >> Andrew, >> >> Are you able to successfully download all the files in the lol bucket? >> It seems like you have some replicas in Riak that do not have copies of all >> of the data from that bucket. That can be resolved in two ways: read-repair >> or active anti-entropy. So I would expect doing a read of each object would >> resolve the issue with differences in bucket listing attempts. >> >> Another thing I would recommend since you are on the Riak 1.4 series is >> to set fold_objects_for_list_keys to true in your Riak CS app.config. It >> enables improvements to bucket listing that rely on features in Riak 1.4 >> and should give you better results. >> >> I am not certain about the upload issue with the proxy. It does seem >> like a proxy issue since the upload succeeds when performed directly >> against the node, but I am not familiar enough with nginx configuration to >> spot any issues. >> >> Cheers, >> >> Kelly >> >> On November 13, 2013 at 10:24:42 PM, Andrew Tynefield ( >> atynefi...@gmail.com <//atynefi...@gmail.com>) wrote: >> >> I appreciate the help Kelly! ( And sorry for the double mail you're >> going to get, accidentally didn't reply to all. ) I've provided the >> requested information below. >> >> app.config: >> http://pastebin.centos.org/5716/ >> >> app.config for riak-cs is managed by puppet, installing the same file on >> both nodes. >> >> riak-admin data: >> >> (10:58:46) [riak] ~ $ riak-admin ring-status >> ================================== Claimant >> =================================== >> Claimant: 'r...@riak.tyne.io' >> Status: up >> Ring Ready: true >> >> ============================== Ownership Handoff >> ============================== >> No pending changes. >> >> ============================== Unreachable Nodes >> ============================== >> All nodes are up and reachable >> >> (10:58:54) [riak] ~ $ riak-admin member-status >> ================================= Membership >> ================================== >> Status Ring Pending Node >> ------------------------------------------------------------------------------- >> valid 50.0% -- 'r...@riak.tyne.io' >> valid 50.0% -- 'r...@riak1.tyne.io' >> ------------------------------------------------------------------------------- >> Valid:2 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 >> >> network data: >> >> (11:05:51) [riak] ~ $ ip a | grep /24 >> inet 192.168.122.90/24 brd 192.168.122.255 scope global eth0 >> inet 192.168.1.19/24 brd 192.168.1.255 scope global eth1 >> (10:59:43) [riak] ~ $ netstat -tunap | grep :8080 >> tcp 0 0 0.0.0.0:8080 0.0.0.0:* >> LISTEN 29437/beam.smp >> (11:03:20) [riak] ~ $ ps auxf | grep 29437 >> root 1507 0.0 0.0 103236 820 pts/4 S+ 23:03 0:00 >> \_ grep 29437 >> riakcs 29437 0.9 1.8 768480 35364 pts/3 Ssl+ 17:23 3:08 \_ >> /usr/lib64/riak-cs/erts-5.9.1/bin/beam.smp -K true -A 64 -W w -- -root >> /usr/lib64/riak-cs -progname riak-cs -- -home /var/lib/riak-cs-control -- >> -boot /usr/lib64/riak-cs/releases/1.4.2/riak-cs -config >> /etc/riak-cs/app.config -pa /usr/lib64/riak-cs/lib/basho-patches -name >> riak...@riak.domain.com -setcookie [redacted] -- console >> >> nginx config: [added the proxy_pass_header to ensure I was reaching >> riak-cs] >> >> upstream riak-cs { >> server 192.168.1.19:8080; >> } >> server { >> listen 80; >> server_name cs.domain.com *.cs.domain.com; >> location / { >> proxy_pass http://riak-cs; >> proxy_set_header Host $host; >> proxy_connect_timeout 59s; >> proxy_send_timeout 600; >> proxy_read_timeout 600; >> #proxy_buffering off; >> proxy_buffers 16 32k; >> proxy_buffer_size 64k; >> proxy_pass_header Server; >> #return 403; >> >> } >> >> } >> >> (11:10:36) [andrew/desktop] ~ $ curl -I cs.domain.com/buckets >> HTTP/1.1 404 Object Not Found >> Date: Thu, 14 Nov 2013 05:10:38 GMT >> Content-Type: application/xml >> Connection: keep-alive >> Server: Riak CS >> Content-Length: 185 >> >> >> Please let me know if there's anything else I can provide, I'm more than >> willing to do so. Also, it may be worthy to note that domain.com in this >> case is an actual registered and resolving domain that has been sed'd out, >> cause archive. >> >> Thank you so much, >> Andrew >> >> >> On Wed, Nov 13, 2013 at 10:57 PM, Kelly McLaughlin <ke...@basho.com>wrote: >> >>> Andy, >>> >>> To try to get a better idea of what might be going on it would be >>> helpful to see what your riak and riak cs app.config files look like. Also >>> the output of riak-admin ring-status and riak-admin member-status could be >>> useful. For the upload issue I am curious if you have changed the port that >>> riak cs is listening on? The default is 8080 and I don't see from your >>> nginx config where you are sending requests to that port. >>> >>> Kelly >>> >>> On November 13, 2013 at 6:49:59 PM, Andrew Tynefield ( >>> atynefi...@gmail.com <//atynefi...@gmail.com>) wrote: >>> >>> Hey guys, >>> >>> I'm working on a dev environment for a riak-cs setup. >>> >>> 2 vms and an external proxy >>> >>> Config of the riak/riak-cs nodes appears to be all complete. I'm >>> encountering two issues I'd like some pointers on where to begin diagnosing >>> before I go around stracing everything. >>> >>> Firstly: >>> When using s3cmd to query riak-cs, I'm receiving differing results on >>> the same commands in succession. Here are the results when going through a >>> proxy: >>> >>> (07:06:09) [andrew/desktop] ~ $ s3cmd -c .s3cfg-riak ls s3://lol >>> 2013-11-13 22:20 84513 s3://lol/kitty.jpg >>> (07:06:10) [andrew/desktop] ~ $ s3cmd -c .s3cfg-riak ls s3://lol >>> 2013-11-13 22:20 84513 s3://lol/kitty.jpg >>> (07:06:11) [andrew/desktop] ~ $ s3cmd -c .s3cfg-riak ls s3://lol >>> 2013-11-13 14:59 93107 s3://lol/_1 >>> 2013-11-13 22:20 84513 s3://lol/kitty.jpg >>> (07:06:12) [andrew/desktop] ~ $ s3cmd -c .s3cfg-riak ls s3://lol >>> DIR s3://lol/kitties/ >>> 2013-11-13 14:59 93107 s3://lol/_1 >>> (07:06:13) [andrew/desktop] ~ $ s3cmd -c .s3cfg-riak ls s3://lol >>> 2013-11-13 22:20 84513 s3://lol/kitty.jpg >>> (07:06:14) [andrew/desktop] ~ $ s3cmd -c .s3cfg-riak ls s3://lol >>> DIR s3://lol/kitties/ >>> 2013-11-13 14:59 93107 s3://lol/_1 >>> >>> And here they are querying one of the nodes directly: >>> (07:05:59) [andrew/desktop] ~ $ s3cmd ls s3://lol >>> DIR s3://lol/kitties/ >>> 2013-11-13 14:59 93107 s3://lol/_1 >>> (07:06:00) [andrew/desktop] ~ $ s3cmd ls s3://lol >>> 2013-11-13 14:59 93107 s3://lol/_1 >>> 2013-11-13 22:20 84513 s3://lol/kitty.jpg >>> (07:06:01) [andrew/desktop] ~ $ s3cmd ls s3://lol >>> 2013-11-13 22:20 84513 s3://lol/kitty.jpg >>> (07:06:02) [andrew/desktop] ~ $ s3cmd ls s3://lol >>> 2013-11-13 22:20 84513 s3://lol/kitty.jpg >>> (07:06:02) [andrew/desktop] ~ $ s3cmd ls s3://lol >>> DIR s3://lol/kitties/ >>> 2013-11-13 14:59 93107 s3://lol/_1 >>> (07:06:03) [andrew/desktop] ~ $ s3cmd ls s3://lol >>> 2013-11-13 22:20 84513 s3://lol/kitty.jpg >>> (07:06:04) [andrew/desktop] ~ $ s3cmd -c .s3cfg-riak ls s3://lol >>> >>> The same results happen regardless of which node I query directly, >>> within 1-2 seconds of executing the command a repeat execution of it >>> returns different results. (They are the same repetitive results, just >>> missing objects on some of the returns) >>> >>> The other issue I'm encountering is with put's. If I put directly to the >>> node, I see something like: >>> >>> (07:09:09) [andrew/desktop] ~ $ s3cmd put >>> Downloads/CentOS-6.4-x86_64-minimal.iso s3://big >>> Downloads/CentOS-6.4-x86_64-minimal.iso -> >>> s3://big/CentOS-6.4-x86_64-minimal.iso [part 1 of 23, 15MB] >>> 15728640 of 15728640 100% in 5s 2.62 MB/s done >>> Downloads/CentOS-6.4-x86_64-minimal.iso -> >>> s3://big/CentOS-6.4-x86_64-minimal.iso [part 2 of 23, 15MB] >>> 15728640 of 15728640 100% in 5s 2.86 MB/s done >>> (... Truncated some of the values for brevity ...) >>> Downloads/CentOS-6.4-x86_64-minimal.iso -> >>> s3://big/CentOS-6.4-x86_64-minimal.iso [part 22 of 23, 15MB] >>> 15728640 of 15728640 100% in 1s 12.06 MB/s done >>> Downloads/CentOS-6.4-x86_64-minimal.iso -> >>> s3://big/CentOS-6.4-x86_64-minimal.iso [part 23 of 23, 12MB] >>> 12929024 of 12929024 100% in 1s 11.70 MB/s done >>> >>> Which is ideally what should occur. However, when I go through the >>> proxy: >>> >>> It starts great for the first chunk, but hangs: >>> >>> Start: >>> Downloads/CentOS-6.4-x86_64-minimal.iso -> s3://big/cent6.minimal.iso >>> [part 19 of 23, 15MB] >>> 8675328 of 15728640 55% in 1s 8.26 MB/s >>> >>> Finish: >>> Downloads/CentOS-6.4-x86_64-minimal.iso -> s3://big/cent6.minimal.iso >>> [part 19 of 23, 15MB] >>> 15728640 of 15728640 100% in 22s 683.57 kB/s done >>> >>> It immediately jumps to 55% (the % varies) and then pauses, sometimes >>> up to 30 seconds and then jumps to [done]. >>> >>> I assume this is in my nginx configuration somewhere, I thought it was a >>> proxy buffer issue, I've since raised those limits and also tried disabling >>> proxy_buffering entirely to no difference. >>> >>> server { >>> listen 80; >>> server_name cs.domain.com *.cs.domain.com; >>> location / { >>> proxy_pass http://riak-cs; >>> proxy_set_header Host $host; >>> proxy_connect_timeout 59s; >>> proxy_send_timeout 600; >>> proxy_read_timeout 600; >>> #proxy_buffering off; >>> proxy_buffers 16 32k; >>> proxy_buffer_size 64k; >>> #return 403; >>> >>> } >>> >>> } >>> >>> (The two nodes are identical in versions) >>> >>> (07:34:47) [riak] ~ $ cat /etc/redhat-release >>> CentOS release 6.4 (Final) >>> (07:45:53) [riak] ~ $ uname -a >>> Linux riak.tyne.io 2.6.32-358.23.2.el6.x86_64 #1 SMP Wed Oct 16 >>> 18:37:12 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >>> (07:46:09) [riak] ~ $ riak version >>> 1.4.2 >>> (07:46:13) [riak] ~ $ riak-cs version >>> 1.4.2 >>> (07:46:25) [riak] ~ $ rpm -qa | grep riak >>> riak-cs-1.4.2-1.el6.x86_64 >>> riak-1.4.2-1.el6.x86_64 >>> >>> All recommended sysctl and ulimit values have been set as described in >>> the docs. >>> >>> I look forward to any assistance with further tracking this down. >>> >>> -- >>> [Andy Tynefield] >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >> >> >> -- >> [Andy Tynefield] >> >> > > > -- > [Andy Tynefield] > > -- [Andy Tynefield]
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
