Hey guys,
I'm working on a dev environment for a riak-cs setup.
2 vms and an external proxy
Config of the riak/riak-cs nodes appears to be all complete. I'm
encountering two issues I'd like some pointers on where to begin diagnosing
before I go around stracing everything.
Firstly:
When using s3cmd to query riak-cs, I'm receiving differing results on the
same commands in succession. Here are the results when going through a
proxy:
(07:06:09) [andrew/desktop] ~ $ s3cmd -c .s3cfg-riak ls s3://lol
2013-11-13 22:20 84513 s3://lol/kitty.jpg
(07:06:10) [andrew/desktop] ~ $ s3cmd -c .s3cfg-riak ls s3://lol
2013-11-13 22:20 84513 s3://lol/kitty.jpg
(07:06:11) [andrew/desktop] ~ $ s3cmd -c .s3cfg-riak ls s3://lol
2013-11-13 14:59 93107 s3://lol/_1
2013-11-13 22:20 84513 s3://lol/kitty.jpg
(07:06:12) [andrew/desktop] ~ $ s3cmd -c .s3cfg-riak ls s3://lol
DIR s3://lol/kitties/
2013-11-13 14:59 93107 s3://lol/_1
(07:06:13) [andrew/desktop] ~ $ s3cmd -c .s3cfg-riak ls s3://lol
2013-11-13 22:20 84513 s3://lol/kitty.jpg
(07:06:14) [andrew/desktop] ~ $ s3cmd -c .s3cfg-riak ls s3://lol
DIR s3://lol/kitties/
2013-11-13 14:59 93107 s3://lol/_1
And here they are querying one of the nodes directly:
(07:05:59) [andrew/desktop] ~ $ s3cmd ls s3://lol
DIR s3://lol/kitties/
2013-11-13 14:59 93107 s3://lol/_1
(07:06:00) [andrew/desktop] ~ $ s3cmd ls s3://lol
2013-11-13 14:59 93107 s3://lol/_1
2013-11-13 22:20 84513 s3://lol/kitty.jpg
(07:06:01) [andrew/desktop] ~ $ s3cmd ls s3://lol
2013-11-13 22:20 84513 s3://lol/kitty.jpg
(07:06:02) [andrew/desktop] ~ $ s3cmd ls s3://lol
2013-11-13 22:20 84513 s3://lol/kitty.jpg
(07:06:02) [andrew/desktop] ~ $ s3cmd ls s3://lol
DIR s3://lol/kitties/
2013-11-13 14:59 93107 s3://lol/_1
(07:06:03) [andrew/desktop] ~ $ s3cmd ls s3://lol
2013-11-13 22:20 84513 s3://lol/kitty.jpg
(07:06:04) [andrew/desktop] ~ $ s3cmd -c .s3cfg-riak ls s3://lol
The same results happen regardless of which node I query directly, within
1-2 seconds of executing the command a repeat execution of it returns
different results. (They are the same repetitive results, just missing
objects on some of the returns)
The other issue I'm encountering is with put's. If I put directly to the
node, I see something like:
(07:09:09) [andrew/desktop] ~ $ s3cmd put
Downloads/CentOS-6.4-x86_64-minimal.iso s3://big
Downloads/CentOS-6.4-x86_64-minimal.iso ->
s3://big/CentOS-6.4-x86_64-minimal.iso [part 1 of 23, 15MB]
15728640 of 15728640 100% in 5s 2.62 MB/s done
Downloads/CentOS-6.4-x86_64-minimal.iso ->
s3://big/CentOS-6.4-x86_64-minimal.iso [part 2 of 23, 15MB]
15728640 of 15728640 100% in 5s 2.86 MB/s done
(... Truncated some of the values for brevity ...)
Downloads/CentOS-6.4-x86_64-minimal.iso ->
s3://big/CentOS-6.4-x86_64-minimal.iso [part 22 of 23, 15MB]
15728640 of 15728640 100% in 1s 12.06 MB/s done
Downloads/CentOS-6.4-x86_64-minimal.iso ->
s3://big/CentOS-6.4-x86_64-minimal.iso [part 23 of 23, 12MB]
12929024 of 12929024 100% in 1s 11.70 MB/s done
Which is ideally what should occur. However, when I go through the proxy:
It starts great for the first chunk, but hangs:
Start:
Downloads/CentOS-6.4-x86_64-minimal.iso -> s3://big/cent6.minimal.iso [part
19 of 23, 15MB]
8675328 of 15728640 55% in 1s 8.26 MB/s
Finish:
Downloads/CentOS-6.4-x86_64-minimal.iso -> s3://big/cent6.minimal.iso
[part 19 of 23, 15MB]
15728640 of 15728640 100% in 22s 683.57 kB/s done
It immediately jumps to 55% (the % varies) and then pauses, sometimes up to
30 seconds and then jumps to [done].
I assume this is in my nginx configuration somewhere, I thought it was a
proxy buffer issue, I've since raised those limits and also tried disabling
proxy_buffering entirely to no difference.
server {
listen 80;
server_name cs.domain.com *.cs.domain.com;
location / {
proxy_pass http://riak-cs;
proxy_set_header Host $host;
proxy_connect_timeout 59s;
proxy_send_timeout 600;
proxy_read_timeout 600;
#proxy_buffering off;
proxy_buffers 16 32k;
proxy_buffer_size 64k;
#return 403;
}
}
(The two nodes are identical in versions)
(07:34:47) [riak] ~ $ cat /etc/redhat-release
CentOS release 6.4 (Final)
(07:45:53) [riak] ~ $ uname -a
Linux riak.tyne.io 2.6.32-358.23.2.el6.x86_64 #1 SMP Wed Oct 16 18:37:12
UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
(07:46:09) [riak] ~ $ riak version
1.4.2
(07:46:13) [riak] ~ $ riak-cs version
1.4.2
(07:46:25) [riak] ~ $ rpm -qa | grep riak
riak-cs-1.4.2-1.el6.x86_64
riak-1.4.2-1.el6.x86_64
All recommended sysctl and ulimit values have been set as described in the
docs.
I look forward to any assistance with further tracking this down.
--
[Andy Tynefield]
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
