On 22.05.2014 15:36, Yehuda Sadeh wrote:
On Thu, May 22, 2014 at 6:16 AM, Georg Höllrigl
<georg.hoellr...@xidras.com> wrote:
Hello List,
Using the radosgw works fine, as long as the amount of data doesn't get too
big.
I have created one bucket that holds many small files, separated into
different "directories". But whenever I try to acess the bucket, I only run
into some timeout. The timeout is at around 30 - 100 seconds. This is
smaller then the Apache timeout of 300 seconds.
I've tried to access the bucket with different clients - one thing is s3cmd
- which still is able to upload things, but takes rather long time, when
listing the contents.
Then I've tried with s3fs-fuse - which throws
ls: reading directory .: Input/output error
Also Cyberduck and S3Browser show a similar behaivor.
Is there an option, to only send back maybe 1000 list entries, like Amazon
das? So that the client might decide, if he want's to list all the contents?
That how it works, it doesn't return more than 1000 entries at once.
OK. I found that in the Requests. So it's the client, that states how
many objects should be in the listing with sending the max-keys=1000
variable:
- - - [23/May/2014:08:49:33 +0000] "GET
/test/?delimiter=%2F&max-keys=1000&prefix HTTP/1.1" 200 715 "-"
"Cyberduck/4.4.4 (14505) (Windows NT (unknown)/6.2) (x86)"
"xidrasservice.com:443"
Are there any timeout values in radosgw?
Are you sure the timeout is in the gateway itself? Could be apache
that is timing out. Will need to see the apache access logs for these
operations, radosgw debug and messenger logs (debug rgw = 20, debug ms
= 1), to give a better answer.
No I'm not sure where the timeout comes from. As far as I can tell,
apache times out after 300 seconds - so that should not be the problem.
I think I found something in the apache logs:
[Fri May 23 08:59:39.385548 2014] [fastcgi:error] [pid 3035:tid
140723006891776] [client 10.0.1.66:46049] FastCGI: comm with server
"/var/www/s3gw.fcgi" aborted: idle timeout (30 sec)
[Fri May 23 08:59:39.385604 2014] [fastcgi:error] [pid 3035:tid
140723006891776] [client 10.0.1.66:46049] FastCGI: incomplete headers (0
bytes) received from server "/var/www/s3gw.fcgi"
I've increased the timeout to 900 in the apache vhosts config:
FastCgiExternalServer /var/www/s3gw.fcgi -socket
/var/run/ceph/radosgw.vvx-ceph-m-02 -idle-timeout 900
Now it's not working, and I don't get a log entry any more.
Most interesting when watching the debug output - I'm getting that rados
successfully finished with the request. But at the same time, the client
tells me, it failed.
I've shortened the log file, as far as I could see, the info repeats
itself...
2014-05-23 09:38:43.051395 7f1b427fc700 1 ====== starting new request
req=0x7f1b3400f1c0 =====
2014-05-23 09:38:43.051597 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.1.199:6800/14453 -- osd_op(client.72942.0:120 UHXW458EH1RVULE1BCEH
[getxattrs,stat] 11.10193f7e ack+read e279) v4 -- ?+0 0x7f1b00004640 con
0x2455930
2014-05-23 09:38:43.053180 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.0 10.0.1.199:6800/14453 23 ==== osd_op_reply(120
UHXW458EH1RVULE1BCEH [getxattrs,stat] v0'0 uv1 ondisk = 0) v6 ====
229+0+20 (1060030390 0 1010060712) 0x7f1b58002540 con 0x2455930
2014-05-23 09:38:43.053380 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.1.199:6800/14453 -- osd_op(client.72942.0:121 UHXW458EH1RVULE1BCEH
[read 0~524288] 11.10193f7e ack+read e279) v4 -- ?+0 0x7f1b000045d0 con
0x2455930
2014-05-23 09:38:43.054359 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.0 10.0.1.199:6800/14453 24 ==== osd_op_reply(121
UHXW458EH1RVULE1BCEH [read 0~8] v0'0 uv1 ondisk = 0) v6 ==== 187+0+8
(3510944971 0 3829959217) 0x7f1b580057b0 con 0x2455930
2014-05-23 09:38:43.054490 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.1.199:6806/15018 -- osd_op(client.72942.0:122 macm [getxattrs,stat]
7.1069f101 ack+read e279) v4 -- ?+0 0x7f1b00006010 con 0x2457de0
2014-05-23 09:38:43.055871 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.2 10.0.1.199:6806/15018 3 ==== osd_op_reply(122 macm
[getxattrs,stat] v0'0 uv46 ondisk = 0) v6 ==== 213+0+91 (22324782 0
2022698800) 0x7f1b500025a0 con 0x2457de0
2014-05-23 09:38:43.055963 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.1.199:6806/15018 -- osd_op(client.72942.0:123 macm [read 0~524288]
7.1069f101 ack+read e279) v4 -- ?+0 0x7f1b00003950 con 0x2457de0
2014-05-23 09:38:43.057087 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.2 10.0.1.199:6806/15018 4 ==== osd_op_reply(123 macm [read 0~310]
v0'0 uv46 ondisk = 0) v6 ==== 171+0+310 (3762965810 0 1648184722)
0x7f1b500026e0 con 0x2457de0
2014-05-23 09:38:43.057364 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.0.26:6809/4834 -- osd_op(client.72942.0:124 store [call
version.read,getxattrs,stat] 5.c5755cee ack+read e279) v4 -- ?+0
0x7f1b000066b0 con 0x7f1b440022e0
2014-05-23 09:38:43.059223 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.7 10.0.0.26:6809/4834 37 ==== osd_op_reply(124 store
[call,getxattrs,stat] v0'0 uv5 ondisk = 0) v6 ==== 256+0+139 (2811949634
0 2819061689) 0x7f1aa4001230 con 0x7f1b440022e0
2014-05-23 09:38:43.059380 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.0.26:6809/4834 -- osd_op(client.72942.0:125 store [call
version.check_conds,call version.read,read 0~524288] 5.c5755cee ack+read
e279) v4 -- ?+0 0x7f1b00006740 con 0x7f1b440022e0
2014-05-23 09:38:43.060690 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.7 10.0.0.26:6809/4834 38 ==== osd_op_reply(125 store [call,call,read
0~118] v0'0 uv5 ondisk = 0) v6 ==== 256+0+166 (2576364209 0 1379163168)
0x7f1aa4000fd0 con 0x7f1b440022e0
2014-05-23 09:38:43.060825 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.1.199:6800/14453 -- osd_op(client.72942.0:126
.bucket.meta.store:default.18219.9 [call version.read,getxattrs,stat]
5.39548562 ack+read e279) v4 -- ?+0 0x7f1b0000a960 con 0x2455930
2014-05-23 09:38:43.062092 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.0 10.0.1.199:6800/14453 25 ==== osd_op_reply(126
.bucket.meta.store:default.18219.9 [call,getxattrs,stat] v0'0 uv7 ondisk
= 0) v6 ==== 285+0+338 (1761283478 0 2546497189) 0x7f1b58002880 con
0x2455930
2014-05-23 09:38:43.062217 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.1.199:6800/14453 -- osd_op(client.72942.0:127
.bucket.meta.store:default.18219.9 [call version.check_conds,call
version.read,read 0~524288] 5.39548562 ack+read e279) v4 -- ?+0
0x7f1b0000c010 con 0x2455930
2014-05-23 09:38:43.063317 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.0 10.0.1.199:6800/14453 26 ==== osd_op_reply(127
.bucket.meta.store:default.18219.9 [call,call,read 0~177] v0'0 uv7
ondisk = 0) v6 ==== 285+0+225 (2041047287 0 1981305190) 0x7f1b58002880
con 0x2455930
2014-05-23 09:38:43.063522 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.0.26:6809/4834 -- osd_op(client.72942.0:128 .dir.default.18219.9
[call rgw.bucket_list] 16.490e8edb ack+read e279) v4 -- ?+0
0x7f1b0000e010 con 0x7f1b440022e0
2014-05-23 09:38:43.088009 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.7 10.0.0.26:6809/4834 39 ==== osd_op_reply(128 .dir.default.18219.9
[call] v0'0 uv6040646 ondisk = 0) v6 ==== 187+0+267770 (2366346899 0
3428393995) 0x7f1aa4001e30 con 0x7f1b440022e0
2014-05-23 09:38:43.094389 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.0.26:6809/4834 -- osd_op(client.72942.0:129 .dir.default.18219.9
[call rgw.bucket_list] 16.490e8edb ack+read e279) v4 -- ?+0
0x7f1b0000aa00 con 0x7f1b440022e0
2014-05-23 09:38:43.114281 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.7 10.0.0.26:6809/4834 40 ==== osd_op_reply(129 .dir.default.18219.9
[call] v0'0 uv6040646 ondisk = 0) v6 ==== 187+0+271384 (2464328238 0
2882404457) 0x7f1aa40013f0 con 0x7f1b440022e0
2014-05-23 09:38:43.119686 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.0.26:6809/4834 -- osd_op(client.72942.0:130 .dir.default.18219.9
[call rgw.bucket_list] 16.490e8edb ack+read e279) v4 -- ?+0
0x7f1b0000aa00 con 0x7f1b440022e0
2014-05-23 09:38:43.136637 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.7 10.0.0.26:6809/4834 41 ==== osd_op_reply(130 .dir.default.18219.9
[call] v0'0 uv6040646 ondisk = 0) v6 ==== 187+0+273877 (2059661670 0
989233300) 0x7f1aa4000d20 con 0x7f1b440022e0
2014-05-23 09:38:43.142046 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.0.26:6809/4834 -- osd_op(client.72942.0:131 .dir.default.18219.9
[call rgw.bucket_list] 16.490e8edb ack+read e279) v4 -- ?+0
0x7f1b0000aa00 con 0x7f1b440022e0
2014-05-23 09:38:43.159032 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.7 10.0.0.26:6809/4834 42 ==== osd_op_reply(131 .dir.default.18219.9
[call] v0'0 uv6040646 ondisk = 0) v6 ==== 187+0+269759 (851541080 0
3284938072) 0x7f1aa4001e50 con 0x7f1b440022e0
2014-05-23 09:38:43.164546 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.0.26:6809/4834 -- osd_op(client.72942.0:132 .dir.default.18219.9
[call rgw.bucket_list] 16.490e8edb ack+read e279) v4 -- ?+0
0x7f1b0000aa00 con 0x7f1b440022e0
2014-05-23 09:38:43.183417 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.7 10.0.0.26:6809/4834 43 ==== osd_op_reply(132 .dir.default.18219.9
[call] v0'0 uv6040646 ondisk = 0) v6 ==== 187+0+267089 (4243678976 0
4183594670) 0x7f1aa4002930 con 0x7f1b440022e0
2014-05-23 09:38:43.188943 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.0.26:6809/4834 -- osd_op(client.72942.0:133 .dir.default.18219.9
[call rgw.bucket_list] 16.490e8edb ack+read e279) v4 -- ?+0
0x7f1b000bf570 con 0x7f1b440022e0
.......
2014-05-23 09:39:56.983266 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.7 10.0.0.26:6809/4834 3054 ==== osd_op_reply(3143
.dir.default.18219.9 [call] v0'0 uv6040646 ondisk = 0) v6 ====
187+0+278878 (2034560412 0 3767871034) 0x7f1aa4002480 con 0x7f1b440022e0
2014-05-23 09:39:56.988770 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.0.26:6809/4834 -- osd_op(client.72942.0:3144 .dir.default.18219.9
[call rgw.bucket_list] 16.490e8edb ack+read e279) v4 -- ?+0
0x7f1b0007e730 con 0x7f1b440022e0
2014-05-23 09:39:57.007595 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.7 10.0.0.26:6809/4834 3055 ==== osd_op_reply(3144
.dir.default.18219.9 [call] v0'0 uv6040646 ondisk = 0) v6 ====
187+0+279377 (1247039802 0 223724024) 0x7f1aa4002930 con 0x7f1b440022e0
2014-05-23 09:39:57.013098 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.0.26:6809/4834 -- osd_op(client.72942.0:3145 .dir.default.18219.9
[call rgw.bucket_list] 16.490e8edb ack+read e279) v4 -- ?+0
0x7f1b0007e730 con 0x7f1b440022e0
2014-05-23 09:39:57.032036 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.7 10.0.0.26:6809/4834 3056 ==== osd_op_reply(3145
.dir.default.18219.9 [call] v0'0 uv6040646 ondisk = 0) v6 ====
187+0+279864 (246046050 0 1972744353) 0x7f1aa4002a80 con 0x7f1b440022e0
2014-05-23 09:39:57.037451 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.0.26:6809/4834 -- osd_op(client.72942.0:3146 .dir.default.18219.9
[call rgw.bucket_list] 16.490e8edb ack+read e279) v4 -- ?+0
0x7f1b001f00e0 con 0x7f1b440022e0
2014-05-23 09:39:57.054351 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.7 10.0.0.26:6809/4834 3057 ==== osd_op_reply(3146
.dir.default.18219.9 [call] v0'0 uv6040646 ondisk = 0) v6 ====
187+0+280405 (1330848831 0 2644350106) 0x7f1aa4002ca0 con 0x7f1b440022e0
2014-05-23 09:39:57.059788 7f1b427fc700 1 -- 10.0.1.107:0/1005898 -->
10.0.0.26:6809/4834 -- osd_op(client.72942.0:3147 .dir.default.18219.9
[call rgw.bucket_list] 16.490e8edb ack+read e279) v4 -- ?+0
0x7f1b001f00e0 con 0x7f1b440022e0
2014-05-23 09:39:57.073656 7f1b96d80700 1 -- 10.0.1.107:0/1005898 <==
osd.7 10.0.0.26:6809/4834 3058 ==== osd_op_reply(3147
.dir.default.18219.9 [call] v0'0 uv6040646 ondisk = 0) v6 ====
187+0+186051 (752043312 0 3398857902) 0x7f1aa4002c10 con 0x7f1b440022e0
2014-05-23 09:39:57.077264 7f1b427fc700 1 ====== req done
req=0x7f1b3400f1c0 http_status=200 ======
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
