Hi guys. I am currently encountering strange problem with radosgw. My setup is: 3 mons, 40OSD 4Tb hdd + 8G SSD journal each on 4 servers and 2 RGW. I have rgw_override_bucket_index_max_shards = 2048 in config and 100 buckets with ~3M objects each. Objects are relatively small: from 1k to 100k. So my problem is big performance degradation for PUT operations with objects number growing. When I started upload, I had ~ 500 r/s for PUT operations. Now I have ~60 r/s.
I have these lines in ceph -w output: slow request 30.190198 seconds old, received at 2016-07-04 03:12:23.354199: osd_op(client.316897.0:3317502521 82.78329d6 3f8804ed-d15b-4a1b-a9ec-e95027dbb8c3.207388.28_22501/1-gallery.webp [create 0~0 [excl],setxattr user.rgw.idtag (54),writefull 0~420,setxattr user.rgw.manifest (738),setxattr user.rgw.acl (175),setxattr user.rgw.content_type (11),setxattr user.rgw.etag (33),call rgw.obj_store_pg_ver,setxattr user.rgw.source_zone (4)] snapc 0=[] ondisk+write+known_if_redirected e4051) currently commit_sent I’ve found similar issue here: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-December/006838.html <http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-December/006838.html> But unfortunately, I can’t find any replies to this question. I suspect what it may be related to objects attributes saving, so, my question is: Anyone else here had similar problems? If so, is there any guidances to solve it? Thanks. Andrey Komarov | system administrator mailto: andrey.koma...@lazada.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
