Hi,
We are currently using Jewel 10.2.7 and recently, we have been experiencing
some issues with objects being deleted using the gc. After a bucket was
unsuccessfully deleted using –purge-objects (first error next discussed
occurred), all of the rgw’s are occasionally becoming unresponsive and
require a restart of the processes before they will accept requests again.
On investigation of the garbage collection, it has an enormous list which
we are struggling to count the length of, but seem stuck on a particular
object which is not updating, shown in the logs below:
2018-04-23 15:16:04.101660 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.290071.4_XXXXXXX/XXXX/XX/XX/XXXXXXX.ZIP
2018-04-23 15:16:04.104231 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_1
2018-04-23 15:16:04.105541 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_2
2018-04-23 15:16:04.176235 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_3
2018-04-23 15:16:04.178435 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_4
2018-04-23 15:16:04.250883 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_5
2018-04-23 15:16:04.297912 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_6
2018-04-23 15:16:04.298803 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_7
2018-04-23 15:16:04.320202 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_8
2018-04-23 15:16:04.340124 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_9
2018-04-23 15:16:04.383924 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_10
2018-04-23 15:16:04.386865 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_11
2018-04-23 15:16:04.389067 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_12
2018-04-23 15:16:04.413938 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_13
2018-04-23 15:16:04.487977 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_14
2018-04-23 15:16:04.544235 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_1
2018-04-23 15:16:04.546978 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_2
2018-04-23 15:16:04.598644 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_3
2018-04-23 15:16:04.629519 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_4
2018-04-23 15:16:04.700492 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_5
2018-04-23 15:16:04.765798 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_6
2018-04-23 15:16:04.772774 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_7
2018-04-23 15:16:04.846379 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_8
2018-04-23 15:16:04.935023 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_9
2018-04-23 15:16:04.937229 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_10
2018-04-23 15:16:04.968289 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_11
2018-04-23 15:16:05.005194 7f1fdcc29a00 0 gc::process: removing
.rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_12
We seem completely unable to get this deleted, and nothing else of
immediate concern is flagging up as a potential cause of all RGWs become
unresponsive at the same time. On the bucket containing this object (the
one we originally tried to purge), I have attempted a further purge passing
the “—bypass-gc” parameter to it, but this also resulted in all rgws
becoming unresponsive within 30 minutes and so I terminated the operation
and restarted the rgws again.
The bucket we attempted to remove has no shards and I have attached the
details below. 90% of the contents of the bucket have already been
successfully removed to our knowledge, and the bucket had no sharding (old
bucket, sharding is now active for new buckets).
root@ceph-rgw-1:~# radosgw-admin --id rgw.ceph-rgw-1 bucket stats
--bucket=xxxxxxxxxxxx
{
"bucket": "xxxxxxxxxxxx",
"pool": ".rgw.buckets",
"index_pool": ".rgw.buckets.index",
"id": "default.290071.4",
"marker": "default.290071.4",
"owner": "yyyyyyyyyy",
"ver": "0#107938549",
"master_ver": "0#0",
"mtime": "2014-10-24 14:58:48.955805",
"max_marker": "0#",
"usage": {
"rgw.none": {
"size_kb": 0,
"size_kb_actual": 0,
"num_objects": 0
},
"rgw.main": {
"size_kb": 186685939,
"size_kb_actual": 189914068,
"num_objects": 1419528
},
"rgw.multimeta": {
"size_kb": 0,
"size_kb_actual": 0,
"num_objects": 24
}
},
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
}
}
If anyone has any thoughts, they’d be greatly appreciated!
Kind Regards,
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
