Hi all,
Still a real novice here and we didn't set up our initial RGW cluster very
well. We have 134 osds and set up our RGW pool with only 64 PGs, thus not all
of our OSDs got data and now we have one that is 95% full.
This apparently has put the cluster into a HEALTH_ERR condition:
[root@carf-ceph-osd01 ~]# ceph health detail
HEALTH_ERR full flag(s) set; 1 full osd(s); 1 pools have many more objects per
pg than average; application not enabled on 6 pool(s); too few PGs per OSD (26
< min 30)
OSDMAP_FLAGS full flag(s) set
OSD_FULL 1 full osd(s)
osd.5 is full
MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average
pool carf01.rgw.buckets.data objects per pg (602762) is more than 18.3752
times cluster average (32803)
There is plenty of space on most of the OSDs and don't know how to go about
fixing this situation. If we update the pg_num and pgp_num settings for this
pool, can we rebalance the data across the OSDs?
Also, seems like this is causing a problem with the RGWs, which was reporting
this error in the logs:
2017-10-16 16:36:47.534461 7fffe6c5c700 1 heartbeat_map is_healthy
'RGWAsyncRadosProcessor::m_tp thread 0x7fffdc447700' had timed out after 600
After trying to restart the RGW, we see this now:
2017-10-17 10:40:38.517002 7fffe6c5c700 1 heartbeat_map is_healthy
'RGWAsyncRadosProcessor::m_tp thread 0x7fffddc4a700' had timed out after 600
2017-10-17 10:40:42.124046 7ffff7fd4e00 0 deferred set uid:gid to 167:167
(ceph:ceph)
2017-10-17 10:40:42.124162 7ffff7fd4e00 0 ceph version 12.2.0
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown),
pid 65313
2017-10-17 10:40:42.245259 7ffff7fd4e00 0 client.769905.objecter FULL, paused
modify 0x55555662fb00 tid 0
2017-10-17 10:45:42.124283 7fffe7bcf700 -1 Initialization timeout, failed to
initialize
2017-10-17 10:45:42.353496 7ffff7fd4e00 0 deferred set uid:gid to 167:167
(ceph:ceph)
2017-10-17 10:45:42.353618 7ffff7fd4e00 0 ceph version 12.2.0
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown),
pid 71842
2017-10-17 10:45:42.388621 7ffff7fd4e00 0 client.769986.objecter FULL, paused
modify 0x55555662fb00 tid 0
2017-10-17 10:50:42.353731 7fffe7bcf700 -1 Initialization timeout, failed to
initialize
Seems pretty evident that the "FULL, paused" is a problem. So if I fix the
first issue the RGW should be ok after?
Thanks in advance,
-Bryan
________________________________
Note: This email is for the confidential use of the named addressee(s) only and
may contain proprietary, confidential or privileged information. If you are not
the intended recipient, you are hereby notified that any review, dissemination
or copying of this email is strictly prohibited, and to please notify the
sender immediately and destroy this email and any attachments. Email
transmission cannot be guaranteed to be secure or error-free. The Company,
therefore, does not make any guarantees as to the completeness or accuracy of
this email or any attachments. This email is for informational purposes only
and does not constitute a recommendation, offer, request or solicitation of any
kind to buy, sell, subscribe, redeem or perform any type of transaction of a
financial product.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
