Hi, We have a discussion going on about which is the correct flag to use for some maintenance on an OSD, should it be "noout" or "norebalance"? This was sparked because we need to take an OSD out of service for a short while to upgrade the firmware.
One school of thought is: - "ceph norebalance" prevents automatic rebalancing of data between OSDs, which Ceph does to ensure all OSDs have roughly the same amount of data. - "ceph noout" on the other hand prevents OSDs from being marked as out-of-service during maintenance, which helps maintain cluster performance and availability. - Additionally, if another OSD fails while the "norebalance" flag is set, the data redundancy and fault tolerance of the Ceph cluster may be compromised. - So if we're going to maintain the performance and reliability we need to set the "ceph noout" flag to prevent the OSD from being marked as OOS during maintenance and allow the automatic data redistribution feature of Ceph to work as intended. The other opinion is: - With the noout flag set, Ceph clients are forced to think that OSD exists and is accessible - so they continue sending requests to such OSD. The OSD also remains in the crush map without any signs that it is actually out. If an additional OSD fails in the cluster with the noout flag set, Ceph is forced to continue thinking that this new failed OSD is OK. It leads to stalled or delayed response from the OSD side to clients. - Norebalance instead takes into account the in/out OSD status, but prevents data rebalance. Clients are also aware of the real OSD status, so no requests go to the OSD which is actually out. If an additional OSD fails - only the required temporary PG are created to maintain at least 2 existing copies of the same data (well, generally it is set by the pool min size). The upstream docs seem pretty clear that noout should be used for maintenance (https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-osd/), but the second opinion strongly suggests that norebalance is actually better and the Ceph docs are out of date. So what is the feedback from the wider community? Thanks, Will _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
