Hi " It’s the clients disabling write-back cache because it’s risky to have dirty data on a cluster getting full. Last time I checked it was hardcoded in kernel client so no way to disable this. Only remedy is increasing ratios like you stated. "
The thing is, why is cephfs running fine again, if the highest osd is above the backfillfull_ratio? e.g. Nearful set to 85% , backfillfull set to 90%, and my fullest disk 90.01%, Then everything is 100% again. Issue is when the fullest OSD is between the values of nearful_ratio & backfillfull_ratio. Even if nearfull is 50%, backfillfull_ratio 55% and fullest disk 53%. Then it still goes into limp mode. Regards On Thu, Jun 19, 2025 at 9:03 AM Jeremi-Ernst Avenant <jer...@idia.ac.za> wrote: > Good day > > We've been struggling with this issue since we've upgraded post 16.2.11 to > 16.2.15 and now up to Reef 18.2.7. We didn't have this issue prior to > upgrading from 16.2.11 to 16.2.15. > > The moment the fullest OSD % is between the nearful and backfillful value, > CephFS goes into a limp mode. Meaning our Client IO goes from ~30 *GiB*/s > to 100 *MiB*/s. Doesn't matter what the threshold values are set. > > If my nearful% is 75% and backfillfull 80%, with the highest OSD 75.01% it > will limp. > If my nearful% is 85% and backfillfull 90%, with the highest OSD 85.01% it > will go into limp mode. > > The only way to get CephFS operational again, is to set both ratios to > either far below or above the highest OSD value. > > e.g. if the fullest is 72% , I need to set it to ceph osd > set-nearfull-ratio 0.76 & ceph osd set-backfillfull-ratio 0.81 > > When I deploy a brand new cluster from scratch on our testbed I get the > same issue, Cephadm / Reef 18.2.4. > > Back in the days I was able to manipulate these settings with the > following as well, but I think since Pacific they are no longer tuneable: > > ceph tell osd.1231 config show | grep -E > 'osd_nearfull_ratio|osd_backfillfull_ratio|osd_full_ratio' > "mon_osd_backfillfull_ratio": "0.900000", "mon_osd_full_ratio": "0.950000", > "mon_osd_nearfull_ratio": "0.850000", > > ceph tell osd.$osd injectargs '--osd_nearfull_ratio=0.85' > ceph tell osd.$osd injectargs '--osd_backfillfull_ratio=0.90' > ceph tell osd.$osd injectargs '--osd_full_ratio=0.95' > > > URL to the issue: https://tracker.ceph.com/issues/70129 > > Any ideas would be greatly appreciated. > > -- > > > > *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist > Inter-University Institute for Data Intensive Astronomy > 5th Floor, Department of Physics and Astronomy, > University of Cape Town > > Tel: 021 959 4137 <0219592327> > Web: www.idia.ac.za | www.ilifu.ac.za > E-mail (IDIA): jer...@idia.ac.za <mfu...@idia.ac.za> > Rondebosch, Cape Town, 7600, South Africa > -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jer...@idia.ac.za <mfu...@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
