> we have just discovered new issue with our cephfs cluster. > All of our SSD osd have been flagged as nearful. > > Ceph df on our cluster looks very wrong. Raw storage usage shows that SSDs > are almost full, but SSD pools are hardly using any space.
One thing that jumps out at me is that these SSDs are all 480GB models. BlueStore has a certain constant amount of overhead that means that very small OSDs can behave capacity-wise in unexpected ways. I suggest checking your pool attributes and CRUSH rules to ensure that * The pools each are using the intended CRUSH rule * Every CRUSH rule specifies a device class, either ssd or hdd. It is all too easy to have a rule in use that does not specify device class, which would mean that SSD capacity is being used by a pool that should only be using HDDs. I see RGW pools, but no index pool. Is RGW in use, or did you trim the `ceph df` output? If RGW isn’t actually being used, you might want to clean up those unused pools. Ah, I see an update with the CRUSH rules … Please send `ceph osd dump | grep pool` * rule 0 replicated_rule item name is `default`, so any pools using this rule will use *all* device classes * rule 1 selects the `nvme` device class. Do you have OSDs tagged with that device class vs `ssd`? * rules 2,3,5,6 select only hdd * rule 4 selects only ssd Check against the pool details output and verify that you don’t have a pool using the wrong CRUSH rule, and especially that no pools are using rule 0. > rados -p cephfs.arm-vol.meta | wc -l confirms that there are about 2.02M > objects in that pool. Has anyone *ever* run `rados bench` against any pool? I suggest looking at `rados ls` for each pool for RADOS objects matching a bench profile/naming. I’ve in the past encountered detritus from bench runs that did’t get cleaned up, e.g. if one ^Cs during the run. A bench cleanup or a very careful pipe through `rados rm` can address that. > Our SSD OSDs are almost full > > ceph osd df ssd > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META > AVAIL %USE VAR PGS STATUS > 807 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 395 MiB 1.3 GiB > 64 GiB 85.78 1.00 85 up > 812 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.0 GiB > 66 GiB 85.23 1.00 74 up > 569 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.0 GiB > 66 GiB 85.24 1.00 72 up > without checking any deeper your problem: maybe the filesystem on your ssds > need a fstrim? They’re presumably BlueStore and thus don’t have a filesystem on them. Fstrim is more of an issue with filesystems build on RBD volumes. I don’t *think* trimming is a thing for CephFS but I could be wrong. > 577 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 334 MiB 1.5 GiB > 67 GiB 85.12 1.00 72 up > 571 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 369 MiB 2.3 GiB > 66 GiB 85.30 1.00 86 up > 578 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 329 MiB 1.8 GiB > 66 GiB 85.19 1.00 68 up > 1775 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 379 MiB 1.2 GiB > 64 GiB 85.70 1.00 76 up > 1838 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 358 MiB 2.3 GiB > 66 GiB 85.29 1.00 74 up > 1783 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 443 MiB 2.5 GiB > 62 GiB 86.05 1.01 98 up > 1846 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 382 MiB 1.7 GiB > 66 GiB 85.18 1.00 85 up > 1854 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 384 MiB 2.4 GiB > 66 GiB 85.33 1.00 80 up > 1855 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 364 MiB 2.6 GiB > 65 GiB 85.37 1.00 78 up > 1761 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 368 MiB 1.7 GiB > 63 GiB 85.86 1.00 79 up > 1826 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 331 MiB 2.2 GiB > 66 GiB 85.29 1.00 73 up > 1773 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 324 MiB 2.0 GiB > 66 GiB 85.24 1.00 73 up > 1836 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 390 MiB 2.3 GiB > 66 GiB 85.30 1.00 82 up > 1771 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 2.3 GiB > 66 GiB 85.31 1.00 75 up > 1834 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 376 MiB 2.1 GiB > 66 GiB 85.27 1.00 79 up > 2111 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 396 MiB 2.0 GiB > 63 GiB 85.90 1.01 84 up > 2112 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 338 MiB 2.2 GiB > 66 GiB 85.27 1.00 77 up > 2272 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 374 MiB 2.0 GiB > 66 GiB 85.24 1.00 76 up > 2279 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 332 MiB 2.5 GiB > 66 GiB 85.34 1.00 77 up > 2276 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 360 MiB 2.4 GiB > 66 GiB 85.34 1.00 82 up > 2285 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 371 MiB 2.2 GiB > 66 GiB 85.27 1.00 79 up > 2275 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 368 MiB 2.3 GiB > 66 GiB 85.31 1.00 79 up > 2282 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 336 MiB 2.0 GiB > 66 GiB 85.23 1.00 75 up > 2271 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 343 MiB 1.6 GiB > 66 GiB 85.15 1.00 81 up > 2278 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 356 MiB 1.7 GiB > 63 GiB 85.85 1.00 77 up > 2277 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 334 MiB 1.3 GiB > 64 GiB 85.70 1.00 72 up > 2286 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 351 MiB 2.3 GiB > 66 GiB 85.30 1.00 74 up > 2347 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 362 MiB 2.6 GiB > 65 GiB 85.38 1.00 81 up > 2348 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 339 MiB 2.0 GiB > 66 GiB 85.23 1.00 76 up > 573 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 404 MiB 2.7 GiB > 65 GiB 85.40 1.00 85 up > 580 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 320 MiB 2.3 GiB > 60 GiB 86.65 1.01 70 up > 169 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 415 MiB 2.0 GiB > 66 GiB 85.24 1.00 88 up > 171 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 323 MiB 2.5 GiB > 66 GiB 85.34 1.00 71 up > 1216 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 364 MiB 2.5 GiB > 66 GiB 85.35 1.00 76 up > 1218 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 334 MiB 1.9 GiB > 63 GiB 85.84 1.00 70 up > 811 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 377 MiB 2.0 GiB > 66 GiB 85.24 1.00 80 up > 815 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 419 MiB 2.6 GiB > 65 GiB 85.38 1.00 91 up > 810 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 312 MiB 2.2 GiB > 66 GiB 85.27 1.00 67 up > 813 ssd 0.43660 1.00000 447 GiB 384 GiB 381 GiB 340 MiB 2.4 GiB > 63 GiB 85.93 1.01 72 up > 1760 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 388 MiB 974 MiB > 67 GiB 85.01 0.99 77 up > 1823 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 366 MiB 2.5 GiB > 66 GiB 85.35 1.00 73 up > 1762 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 361 MiB 1.4 GiB > 67 GiB 85.10 1.00 75 up > 1824 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 278 MiB 2.6 GiB > 62 GiB 86.06 1.01 65 up > 1766 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 377 MiB 1.9 GiB > 60 GiB 86.58 1.01 82 up > 1830 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 339 MiB 2.6 GiB > 65 GiB 85.36 1.00 74 up > 1755 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 388 MiB 2.5 GiB > 65 GiB 85.36 1.00 77 up > 1818 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 2.3 GiB > 66 GiB 85.31 1.00 75 up > 1774 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 315 MiB 2.2 GiB > 66 GiB 85.26 1.00 67 up > 1837 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 311 MiB 2.5 GiB > 66 GiB 85.33 1.00 68 up > 1748 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 339 MiB 2.4 GiB > 66 GiB 85.31 1.00 76 up > 1812 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 336 MiB 2.2 GiB > 66 GiB 85.28 1.00 77 up > 1337 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 312 MiB 2.3 GiB > 66 GiB 85.30 1.00 64 up > 1338 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 331 MiB 1.9 GiB > 63 GiB 85.89 1.00 76 up > 1091 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 329 MiB 2.5 GiB > 66 GiB 85.34 1.00 69 up > 1096 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 373 MiB 1.6 GiB > 63 GiB 85.85 1.00 82 up > 1093 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 379 MiB 2.6 GiB > 65 GiB 85.38 1.00 84 up > 1098 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 332 MiB 2.3 GiB > 66 GiB 85.30 1.00 79 up > 1752 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 350 MiB 1.5 GiB > 67 GiB 85.11 1.00 80 up > 1815 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 349 MiB 2.3 GiB > 66 GiB 85.30 1.00 70 up > 1754 ssd 0.43660 1.00000 447 GiB 387 GiB 384 GiB 362 MiB 2.0 GiB > 60 GiB 86.48 1.01 75 up > 1816 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 346 MiB 2.5 GiB > 66 GiB 85.35 1.00 72 up > 1753 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 348 MiB 1.6 GiB > 64 GiB 85.80 1.00 75 up > 1817 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 356 MiB 2.4 GiB > 66 GiB 85.33 1.00 75 up > 1739 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 363 MiB 1.3 GiB > 67 GiB 85.09 1.00 75 up > 1804 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 365 MiB 2.6 GiB > 62 GiB 86.03 1.01 78 up > 1763 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 369 MiB 2.6 GiB > 62 GiB 86.04 1.01 76 up > 1825 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 379 MiB 2.2 GiB > 63 GiB 85.91 1.01 83 up > 1738 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 354 MiB 1.9 GiB > 63 GiB 85.84 1.00 75 up > 1802 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 339 MiB 1.6 GiB > 63 GiB 85.83 1.00 77 up > 1661 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 320 MiB 2.4 GiB > 66 GiB 85.33 1.00 65 up > 1666 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 367 MiB 2.3 GiB > 66 GiB 85.31 1.00 80 up > 1663 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 444 MiB 1.9 GiB > 63 GiB 85.91 1.01 98 up > 1669 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 375 MiB 2.2 GiB > 66 GiB 85.29 1.00 76 up > 1665 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 341 MiB 2.1 GiB > 66 GiB 85.26 1.00 75 up > 1671 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 363 MiB 2.4 GiB > 66 GiB 85.33 1.00 76 up > 1726 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 356 MiB 1.7 GiB > 64 GiB 85.78 1.00 77 up > 1790 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 349 MiB 2.3 GiB > 66 GiB 85.29 1.00 80 up > 1765 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 437 MiB 2.5 GiB > 62 GiB 86.07 1.01 91 up > 1828 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 356 MiB 2.3 GiB > 66 GiB 85.31 1.00 78 up > 1744 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 355 MiB 2.5 GiB > 62 GiB 86.03 1.01 77 up > 1807 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 327 MiB 1.9 GiB > 60 GiB 86.51 1.01 74 up > 1768 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 415 MiB 2.2 GiB > 66 GiB 85.29 1.00 88 up > 1831 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 332 MiB 2.0 GiB > 66 GiB 85.23 1.00 71 up > 1756 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 339 MiB 1.3 GiB > 67 GiB 85.07 1.00 65 up > 1819 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 323 MiB 2.3 GiB > 66 GiB 85.28 1.00 67 up > 1772 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 407 MiB 1.8 GiB > 66 GiB 85.21 1.00 87 up > 1835 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 1.5 GiB > 67 GiB 85.11 1.00 74 up > 1879 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 342 MiB 2.5 GiB > 66 GiB 85.34 1.00 70 up > 1882 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 310 MiB 2.2 GiB > 66 GiB 85.28 1.00 66 up > 1880 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 344 MiB 2.6 GiB > 65 GiB 85.36 1.00 79 up > 1883 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 319 MiB 2.6 GiB > 65 GiB 85.36 1.00 65 up > 1881 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 313 MiB 2.1 GiB > 66 GiB 85.25 1.00 64 up > 1884 ssd 0.43660 1.00000 447 GiB 390 GiB 388 GiB 387 MiB 2.1 GiB > 57 GiB 87.27 1.02 87 up > 1747 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 426 MiB 2.6 GiB > 65 GiB 85.38 1.00 87 up > 1809 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 359 MiB 1.6 GiB > 66 GiB 85.14 1.00 77 up > 1751 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 340 MiB 2.6 GiB > 65 GiB 85.36 1.00 74 up > 1813 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 367 MiB 2.2 GiB > 63 GiB 85.93 1.01 81 up > 1785 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 403 MiB 2.5 GiB > 66 GiB 85.35 1.00 81 up > 1848 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 370 MiB 2.5 GiB > 66 GiB 85.34 1.00 74 up > 1749 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 337 MiB 1.5 GiB > 67 GiB 85.12 1.00 74 up > 1811 ssd 0.43660 1.00000 447 GiB 388 GiB 385 GiB 335 MiB 2.5 GiB > 59 GiB 86.70 1.01 74 up > 1725 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 361 MiB 2.5 GiB > 65 GiB 85.35 1.00 73 up > 1788 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 352 MiB 2.6 GiB > 65 GiB 85.36 1.00 74 up > 1779 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 363 MiB 1.1 GiB > 67 GiB 85.03 0.99 77 up > 1843 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 387 MiB 2.4 GiB > 66 GiB 85.34 1.00 78 up > 1967 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 333 MiB 2.2 GiB > 66 GiB 85.28 1.00 66 up > 1970 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 382 MiB 1.9 GiB > 66 GiB 85.21 1.00 78 up > 1968 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 337 MiB 1.9 GiB > 66 GiB 85.22 1.00 71 up > 1971 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 354 MiB 2.1 GiB > 66 GiB 85.26 1.00 79 up > 1966 ssd 0.43660 1.00000 447 GiB 384 GiB 381 GiB 352 MiB 1.8 GiB > 64 GiB 85.79 1.00 82 up > 1969 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 315 MiB 2.3 GiB > 63 GiB 85.99 1.01 73 up > 1781 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 381 MiB 1.8 GiB > 66 GiB 85.20 1.00 81 up > 1844 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 330 MiB 2.1 GiB > 66 GiB 85.25 1.00 71 up > 1743 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 360 MiB 2.1 GiB > 63 GiB 85.95 1.01 79 up > 1806 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 385 MiB 2.4 GiB > 66 GiB 85.34 1.00 78 up > 1778 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 354 MiB 2.5 GiB > 66 GiB 85.35 1.00 74 up > 1839 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 348 MiB 1.6 GiB > 66 GiB 85.15 1.00 79 up > 1757 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 409 MiB 2.5 GiB > 65 GiB 85.36 1.00 82 up > 1820 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 335 MiB 2.1 GiB > 66 GiB 85.26 1.00 73 up > 1758 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 403 MiB 2.5 GiB > 63 GiB 86.02 1.01 81 up > 1821 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.1 GiB > 66 GiB 85.27 1.00 80 up > 1787 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 281 MiB 2.1 GiB > 66 GiB 85.24 1.00 64 up > 1850 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 260 MiB 2.6 GiB > 65 GiB 85.36 1.00 59 up > 2054 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 318 MiB 1.6 GiB > 66 GiB 85.14 1.00 68 up > 2057 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.3 GiB > 66 GiB 85.31 1.00 78 up > 2055 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 343 MiB 2.1 GiB > 66 GiB 85.26 1.00 79 up > 2058 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 289 MiB 2.5 GiB > 63 GiB 85.99 1.01 65 up > 2053 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 335 MiB 2.0 GiB > 66 GiB 85.23 1.00 74 up > 2056 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 362 MiB 2.3 GiB > 63 GiB 85.93 1.01 77 up > 1735 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 363 MiB 1.8 GiB > 66 GiB 85.20 1.00 76 up > 1799 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 360 MiB 2.1 GiB > 66 GiB 85.27 1.00 74 up > 1784 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 361 MiB 1.8 GiB > 66 GiB 85.19 1.00 74 up > 1847 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 370 MiB 2.4 GiB > 66 GiB 85.33 1.00 74 up > 1740 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 368 MiB 2.4 GiB > 66 GiB 85.33 1.00 72 up > 1801 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 362 MiB 2.1 GiB > 66 GiB 85.26 1.00 74 up > 1776 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 367 MiB 2.1 GiB > 66 GiB 85.25 1.00 80 up > 1841 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 393 MiB 2.1 GiB > 66 GiB 85.26 1.00 86 up > 1742 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 358 MiB 1.6 GiB > 66 GiB 85.14 1.00 74 up > 1805 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 330 MiB 2.6 GiB > 62 GiB 86.05 1.01 75 up > 1764 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.4 GiB > 66 GiB 85.32 1.00 74 up > 1827 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 434 MiB 2.3 GiB > 63 GiB 86.00 1.01 88 up > 2301 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 355 MiB 2.3 GiB > 66 GiB 85.30 1.00 79 up > 2309 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 315 MiB 1.7 GiB > 66 GiB 85.17 1.00 69 up > 2302 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 314 MiB 2.5 GiB > 66 GiB 85.33 1.00 68 up > 2315 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 333 MiB 2.3 GiB > 66 GiB 85.30 1.00 70 up > 2308 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 356 MiB 2.4 GiB > 66 GiB 85.33 1.00 79 up > 2316 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 366 MiB 2.3 GiB > 66 GiB 85.29 1.00 76 up > 1733 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 348 MiB 1.9 GiB > 63 GiB 85.86 1.00 74 up > 1796 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 352 MiB 2.6 GiB > 65 GiB 85.38 1.00 74 up > 1782 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 313 MiB 1.4 GiB > 64 GiB 85.77 1.00 70 up > 1845 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 379 MiB 2.2 GiB > 66 GiB 85.29 1.00 81 up > 1730 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.2 GiB > 66 GiB 85.29 1.00 74 up > 1794 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 377 MiB 2.1 GiB > 63 GiB 85.94 1.01 80 up > 1786 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 381 MiB 2.6 GiB > 65 GiB 85.37 1.00 76 up > 1849 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 337 MiB 963 MiB > 64 GiB 85.63 1.00 76 up > 1741 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 358 MiB 1.4 GiB > 67 GiB 85.09 1.00 75 up > 1803 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 383 MiB 2.7 GiB > 65 GiB 85.40 1.00 78 up > 1750 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 353 MiB 2.1 GiB > 66 GiB 85.26 1.00 70 up > 1814 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 351 MiB 772 MiB > 67 GiB 84.95 0.99 81 up > > Both ssd pools run with basic 3x replication and rack failure domain. > cephfs.arm-vol.data pool is EC pool on HDD OSDs. > I've checked few OSDs and pools are not overlapping over. > > Cluster runs on 19.2.0 and OSD nodes are ARM64. > In emergency I'm planning to just move these pool to HDDs OSD. > > Do you have any idea what could have filled these SSDs and how to clean it up? > > Today I've > > > Best regards > Adam Prycki > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
