Hello,> One thing that jumps out at me is that these SSDs are all 480GB models. BlueStore has a certain constant amount of overhead that means that very small OSDs can behave capacity-wise in unexpected ways.
We didn't really had a choice when it came to purchase of these SSDs. Is it mentioned and explained somewhere in the documentation?I've heard that rocksdb was sensitive to device size but I've assumed this issue was fixed while ago.
> I suggest checking your pool attributes and CRUSH rules to ensure that I've checked all pools are on devices they are supposed to be.Every crush rules has device class (and ceph osd pool autoscale-status works correctly)
> I see RGW pools, but no index pool. Is RGW in use, or did you trim the `ceph df` output? If RGW isn’t actually being used, you might want to clean up those unused pools. RGW pools are just leftovers from some experiments. We do not have running RGW daemons on this cluster.
> Has anyone *ever* run `rados bench` against any pool? I suggest looking at `rados ls` for each pool for RADOS objects matching a bench profile/naming. I’ve in the past encountered detritus from bench runs that did’t get cleaned up, e.g. if one ^Cs during the run. A bench cleanup or a very careful pipe through `rados rm` can address that.
Probably not. We usually use randos bench on dedicated pool.s Evacuating all PGs from OSD doesn't lower it's usage. For example1814 ssd 0 1.00000 447 GiB 382 GiB 381 GiB 351 MiB 1.0 GiB 65 GiB 85.44 1.00 0 up I suspect its OSD/bluestore issue since data usage seems not to come from PGs.
Best regards Adam W dniu 5.05.2025 o 16:31, Anthony D'Atri pisze:
we have just discovered new issue with our cephfs cluster. All of our SSD osd have been flagged as nearful. Ceph df on our cluster looks very wrong. Raw storage usage shows that SSDs are almost full, but SSD pools are hardly using any space.One thing that jumps out at me is that these SSDs are all 480GB models. BlueStore has a certain constant amount of overhead that means that very small OSDs can behave capacity-wise in unexpected ways. I suggest checking your pool attributes and CRUSH rules to ensure that * The pools each are using the intended CRUSH rule * Every CRUSH rule specifies a device class, either ssd or hdd. It is all too easy to have a rule in use that does not specify device class, which would mean that SSD capacity is being used by a pool that should only be using HDDs. I see RGW pools, but no index pool. Is RGW in use, or did you trim the `ceph df` output? If RGW isn’t actually being used, you might want to clean up those unused pools. Ah, I see an update with the CRUSH rules … Please send `ceph osd dump | grep pool` * rule 0 replicated_rule item name is `default`, so any pools using this rule will use *all* device classes * rule 1 selects the `nvme` device class. Do you have OSDs tagged with that device class vs `ssd`? * rules 2,3,5,6 select only hdd * rule 4 selects only ssd Check against the pool details output and verify that you don’t have a pool using the wrong CRUSH rule, and especially that no pools are using rule 0.rados -p cephfs.arm-vol.meta | wc -l confirms that there are about 2.02M objects in that pool.Has anyone *ever* run `rados bench` against any pool? I suggest looking at `rados ls` for each pool for RADOS objects matching a bench profile/naming. I’ve in the past encountered detritus from bench runs that did’t get cleaned up, e.g. if one ^Cs during the run. A bench cleanup or a very careful pipe through `rados rm` can address that.Our SSD OSDs are almost full ceph osd df ssd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 807 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 395 MiB 1.3 GiB 64 GiB 85.78 1.00 85 up 812 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.0 GiB 66 GiB 85.23 1.00 74 up 569 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.0 GiB 66 GiB 85.24 1.00 72 upwithout checking any deeper your problem: maybe the filesystem on your ssds need a fstrim?They’re presumably BlueStore and thus don’t have a filesystem on them. Fstrim is more of an issue with filesystems build on RBD volumes. I don’t *think* trimming is a thing for CephFS but I could be wrong.577 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 334 MiB 1.5 GiB 67 GiB 85.12 1.00 72 up 571 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 369 MiB 2.3 GiB 66 GiB 85.30 1.00 86 up 578 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 329 MiB 1.8 GiB 66 GiB 85.19 1.00 68 up 1775 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 379 MiB 1.2 GiB 64 GiB 85.70 1.00 76 up 1838 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 358 MiB 2.3 GiB 66 GiB 85.29 1.00 74 up 1783 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 443 MiB 2.5 GiB 62 GiB 86.05 1.01 98 up 1846 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 382 MiB 1.7 GiB 66 GiB 85.18 1.00 85 up 1854 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 384 MiB 2.4 GiB 66 GiB 85.33 1.00 80 up 1855 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 364 MiB 2.6 GiB 65 GiB 85.37 1.00 78 up 1761 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 368 MiB 1.7 GiB 63 GiB 85.86 1.00 79 up 1826 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 331 MiB 2.2 GiB 66 GiB 85.29 1.00 73 up 1773 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 324 MiB 2.0 GiB 66 GiB 85.24 1.00 73 up 1836 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 390 MiB 2.3 GiB 66 GiB 85.30 1.00 82 up 1771 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 2.3 GiB 66 GiB 85.31 1.00 75 up 1834 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 376 MiB 2.1 GiB 66 GiB 85.27 1.00 79 up 2111 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 396 MiB 2.0 GiB 63 GiB 85.90 1.01 84 up 2112 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 338 MiB 2.2 GiB 66 GiB 85.27 1.00 77 up 2272 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 374 MiB 2.0 GiB 66 GiB 85.24 1.00 76 up 2279 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 332 MiB 2.5 GiB 66 GiB 85.34 1.00 77 up 2276 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 360 MiB 2.4 GiB 66 GiB 85.34 1.00 82 up 2285 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 371 MiB 2.2 GiB 66 GiB 85.27 1.00 79 up 2275 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 368 MiB 2.3 GiB 66 GiB 85.31 1.00 79 up 2282 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 336 MiB 2.0 GiB 66 GiB 85.23 1.00 75 up 2271 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 343 MiB 1.6 GiB 66 GiB 85.15 1.00 81 up 2278 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 356 MiB 1.7 GiB 63 GiB 85.85 1.00 77 up 2277 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 334 MiB 1.3 GiB 64 GiB 85.70 1.00 72 up 2286 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 351 MiB 2.3 GiB 66 GiB 85.30 1.00 74 up 2347 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 362 MiB 2.6 GiB 65 GiB 85.38 1.00 81 up 2348 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 339 MiB 2.0 GiB 66 GiB 85.23 1.00 76 up 573 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 404 MiB 2.7 GiB 65 GiB 85.40 1.00 85 up 580 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 320 MiB 2.3 GiB 60 GiB 86.65 1.01 70 up 169 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 415 MiB 2.0 GiB 66 GiB 85.24 1.00 88 up 171 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 323 MiB 2.5 GiB 66 GiB 85.34 1.00 71 up 1216 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 364 MiB 2.5 GiB 66 GiB 85.35 1.00 76 up 1218 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 334 MiB 1.9 GiB 63 GiB 85.84 1.00 70 up 811 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 377 MiB 2.0 GiB 66 GiB 85.24 1.00 80 up 815 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 419 MiB 2.6 GiB 65 GiB 85.38 1.00 91 up 810 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 312 MiB 2.2 GiB 66 GiB 85.27 1.00 67 up 813 ssd 0.43660 1.00000 447 GiB 384 GiB 381 GiB 340 MiB 2.4 GiB 63 GiB 85.93 1.01 72 up 1760 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 388 MiB 974 MiB 67 GiB 85.01 0.99 77 up 1823 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 366 MiB 2.5 GiB 66 GiB 85.35 1.00 73 up 1762 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 361 MiB 1.4 GiB 67 GiB 85.10 1.00 75 up 1824 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 278 MiB 2.6 GiB 62 GiB 86.06 1.01 65 up 1766 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 377 MiB 1.9 GiB 60 GiB 86.58 1.01 82 up 1830 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 339 MiB 2.6 GiB 65 GiB 85.36 1.00 74 up 1755 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 388 MiB 2.5 GiB 65 GiB 85.36 1.00 77 up 1818 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 2.3 GiB 66 GiB 85.31 1.00 75 up 1774 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 315 MiB 2.2 GiB 66 GiB 85.26 1.00 67 up 1837 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 311 MiB 2.5 GiB 66 GiB 85.33 1.00 68 up 1748 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 339 MiB 2.4 GiB 66 GiB 85.31 1.00 76 up 1812 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 336 MiB 2.2 GiB 66 GiB 85.28 1.00 77 up 1337 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 312 MiB 2.3 GiB 66 GiB 85.30 1.00 64 up 1338 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 331 MiB 1.9 GiB 63 GiB 85.89 1.00 76 up 1091 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 329 MiB 2.5 GiB 66 GiB 85.34 1.00 69 up 1096 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 373 MiB 1.6 GiB 63 GiB 85.85 1.00 82 up 1093 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 379 MiB 2.6 GiB 65 GiB 85.38 1.00 84 up 1098 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 332 MiB 2.3 GiB 66 GiB 85.30 1.00 79 up 1752 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 350 MiB 1.5 GiB 67 GiB 85.11 1.00 80 up 1815 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 349 MiB 2.3 GiB 66 GiB 85.30 1.00 70 up 1754 ssd 0.43660 1.00000 447 GiB 387 GiB 384 GiB 362 MiB 2.0 GiB 60 GiB 86.48 1.01 75 up 1816 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 346 MiB 2.5 GiB 66 GiB 85.35 1.00 72 up 1753 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 348 MiB 1.6 GiB 64 GiB 85.80 1.00 75 up 1817 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 356 MiB 2.4 GiB 66 GiB 85.33 1.00 75 up 1739 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 363 MiB 1.3 GiB 67 GiB 85.09 1.00 75 up 1804 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 365 MiB 2.6 GiB 62 GiB 86.03 1.01 78 up 1763 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 369 MiB 2.6 GiB 62 GiB 86.04 1.01 76 up 1825 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 379 MiB 2.2 GiB 63 GiB 85.91 1.01 83 up 1738 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 354 MiB 1.9 GiB 63 GiB 85.84 1.00 75 up 1802 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 339 MiB 1.6 GiB 63 GiB 85.83 1.00 77 up 1661 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 320 MiB 2.4 GiB 66 GiB 85.33 1.00 65 up 1666 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 367 MiB 2.3 GiB 66 GiB 85.31 1.00 80 up 1663 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 444 MiB 1.9 GiB 63 GiB 85.91 1.01 98 up 1669 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 375 MiB 2.2 GiB 66 GiB 85.29 1.00 76 up 1665 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 341 MiB 2.1 GiB 66 GiB 85.26 1.00 75 up 1671 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 363 MiB 2.4 GiB 66 GiB 85.33 1.00 76 up 1726 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 356 MiB 1.7 GiB 64 GiB 85.78 1.00 77 up 1790 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 349 MiB 2.3 GiB 66 GiB 85.29 1.00 80 up 1765 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 437 MiB 2.5 GiB 62 GiB 86.07 1.01 91 up 1828 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 356 MiB 2.3 GiB 66 GiB 85.31 1.00 78 up 1744 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 355 MiB 2.5 GiB 62 GiB 86.03 1.01 77 up 1807 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 327 MiB 1.9 GiB 60 GiB 86.51 1.01 74 up 1768 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 415 MiB 2.2 GiB 66 GiB 85.29 1.00 88 up 1831 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 332 MiB 2.0 GiB 66 GiB 85.23 1.00 71 up 1756 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 339 MiB 1.3 GiB 67 GiB 85.07 1.00 65 up 1819 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 323 MiB 2.3 GiB 66 GiB 85.28 1.00 67 up 1772 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 407 MiB 1.8 GiB 66 GiB 85.21 1.00 87 up 1835 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 1.5 GiB 67 GiB 85.11 1.00 74 up 1879 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 342 MiB 2.5 GiB 66 GiB 85.34 1.00 70 up 1882 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 310 MiB 2.2 GiB 66 GiB 85.28 1.00 66 up 1880 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 344 MiB 2.6 GiB 65 GiB 85.36 1.00 79 up 1883 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 319 MiB 2.6 GiB 65 GiB 85.36 1.00 65 up 1881 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 313 MiB 2.1 GiB 66 GiB 85.25 1.00 64 up 1884 ssd 0.43660 1.00000 447 GiB 390 GiB 388 GiB 387 MiB 2.1 GiB 57 GiB 87.27 1.02 87 up 1747 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 426 MiB 2.6 GiB 65 GiB 85.38 1.00 87 up 1809 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 359 MiB 1.6 GiB 66 GiB 85.14 1.00 77 up 1751 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 340 MiB 2.6 GiB 65 GiB 85.36 1.00 74 up 1813 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 367 MiB 2.2 GiB 63 GiB 85.93 1.01 81 up 1785 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 403 MiB 2.5 GiB 66 GiB 85.35 1.00 81 up 1848 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 370 MiB 2.5 GiB 66 GiB 85.34 1.00 74 up 1749 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 337 MiB 1.5 GiB 67 GiB 85.12 1.00 74 up 1811 ssd 0.43660 1.00000 447 GiB 388 GiB 385 GiB 335 MiB 2.5 GiB 59 GiB 86.70 1.01 74 up 1725 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 361 MiB 2.5 GiB 65 GiB 85.35 1.00 73 up 1788 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 352 MiB 2.6 GiB 65 GiB 85.36 1.00 74 up 1779 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 363 MiB 1.1 GiB 67 GiB 85.03 0.99 77 up 1843 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 387 MiB 2.4 GiB 66 GiB 85.34 1.00 78 up 1967 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 333 MiB 2.2 GiB 66 GiB 85.28 1.00 66 up 1970 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 382 MiB 1.9 GiB 66 GiB 85.21 1.00 78 up 1968 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 337 MiB 1.9 GiB 66 GiB 85.22 1.00 71 up 1971 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 354 MiB 2.1 GiB 66 GiB 85.26 1.00 79 up 1966 ssd 0.43660 1.00000 447 GiB 384 GiB 381 GiB 352 MiB 1.8 GiB 64 GiB 85.79 1.00 82 up 1969 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 315 MiB 2.3 GiB 63 GiB 85.99 1.01 73 up 1781 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 381 MiB 1.8 GiB 66 GiB 85.20 1.00 81 up 1844 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 330 MiB 2.1 GiB 66 GiB 85.25 1.00 71 up 1743 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 360 MiB 2.1 GiB 63 GiB 85.95 1.01 79 up 1806 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 385 MiB 2.4 GiB 66 GiB 85.34 1.00 78 up 1778 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 354 MiB 2.5 GiB 66 GiB 85.35 1.00 74 up 1839 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 348 MiB 1.6 GiB 66 GiB 85.15 1.00 79 up 1757 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 409 MiB 2.5 GiB 65 GiB 85.36 1.00 82 up 1820 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 335 MiB 2.1 GiB 66 GiB 85.26 1.00 73 up 1758 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 403 MiB 2.5 GiB 63 GiB 86.02 1.01 81 up 1821 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.1 GiB 66 GiB 85.27 1.00 80 up 1787 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 281 MiB 2.1 GiB 66 GiB 85.24 1.00 64 up 1850 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 260 MiB 2.6 GiB 65 GiB 85.36 1.00 59 up 2054 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 318 MiB 1.6 GiB 66 GiB 85.14 1.00 68 up 2057 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.3 GiB 66 GiB 85.31 1.00 78 up 2055 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 343 MiB 2.1 GiB 66 GiB 85.26 1.00 79 up 2058 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 289 MiB 2.5 GiB 63 GiB 85.99 1.01 65 up 2053 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 335 MiB 2.0 GiB 66 GiB 85.23 1.00 74 up 2056 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 362 MiB 2.3 GiB 63 GiB 85.93 1.01 77 up 1735 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 363 MiB 1.8 GiB 66 GiB 85.20 1.00 76 up 1799 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 360 MiB 2.1 GiB 66 GiB 85.27 1.00 74 up 1784 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 361 MiB 1.8 GiB 66 GiB 85.19 1.00 74 up 1847 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 370 MiB 2.4 GiB 66 GiB 85.33 1.00 74 up 1740 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 368 MiB 2.4 GiB 66 GiB 85.33 1.00 72 up 1801 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 362 MiB 2.1 GiB 66 GiB 85.26 1.00 74 up 1776 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 367 MiB 2.1 GiB 66 GiB 85.25 1.00 80 up 1841 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 393 MiB 2.1 GiB 66 GiB 85.26 1.00 86 up 1742 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 358 MiB 1.6 GiB 66 GiB 85.14 1.00 74 up 1805 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 330 MiB 2.6 GiB 62 GiB 86.05 1.01 75 up 1764 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.4 GiB 66 GiB 85.32 1.00 74 up 1827 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 434 MiB 2.3 GiB 63 GiB 86.00 1.01 88 up 2301 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 355 MiB 2.3 GiB 66 GiB 85.30 1.00 79 up 2309 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 315 MiB 1.7 GiB 66 GiB 85.17 1.00 69 up 2302 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 314 MiB 2.5 GiB 66 GiB 85.33 1.00 68 up 2315 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 333 MiB 2.3 GiB 66 GiB 85.30 1.00 70 up 2308 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 356 MiB 2.4 GiB 66 GiB 85.33 1.00 79 up 2316 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 366 MiB 2.3 GiB 66 GiB 85.29 1.00 76 up 1733 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 348 MiB 1.9 GiB 63 GiB 85.86 1.00 74 up 1796 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 352 MiB 2.6 GiB 65 GiB 85.38 1.00 74 up 1782 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 313 MiB 1.4 GiB 64 GiB 85.77 1.00 70 up 1845 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 379 MiB 2.2 GiB 66 GiB 85.29 1.00 81 up 1730 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.2 GiB 66 GiB 85.29 1.00 74 up 1794 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 377 MiB 2.1 GiB 63 GiB 85.94 1.01 80 up 1786 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 381 MiB 2.6 GiB 65 GiB 85.37 1.00 76 up 1849 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 337 MiB 963 MiB 64 GiB 85.63 1.00 76 up 1741 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 358 MiB 1.4 GiB 67 GiB 85.09 1.00 75 up 1803 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 383 MiB 2.7 GiB 65 GiB 85.40 1.00 78 up 1750 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 353 MiB 2.1 GiB 66 GiB 85.26 1.00 70 up 1814 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 351 MiB 772 MiB 67 GiB 84.95 0.99 81 up Both ssd pools run with basic 3x replication and rack failure domain. cephfs.arm-vol.data pool is EC pool on HDD OSDs. I've checked few OSDs and pools are not overlapping over. Cluster runs on 19.2.0 and OSD nodes are ARM64. In emergency I'm planning to just move these pool to HDDs OSD. Do you have any idea what could have filled these SSDs and how to clean it up? Today I've Best regards Adam Prycki _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
smime.p7s
Description: Kryptograficzna sygnatura S/MIME
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
