Hopefully someone can sanity check me here, but I'm getting the feeling that the MAX AVAIL in ceph df isn't reporting the correct value in 14.2.8 (mon/mgr/mds are .8, most OSDs are .7)
> RAW STORAGE: > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 530 TiB 163 TiB 366 TiB 367 TiB 69.19 > ssd 107 TiB 37 TiB 70 TiB 70 TiB 65.33 > TOTAL 637 TiB 201 TiB 436 TiB 437 TiB 68.54 > > POOLS: > POOL ID STORED OBJECTS USED > %USED MAX AVAIL > fs-metadata 16 44 GiB 4.16M 44 GiB > 0.25 5.6 TiB > cephfs-hdd-3x 17 46 TiB 109.54M 144 TiB > 61.81 30 TiB > objects-hybrid 20 46 TiB 537.08M 187 TiB > 91.71 5.6 TiB > objects-hdd 24 224 GiB 50.81k 676 GiB > 0.74 30 TiB > rbd-hybrid 29 3.8 TiB 1.19M 11 TiB > 40.38 5.6 TiB > device_health_metrics 33 270 MiB 327 270 MiB > 0 30 TiB > rbd-ssd 34 4.2 TiB 1.19M 12 TiB > 41.55 5.6 TiB > cephfs-hdd-ec73 37 42 TiB 30.35M 72 TiB > 44.78 62 TiB I have a few pools that don't look like they are calculating the available storage for that pool correctly. Specifically, any of my hybrid pools (20,29) or all-SSD pools (16,34). For my hybrid pools, I have a crush rule of take 1 of host in the ssd root, take -1 chassis in the hdd root. For my ssd pools, I have a crush rule of take 0 of host in the ssd root. Now I have 60 ssd osds 1.92T each, and sadly distribution is imperfect (leaving those issues out of this), and I have plenty of underfull and overfull osds, which I am trying to manually reweighs to get my most full's down to free up space: > $ ceph osd df class ssd | sort -k17 > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL > %USE VAR PGS STATUS > MIN/MAX VAR: 0.80/1.21 STDDEV: 5.56 > TOTAL 107 TiB 70 TiB 68 TiB 163 GiB 431 GiB 37 TiB > 65.33 > 28 ssd 1.77879 1.00000 1.8 TiB 951 GiB 916 GiB 7.1 GiB 5.8 GiB 871 GiB > 52.20 0.80 68 up > 33 ssd 1.77879 1.00000 1.8 TiB 1.0 TiB 1010 GiB 6.0 MiB 5.9 GiB 777 GiB > 57.33 0.88 74 up > 47 ssd 1.77879 1.00000 1.8 TiB 1.0 TiB 1011 GiB 6.7 GiB 6.4 GiB 776 GiB > 57.38 0.88 75 up > [SNIP] > 57 ssd 1.77879 0.98000 1.8 TiB 1.4 TiB 1.3 TiB 6.2 GiB 8.6 GiB 417 GiB > 77.08 1.18 102 up > 107 ssd 1.80429 1.00000 1.8 TiB 1.4 TiB 1.3 TiB 7.0 GiB 8.7 GiB 422 GiB > 77.15 1.18 102 up > 50 ssd 1.77879 1.00000 1.8 TiB 1.4 TiB 1.4 TiB 5.5 MiB 8.6 GiB 381 GiB > 79.10 1.21 105 up > 60 ssd 1.77879 0.92000 1.8 TiB 1.4 TiB 1.4 TiB 6.2 MiB 9.0 GiB 379 GiB > 79.17 1.21 105 up That said, as a straw man argument, ~380GiB free, times 60 OSDs, should be ~22.8TiB free, if all OSD's grew evenly, which they won't, which is still far short of 37TiB raw free, as expected. However, what doesn't track is the 5.6TiB available at the pools level, even for a 3x replicated pool (5.6*3=16.8TiB, which is about 34% less than my napkin math, which would be 22.8/3=7.6TiB. But what tracks even less is the hybrid pools, which use 1/3 of what the 3x-replicated data consumes. Meaning if my napkin math is right, should show ~22.8TiB free. Am I grossly mis-understanding how this is calculated? Maybe this is fixed in Octopus? Just trying to get a grasp on what I'm seeing not matching expectations. Thanks, Reed
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
