On Tue, Jan 21, 2025 at 7:12 AM Anthony D'Atri <[email protected]> wrote: > > On Jan 21, 2025, at 7:59 AM, Kasper Rasmussen > > <[email protected]> wrote: > > > > 1 - Why do this result in such a high - objects degraded - percentage? > > I suspect that’s a function of the new topology having changed the mappings > of multiple OSDs for given PGs. It’s subtle, but when you move hosts into > rack CRUSH buckets, that’s a different set of inputs into the CRUSH hash > function, so the mappings that come out are different, even though you > haven’t changed the rules and would think that hosts are hosts.
Also, in the process of moving the hosts one by one, each step creates a new topology which can change the ordering of hosts, incrementally putting things out of whack. > > 2 - Why do PGs get undersized? > > That often means that CRUSH can’t find a complete set of placements. In your > situation maybe those would resolve themselves when you unleash the recovery > hounds. We started noticing this kind of issue around pacific, but haven't fully tracked down what broke yet. See https://tracker.ceph.com/issues/56046 for similar. Undersized or degraded should only happen -- by design -- if objects were modified while the PG did not have 3 OSDs up and acting. Kaspar: I assume the cluster was idle during your tests? Also -- can you reproduce it without norecover/nobackfill set ? Could you simplify your reproducer down to: > HEALTH_OK > ceph osd crush move ksr-ceph-osd1 rack=rack1 > ceph pg ls undersized / degraded # get a pgid of a degraded PG > ceph pg $pgid query Cheers, dan -- Dan van der Ster CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | [email protected] _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
