What's "ceph osd df" look like? The thing here will likely be juggling % to get you unlocked.
Jul 10, 2025 16:38:08 Devender Singh <deven...@netskrt.io>: > > Hello all > > > Need some help why nothing is happening… version is 19.2.2. > Tried repairing the pgs, repair and force-backfill and force-recovery but > nothing initiating backfills. > > # ceph -v > ceph version 19.2.2 > > # ceph -s > cluster: > id: 79d4fade-ffa9-11eb-a57d-f398f000cfc1 > health: HEALTH_WARN > 1 OSD(s) experiencing slow operations in BlueStore > nodeep-scrub flag(s) set > Degraded data redundancy: 6114281/137592495 objects degraded > (4.444%), 124 pgs degraded, 124 pgs undersized > 270 pgs not deep-scrubbed in time > 1 subtrees have overcommitted pool target_size_bytes > > services: > mon: 3 daemons, quorum ld-converged03n,ld-converged01n,ld-converged02n > (age 5d) > mgr: ld-converged01n.gnxoye(active, since 5d), standbys: > ld-converged02n.odvopy > mds: 4/4 daemons up, 2 standby, 2 hot standby > osd: 24 osds: 24 up (since 29h), 24 in (since 29h); 292 remapped pgs > flags nodeep-scrub > rgw: 2 daemons active (2 hosts, 1 zones) > > data: > volumes: 4/4 healthy > pools: 24 pools, 1345 pgs > objects: 32.22M objects, 98 TiB > usage: 196 TiB used, 83 TiB / 279 TiB avail > pgs: 6114281/137592495 objects degraded (4.444%) > 14349193/137592495 objects misplaced (10.429%) > 864 active+clean > 237 active+clean+remapped > 124 active+undersized+degraded > 55 active+clean+scrubbing > 45 active+clean+remapped+scrubbing > 10 active+clean+remapped+scrubbing+deep > 10 active+clean+scrubbing+deep > > io: > client: 50 KiB/s rd, 1.4 MiB/s wr, 8 op/s rd, 205 op/s wr > > > # ceph health detail > HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore; nodeep-scrub > flag(s) set; Degraded data redundancy: 6114283/137592537 objects degraded > (4.444%), 124 pgs degraded, 124 pgs undersized; 270 pgs not deep-scrubbed in > time; 1 subtrees have overcommitted pool target_size_bytes > [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in > BlueStore > osd.21 observed slow operation indications in BlueStore > [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set > [WRN] PG_DEGRADED: Degraded data redundancy: 6114283/137592537 objects > degraded (4.444%), 124 pgs degraded, 124 pgs undersized > pg 26.3 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [31,14,NONE,0,8] > pg 26.4 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [2,8,NONE,24,35] > pg 26.7 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [2,15,31,7,NONE] > pg 26.d is stuck undersized for 94m, current state > active+undersized+degraded, last acting [21,12,30,NONE,9] > pg 26.11 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [13,8,30,NONE,3] > pg 26.3c is stuck undersized for 94m, current state > active+undersized+degraded, last acting [0,7,35,5,NONE] > pg 26.3d is stuck undersized for 94m, current state > active+undersized+degraded, last acting [9,NONE,14,6,31] > pg 27.0 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [2,31,7,15,NONE] > pg 27.2 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [0,35,11,NONE,31] > pg 27.3 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [35,31,4,1,NONE] > pg 27.b is active+undersized+degraded, acting [2,8,NONE,5,12] > pg 27.c is stuck undersized for 94m, current state > active+undersized+degraded, last acting [2,8,12,31,NONE] > pg 27.10 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [23,8,14,NONE,0] > pg 27.14 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [30,16,9,NONE,6] > pg 27.17 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [23,4,14,NONE,1] > pg 27.1b is stuck undersized for 94m, current state > active+undersized+degraded, last acting [12,10,NONE,2,30] > pg 27.26 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [0,35,10,24,NONE] > pg 27.3c is stuck undersized for 94m, current state > active+undersized+degraded, last acting [32,9,2,13,NONE] > pg 33.1a is stuck undersized for 94m, current state > active+undersized+degraded, last acting [0,NONE,31,35,4] > pg 48.22 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [30,0,7,NONE,12] > pg 48.24 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [5,1,NONE,13,9] > pg 48.26 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [9,3,35,NONE,30] > pg 48.28 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [9,NONE,3,14,5] > pg 48.2b is stuck undersized for 94m, current state > active+undersized+degraded, last acting [15,4,5,NONE,0] > pg 48.2e is stuck undersized for 94m, current state > active+undersized+degraded, last acting [15,7,31,3,NONE] > pg 48.30 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [15,31,21,NONE,9] > pg 48.31 is stuck undersized for 9M, current state > active+undersized+degraded, last acting [4,1,13,30,NONE] > pg 48.32 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [4,0,5,13,NONE] > pg 48.41 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [8,32,NONE,0,16] > pg 48.43 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [30,0,12,7,NONE] > pg 48.4b is stuck undersized for 94m, current state > active+undersized+degraded, last acting [13,6,NONE,10,24] > pg 48.4f is stuck undersized for 94m, current state > active+undersized+degraded, last acting [30,15,2,7,NONE] > pg 48.50 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [2,9,NONE,13,24] > pg 48.53 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [4,30,14,3,NONE] > pg 48.54 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [9,6,14,NONE,30] > pg 48.56 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [35,9,5,1,NONE] > pg 48.59 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [30,14,7,NONE,3] > pg 48.5d is stuck undersized for 94m, current state > active+undersized+degraded, last acting [10,35,NONE,3,5] > pg 48.5f is stuck undersized for 94m, current state > active+undersized+degraded, last acting [5,13,8,0,NONE] > pg 48.62 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [4,2,32,35,NONE] > pg 48.70 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [3,10,5,15,NONE] > pg 48.72 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [1,31,11,15,NONE] > pg 48.75 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [3,24,NONE,9,35] > pg 48.7d is stuck undersized for 94m, current state > active+undersized+degraded, last acting [5,10,14,1,NONE] > pg 48.7e is stuck undersized for 94m, current state > active+undersized+degraded, last acting [32,NONE,15,9,0] > pg 48.82 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [24,13,3,9,NONE] > pg 48.86 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [23,9,12,2,NONE] > pg 48.8e is stuck undersized for 94m, current state > active+undersized+degraded, last acting [2,7,12,NONE,32] > pg 48.90 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [11,2,NONE,12,30] > pg 48.97 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [0,8,16,5,NONE] > pg 48.98 is stuck undersized for 94m, current state > active+undersized+degraded, last acting [23,9,12,2,NONE] > [WRN] PG_NOT_DEEP_SCRUBBED: 270 pgs not deep-scrubbed in time > pg 48.9b not deep-scrubbed since 2025-06-24T22:11:31.466631+0000 > pg 48.97 not deep-scrubbed since 2025-06-28T14:03:27.678577+0000 > pg 59.9d not deep-scrubbed since 2025-06-23T02:38:49.854500+0000 > pg 48.90 not deep-scrubbed since 2025-06-27T07:20:46.510366+0000 > pg 48.8e not deep-scrubbed since 2025-06-20T03:44:19.531953+0000 > pg 48.8c not deep-scrubbed since 2025-06-27T04:38:11.342009+0000 > pg 48.86 not deep-scrubbed since 2025-06-24T02:01:54.551428+0000 > pg 59.8e not deep-scrubbed since 2025-06-22T17:48:29.668256+0000 > pg 48.82 not deep-scrubbed since 2025-06-28T06:27:01.586778+0000 > pg 59.76 not deep-scrubbed since 2025-06-26T04:36:17.368921+0000 > pg 48.7d not deep-scrubbed since 2025-06-26T14:26:17.237509+0000 > pg 48.7c not deep-scrubbed since 2025-06-23T18:57:42.886560+0000 > pg 48.7b not deep-scrubbed since 2025-06-25T04:40:51.163596+0000 > pg 59.71 not deep-scrubbed since 2025-06-23T22:57:42.807731+0000 > pg 59.73 not deep-scrubbed since 2025-06-23T15:06:04.642059+0000 > pg 58.72 not deep-scrubbed since 2025-06-26T08:28:44.995587+0000 > pg 48.75 not deep-scrubbed since 2025-06-25T13:33:41.368882+0000 > pg 48.74 not deep-scrubbed since 2025-06-22T18:15:30.731614+0000 > pg 48.72 not deep-scrubbed since 2025-06-22T18:07:01.392496+0000 > pg 48.71 not deep-scrubbed since 2025-06-23T15:40:23.134466+0000 > pg 48.70 not deep-scrubbed since 2025-06-22T13:17:09.447156+0000 > pg 59.66 not deep-scrubbed since 2025-06-24T21:58:12.342232+0000 > pg 48.6d not deep-scrubbed since 2025-06-23T18:59:57.769702+0000 > pg 58.66 not deep-scrubbed since 2025-06-22T18:23:49.451971+0000 > pg 59.60 not deep-scrubbed since 2025-06-28T11:19:15.679226+0000 > pg 58.60 not deep-scrubbed since 2025-06-27T14:13:20.771028+0000 > pg 58.62 not deep-scrubbed since 2025-06-23T01:20:37.872422+0000 > pg 58.6f not deep-scrubbed since 2025-06-27T13:40:17.031152+0000 > pg 58.6b not deep-scrubbed since 2025-06-26T20:17:57.399397+0000 > pg 59.6b not deep-scrubbed since 2025-06-25T07:55:45.671580+0000 > pg 59.54 not deep-scrubbed since 2025-06-26T12:38:31.713680+0000 > pg 48.5f not deep-scrubbed since 2025-06-27T07:22:23.643527+0000 > pg 59.53 not deep-scrubbed since 2025-06-24T06:40:33.169587+0000 > pg 48.58 not deep-scrubbed since 2025-06-27T17:31:04.072180+0000 > pg 48.57 not deep-scrubbed since 2025-06-26T03:20:16.605485+0000 > pg 48.56 not deep-scrubbed since 2025-06-28T05:41:54.098155+0000 > pg 58.5f not deep-scrubbed since 2025-06-28T12:19:16.661402+0000 > pg 48.52 not deep-scrubbed since 2025-06-24T14:49:29.295937+0000 > pg 48.50 not deep-scrubbed since 2025-06-28T01:09:37.227387+0000 > pg 48.4f not deep-scrubbed since 2025-06-20T01:16:09.369233+0000 > pg 58.46 not deep-scrubbed since 2025-06-26T19:20:06.436318+0000 > pg 59.40 not deep-scrubbed since 2025-06-28T13:21:24.493082+0000 > pg 59.4d not deep-scrubbed since 2025-06-24T14:10:53.198832+0000 > pg 58.4c not deep-scrubbed since 2025-06-22T18:24:20.137028+0000 > pg 48.43 not deep-scrubbed since 2025-06-18T23:08:02.129787+0000 > pg 59.49 not deep-scrubbed since 2025-06-26T03:05:00.546786+0000 > pg 48.41 not deep-scrubbed since 2025-06-20T09:38:33.831563+0000 > pg 48.40 not deep-scrubbed since 2025-06-22T21:41:26.468848+0000 > pg 47.b not deep-scrubbed since 2025-06-28T11:03:41.301804+0000 > pg 46.a not deep-scrubbed since 2025-06-26T14:59:32.312819+0000 > 220 more pgs... > [WRN] POOL_TARGET_SIZE_BYTES_OVERCOMMITTED: 1 subtrees have overcommitted > pool target_size_bytes > > Regards > Dev > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
