Hello all
I have a cluster where my cluster is in hung state, Some back fills are there
but I reduced it to 1 but still upgrade not progressing…
Please help…
```# ceph health detail
HEALTH_WARN 8 OSD(s) experiencing slow operations in BlueStore; Failed to apply
2 service(s): osd.all-available-devices,osd.iops_optimized; 1 failed cephadm
daemon(s); failed to probe daemons or devices; noscrub,nodeep-scrub flag(s)
set; Degraded data redundancy: 1600150/39365198 objects degraded (4.065%), 93
pgs degraded, 103 pgs undersized; 127 pgs not deep-scrubbed in time
[WRN] BLUESTORE_SLOW_OP_ALERT: 8 OSD(s) experiencing slow operations in
BlueStore
osd.5 observed slow operation indications in BlueStore
osd.9 observed slow operation indications in BlueStore
osd.18 observed slow operation indications in BlueStore
osd.36 observed slow operation indications in BlueStore
osd.59 observed slow operation indications in BlueStore
osd.66 observed slow operation indications in BlueStore
osd.106 observed slow operation indications in BlueStore
osd.110 observed slow operation indications in BlueStore
[WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 2 service(s):
osd.all-available-devices,osd.iops_optimized
osd.all-available-devices: Command timed out on host cephadm deploy (osd
daemon) (default 900 second timeout)
osd.iops_optimized: Command timed out on host cephadm deploy (osd daemon)
(default 900 second timeout)
[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)
daemon osd.19 on phl-prod-host04n.example.comis in error state
[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
Command "cephadm ceph-volume -- inventory" timed out on host
phl-prod-converged03n.phl.netskrt.org (default 900 second timeout)
[WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set
[WRN] PG_DEGRADED: Degraded data redundancy: 1600150/39365198 objects degraded
(4.065%), 93 pgs degraded, 103 pgs undersized
pg 25.0 is stuck undersized for 21h, current state
active+undersized+degraded+remapped+backfill_wait, last acting [90,77,NONE,38,3]
pg 25.1 is stuck undersized for 23h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[131,1,66,NONE,72]
pg 25.c is stuck undersized for 23h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[1,135,110,28,NONE]
pg 25.e is active+undersized+degraded+remapped+backfill_wait, acting
[18,20,108,101,NONE]
pg 25.14 is stuck undersized for 19h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[32,13,NONE,65,97]
pg 25.17 is stuck undersized for 19h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[32,65,110,23,NONE]
pg 25.1a is stuck undersized for 23h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[97,66,NONE,112,139]
pg 25.1c is stuck undersized for 2h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[102,136,97,NONE,45]
pg 25.1f is stuck undersized for 8h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[NONE,66,7,39,109]
pg 26.45 is stuck undersized for 23h, current state
active+undersized+degraded+remapped+backfill_wait, last acting [32,47]
pg 26.48 is stuck undersized for 23h, current state
active+undersized+degraded+remapped+backfill_wait, last acting [113,55]
pg 26.4b is stuck undersized for 8h, current state
active+undersized+degraded+remapped+backfill_wait, last acting [105,7]
pg 26.59 is stuck undersized for 23h, current state
active+undersized+degraded+remapped+backfill_wait, last acting [35,40]
pg 26.6a is stuck undersized for 7h, current state
active+undersized+degraded+remapped+backfill_wait, last acting [25,45]
pg 26.74 is stuck undersized for 23h, current state
active+undersized+degraded+remapped+backfill_wait, last acting [112,131]
pg 26.77 is stuck undersized for 2h, current state
active+undersized+degraded+remapped+backfill_wait, last acting [17,105]
pg 26.9c is stuck undersized for 19h, current state
active+undersized+degraded+remapped+backfill_wait, last acting [47,76]
pg 26.bf is stuck undersized for 4m, current state
active+undersized+degraded+remapped+backfill_wait, last acting [32,26]
pg 26.c2 is stuck undersized for 2h, current state
active+undersized+degraded+remapped+backfill_wait, last acting [97,135]
pg 26.ec is stuck undersized for 4m, current state
active+undersized+degraded+remapped+backfill_wait, last acting [26,80]
pg 26.f7 is stuck undersized for 4m, current state
active+undersized+degraded+remapped+backfill_wait, last acting [49,105]
pg 31.12 is stuck undersized for 4m, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[47,110,131,37,NONE]
pg 31.19 is stuck undersized for 2h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[NONE,87,3,137,39]
pg 31.1b is stuck undersized for 2h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[108,NONE,42,102,97]
pg 31.1c is stuck undersized for 19h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[113,66,NONE,45,72]
pg 31.41 is stuck undersized for 2h, current state
active+undersized+remapped+backfill_wait, last acting [14,101,NONE,67,18]
pg 31.42 is stuck undersized for 7h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[37,134,NONE,82,8]
pg 31.44 is stuck undersized for 4m, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[49,NONE,101,3,62]
pg 31.46 is stuck undersized for 4m, current state
active+undersized+degraded+remapped+backfilling, last acting [95,38,NONE,25,102]
pg 31.47 is stuck undersized for 18h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[60,NONE,130,72,110]
pg 31.4a is stuck undersized for 4h, current state
active+undersized+degraded+remapped+backfilling, last acting [66,NONE,135,82,14]
pg 31.4c is stuck undersized for 4m, current state
active+undersized+degraded+remapped+backfill_wait, last acting [34,NONE,1,82,18]
pg 31.4d is stuck undersized for 2h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[101,13,65,30,NONE]
pg 31.52 is stuck undersized for 4m, current state
active+undersized+remapped+backfill_wait, last acting [26,112,66,NONE,135]
pg 31.53 is stuck undersized for 17h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[NONE,24,11,42,110]
pg 31.55 is stuck undersized for 4m, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[116,NONE,27,4,117]
pg 31.57 is stuck undersized for 4m, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[17,15,110,NONE,1]
pg 31.5a is stuck undersized for 19h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[13,24,124,26,NONE]
pg 31.5b is stuck undersized for 4m, current state
active+undersized+degraded+remapped+backfill_wait, last acting [NONE,31,8,27,42]
pg 31.5c is stuck undersized for 2h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[NONE,60,28,18,119]
pg 31.5d is stuck undersized for 17h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[124,NONE,85,6,11]
pg 31.5e is stuck undersized for 2h, current state
active+undersized+degraded+remapped+backfill_wait, last acting [NONE,4,1,23,138]
pg 31.60 is stuck undersized for 18h, current state
active+undersized+degraded+remapped+backfilling, last acting
[37,11,102,NONE,133]
pg 31.64 is stuck undersized for 4m, current state
active+undersized+remapped+backfill_wait, last acting [26,106,45,34,NONE]
pg 31.65 is stuck undersized for 4h, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[NONE,14,127,62,3]
pg 31.66 is stuck undersized for 4m, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[34,NONE,35,23,59]
pg 31.67 is stuck undersized for 4m, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[67,24,127,8,NONE]
pg 31.68 is stuck undersized for 4m, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[32,41,18,17,NONE]
pg 31.70 is stuck undersized for 19h, current state
active+undersized+degraded+remapped+backfilling, last acting [106,38,1,NONE,97]
pg 31.75 is stuck undersized for 2h, current state
active+undersized+degraded+remapped+backfill_wait, last acting [26,31,34,8,NONE]
pg 31.7f is stuck undersized for 2h, current state
active+undersized+remapped+backfilling, last acting [NONE,101,110,10,45]
[WRN] PG_NOT_DEEP_SCRUBBED: 127 pgs not deep-scrubbed in time
pg 26.e5 not deep-scrubbed since 2025-06-07T05:59:22.612553+0000
pg 26.ca not deep-scrubbed since 2025-06-06T18:21:07.435823+0000
pg 26.bf not deep-scrubbed since 2025-06-03T07:29:02.791095+0000
pg 26.be not deep-scrubbed since 2025-06-04T07:55:54.389714+0000
pg 26.a5 not deep-scrubbed since 2025-06-07T07:28:56.878096+0000
pg 26.85 not deep-scrubbed since 2025-06-03T15:18:59.395929+0000
pg 26.7f not deep-scrubbed since 2025-06-04T21:29:28.412637+0000
pg 26.7e not deep-scrubbed since 2025-06-03T09:14:19.585388+0000
pg 26.7d not deep-scrubbed since 2025-06-03T18:37:30.931020+0000
pg 26.7c not deep-scrubbed since 2025-06-04T13:38:00.061488+0000
pg 26.73 not deep-scrubbed since 2025-06-03T06:20:15.111819+0000
pg 26.6f not deep-scrubbed since 2025-06-03T13:45:24.880397+0000
pg 26.6e not deep-scrubbed since 2025-05-26T23:15:32.099862+0000
pg 26.6d not deep-scrubbed since 2025-06-04T14:04:10.449101+0000
pg 31.62 not deep-scrubbed since 2025-06-03T13:34:49.518456+0000
pg 26.65 not deep-scrubbed since 2025-06-04T07:56:25.353411+0000
pg 31.66 not deep-scrubbed since 2025-06-03T10:32:05.364424+0000
pg 26.62 not deep-scrubbed since 2025-06-04T09:35:58.267976+0000
pg 31.65 not deep-scrubbed since 2025-06-03T16:04:40.003140+0000
pg 31.5b not deep-scrubbed since 2025-06-03T14:18:18.835477+0000
pg 26.5d not deep-scrubbed since 2025-06-04T15:14:30.870252+0000
pg 31.58 not deep-scrubbed since 2025-06-03T03:09:27.568605+0000
pg 26.5c not deep-scrubbed since 2025-06-03T01:57:27.644129+0000
pg 31.5f not deep-scrubbed since 2025-06-03T05:53:20.860393+0000
pg 31.52 not deep-scrubbed since 2025-05-27T00:01:27.040861+0000
pg 31.53 not deep-scrubbed since 2025-05-24T09:37:58.964829+0000
pg 26.55 not deep-scrubbed since 2025-06-04T21:25:34.135356+0000
pg 26.54 not deep-scrubbed since 2025-06-04T06:07:12.978734+0000
pg 31.56 not deep-scrubbed since 2025-06-04T12:58:17.599712+0000
pg 31.57 not deep-scrubbed since 2025-06-03T07:02:16.859990+0000
pg 26.51 not deep-scrubbed since 2025-06-03T05:42:22.435483+0000
pg 26.4f not deep-scrubbed since 2025-06-03T09:10:22.617328+0000
pg 31.4a not deep-scrubbed since 2025-05-28T00:54:55.246532+0000
pg 26.4e not deep-scrubbed since 2025-06-03T11:16:49.278513+0000
pg 31.4b not deep-scrubbed since 2025-06-03T10:24:24.123351+0000
pg 26.4d not deep-scrubbed since 2025-06-04T19:01:44.614410+0000
pg 31.49 not deep-scrubbed since 2025-05-28T04:56:29.368285+0000
pg 31.42 not deep-scrubbed since 2025-05-28T08:38:57.151865+0000
pg 26.41 not deep-scrubbed since 2025-06-03T05:47:35.443867+0000
pg 26.40 not deep-scrubbed since 2025-06-03T05:43:13.283668+0000
pg 25.17 not deep-scrubbed since 2025-06-03T09:26:52.253625+0000
pg 32.2e not deep-scrubbed since 2025-06-05T13:01:06.175389+0000
pg 22.1a not deep-scrubbed since 2025-06-07T01:40:45.063268+0000
pg 31.13 not deep-scrubbed since 2025-06-03T12:21:17.965218+0000
pg 22.1b not deep-scrubbed since 2025-06-04T12:22:44.947751+0000
pg 25.14 not deep-scrubbed since 2025-05-28T06:26:32.552200+0000
pg 26.17 not deep-scrubbed since 2025-06-03T18:37:26.617483+0000
pg 31.12 not deep-scrubbed since 2025-05-28T08:39:23.271194+0000
pg 31.1c not deep-scrubbed since 2025-06-03T08:17:51.230187+0000
pg 25.1f not deep-scrubbed since 2025-05-28T00:19:23.653883+0000
77 more pgs…
Regards
Dev
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
