Running 0.94.5 as part of a Openstack enviroment, our ceph setup is 3x OSD
Nodes 3x MON Nodes, yesterday we had a aircon outage in our hosting
enviroment, 1 OSD node failed (offline with a the journal SSD dead) left
with 2 nodes running correctly, 2 hours later a second OSD node failed
complaining of readwrite errors to the physical drives, i assume this was a
heat issue as when rebooted this came back online ok and ceph started to
repair itself. We have since brought the first failed node back on by
replacing the ssd and recreating the journals hoping it would all repair..
Our pools are min 2 repl.
The problem we have is client IO (read) is totally blocked, and when I
query the stuck PG's it just hangs..
For example the check version command just errors with:
Error EINTR: problem getting command descriptions from on various OSD's so
I cannot even query the inactive PG's
root@node31-a4:~# ceph -s
cluster 7c24e1b9-24b3-4a1b-8889-9b2d7fd88cd2
health HEALTH_WARN
83 pgs backfill
2 pgs backfill_toofull
3 pgs backfilling
48 pgs degraded
1 pgs down
31 pgs incomplete
1 pgs recovering
29 pgs recovery_wait
1 pgs stale
48 pgs stuck degraded
31 pgs stuck inactive
1 pgs stuck stale
148 pgs stuck unclean
17 pgs stuck undersized
17 pgs undersized
599 requests are blocked > 32 sec
recovery 111489/4697618 objects degraded (2.373%)
recovery 772268/4697618 objects misplaced (16.440%)
recovery 1/2171314 unfound (0.000%)
monmap e5: 3 mons at {bc07s12-a7=
172.27.16.11:6789/0,bc07s13-a7=172.27.16.21:6789/0,bc07s14-a7=172.27.16.15:6789/0
}
election epoch 198, quorum 0,1,2
bc07s12-a7,bc07s14-a7,bc07s13-a7
osdmap e18727: 25 osds: 25 up, 25 in; 90 remapped pgs
pgmap v70996322: 1792 pgs, 13 pools, 8210 GB data, 2120 kobjects
16783 GB used, 6487 GB / 23270 GB avail
111489/4697618 objects degraded (2.373%)
772268/4697618 objects misplaced (16.440%)
1/2171314 unfound (0.000%)
1639 active+clean
66 active+remapped+wait_backfill
30 incomplete
25 active+recovery_wait+degraded
15 active+undersized+degraded+remapped+wait_backfill
4 active+recovery_wait+degraded+remapped
4 active+clean+scrubbing
2 active+remapped+wait_backfill+backfill_toofull
1 down+incomplete
1 active+remapped+backfilling
1 active+clean+scrubbing+deep
1 stale+active+undersized+degraded
1 active+undersized+degraded+remapped+backfilling
1 active+degraded+remapped+backfilling
1 active+recovering+degraded
recovery io 29385 kB/s, 7 objects/s
client io 5877 B/s wr, 1 op/s
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
