Hi David,
Yes heath detail outputs all the errors etc and recovery / backfill is
going on, just taking time 25% misplaced and 1.5 degraded.
I can list out the pools and see sizes etc..
My main problem is I have no client IO from a read perspective, I cannot
start vms I'm openstack and ceph -w states that rd is zero most of the time.
I can restart the nodes and the osd reconnect and come in correctly..
I was chasing a missing object but was able to correct that by setting the
crush weight to 0.0 on the secondary.
Kind regards
Lee
On Sun, 2 Sep 2018, 13:43 David C, <dcsysengin...@gmail.com> wrote:
> Does "ceph health detail" work?
> Have you manually confirmed the OSDs on the nodes are working?
> What was the replica size of the pools?
> Are you seeing any progress with the recovery?
>
>
>
> On Sun, Sep 2, 2018 at 9:42 AM Lee <lqui...@gmail.com> wrote:
>
>> Running 0.94.5 as part of a Openstack enviroment, our ceph setup is 3x
>> OSD Nodes 3x MON Nodes, yesterday we had a aircon outage in our hosting
>> enviroment, 1 OSD node failed (offline with a the journal SSD dead) left
>> with 2 nodes running correctly, 2 hours later a second OSD node failed
>> complaining of readwrite errors to the physical drives, i assume this was a
>> heat issue as when rebooted this came back online ok and ceph started to
>> repair itself. We have since brought the first failed node back on by
>> replacing the ssd and recreating the journals hoping it would all repair..
>> Our pools are min 2 repl.
>>
>> The problem we have is client IO (read) is totally blocked, and when I
>> query the stuck PG's it just hangs..
>>
>> For example the check version command just errors with:
>>
>> Error EINTR: problem getting command descriptions from on various OSD's
>> so I cannot even query the inactive PG's
>>
>> root@node31-a4:~# ceph -s
>> cluster 7c24e1b9-24b3-4a1b-8889-9b2d7fd88cd2
>> health HEALTH_WARN
>> 83 pgs backfill
>> 2 pgs backfill_toofull
>> 3 pgs backfilling
>> 48 pgs degraded
>> 1 pgs down
>> 31 pgs incomplete
>> 1 pgs recovering
>> 29 pgs recovery_wait
>> 1 pgs stale
>> 48 pgs stuck degraded
>> 31 pgs stuck inactive
>> 1 pgs stuck stale
>> 148 pgs stuck unclean
>> 17 pgs stuck undersized
>> 17 pgs undersized
>> 599 requests are blocked > 32 sec
>> recovery 111489/4697618 objects degraded (2.373%)
>> recovery 772268/4697618 objects misplaced (16.440%)
>> recovery 1/2171314 unfound (0.000%)
>> monmap e5: 3 mons at {bc07s12-a7=
>> 172.27.16.11:6789/0,bc07s13-a7=172.27.16.21:6789/0,bc07s14-a7=172.27.16.15:6789/0
>> }
>> election epoch 198, quorum 0,1,2
>> bc07s12-a7,bc07s14-a7,bc07s13-a7
>> osdmap e18727: 25 osds: 25 up, 25 in; 90 remapped pgs
>> pgmap v70996322: 1792 pgs, 13 pools, 8210 GB data, 2120 kobjects
>> 16783 GB used, 6487 GB / 23270 GB avail
>> 111489/4697618 objects degraded (2.373%)
>> 772268/4697618 objects misplaced (16.440%)
>> 1/2171314 unfound (0.000%)
>> 1639 active+clean
>> 66 active+remapped+wait_backfill
>> 30 incomplete
>> 25 active+recovery_wait+degraded
>> 15 active+undersized+degraded+remapped+wait_backfill
>> 4 active+recovery_wait+degraded+remapped
>> 4 active+clean+scrubbing
>> 2 active+remapped+wait_backfill+backfill_toofull
>> 1 down+incomplete
>> 1 active+remapped+backfilling
>> 1 active+clean+scrubbing+deep
>> 1 stale+active+undersized+degraded
>> 1 active+undersized+degraded+remapped+backfilling
>> 1 active+degraded+remapped+backfilling
>> 1 active+recovering+degraded
>> recovery io 29385 kB/s, 7 objects/s
>> client io 5877 B/s wr, 1 op/s
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
