Hello, we have rebooted switch and then ceph cluster stopped working.
Ceph version: 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) Ceph status: HEALTH_WARN 256 pgs peering; 256 pgs stuck inactive; 256 pgs stuck unclean; 109 requests are blocked > 32 sec; 6 osds have slow requests; mds cluster is degraded Osds are not overloaded. I've discussed it on irc so i've attached log from there (which contains all relevant info). Do you think that there is any chance to make this cluster healthy? Thank you ****************************************************************************** This email and any files transmitted with it are confidential and may be subject to copyright. They are intended solely for the use of the individual or entity to which they are addressed. If you have received this message in error please notify AFS immediately by return email. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of AFS, except where an authorized sender specifically states them to be the views of AFS. It is your responsibility to verify this email and any attachments for the presence of viruses. AFS accepts no liability for any damage caused by any virus transmitted.
<omelkam>Hi, we have problem with ceph cluster after switch restart - every pg is in peering state, any idea? <omelkam>here is attached output of # ceph health detail http://pastebin.com/BbGmYepX <omelkam>and output of ceph pg 3.f query http://pastebin.com/MDQFg4A0 <omelkam>please help <andreask>omelkam: ceph osd tree? <omelkam>all osds are up <omelkam>andreask: output of ceph osd tree - http://pastebin.com/siwHtAxa <andreask>omelkam: and your ceph version? <omelkam>ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) <andreask>omelkam: and ceph -s ? <omelkam>andreask: here http://pastebin.com/VxBG5wRb <andreask>omelkam: and they don't continue in peering? ... so ceph -w is silent? <omelkam>andreask: dont know, here is output of ceph -w: http://pastebin.com/24QBbcQe <andreask>you are using cephfs? <omelkam>andreask: rbd and cephfs <omelkam>we have connected rbd pool to proxmox and stored disk images on it, and cephfs is used only to store iso images of installers <andreask>do the osds log erorrs? <omelkam>andreask: same as ceph-w (about slow requests) <andreask>omelkam: you see your osd servers being under high load? <omelkam>andreask: no, there are absolutely no iops on osds; cpu load is about 2% and memory load about 5% <andreask>omelkam: and the mds does log anything? <omelkam>andreask: here is log from mds http://pastebin.com/dH39G92j <andreask>omelkam: but all network problems are solved and you can see connections between the osd processes? <omelkam>andreask: ping is working, all osds are up, but i dont know how can i check if there are connections between osd processes <andreask>omelkam: you can see any connections with ports 68xx in netstat? <omelkam>andreask: yes i can <andreask>omelkam: do you have pulic and cluster network? <omelkam>andreask: also, yes <andreask>any change only one of the networks is working? <andreask>esspecially the cluster-network is _not_ working <omelkam>andreask: ping on both networks is working <andreask>and the connections you see are on both networks? <omelkam>andreask: here is output of netstat http://pastebin.com/Hy5ZZB0q - 192.168.10.0/24 is public network, 192.168.11.0/24 is cluster network <omelkam>andreask: i have compared md5 hash of one pg (on filesystem) from both osds and it is same, so i dont know why that pg is in peering state <andreask>yeah, that is strange .... have you tried stopping all osds and mds and only start two osds on different servers? <omelkam>andreask: nope, i can try it <omelkam>andreask: now it looks like this: 225 pgs down; 256 pgs peering; 117 pgs stale; 256 pgs stuck inactive; 256 pgs stuck unclean; 119 requests are blocked > 32 sec; mds cluster is degraded; mds SKS13038-ceph is laggy; 4/6 in osds are down <andreask>omelkam: and it does not change looking at ceph -w? <omelkam>andreask: it only adds info about laggy or crashed mds <andreask>hmm ... so no difference, presumable also if you start the other osds <omelkam>andreask: do you think that when we mark three osds from one machine as lost, it can help? <andreask>is this a productive system? <omelkam>andreask: yes <andreask>I'd start with increasing ods debug log level, there might be more information in there <omelkam>andreask: we have tried this, but i do not see any interesting info in this log - http://pastebin.com/Mr637AEi <andreask>not really ... do you also have the debug log from starting an osd? <omelkam>andreask: ok, i have created log file, but it has 27 MB, so i have uploaded it to google drive: https://drive.google.com/file/d/0B5KOgo4ZJ0kVdXFXd1lVTndBeUU/edit?usp=sharing <andreask>omelkam: the log does not look suspicious to me, seems like osd is done with peering <omelkam>andreask: any chance that we can restore data from cluster with this state? <andreask>hmm ... try to start osds only on one server <omelkam>andreask: i have tried it before, same state <andreask>can you show "ceph osd dump" please <omelkam>andreask: here http://pastebin.com/szGNgUDQ <andreask>omelkam: also if you take the stopped osds out of data distribution? <omelkam>andreask: how can i do this? <andreask>with "ceph osd out _osd_" <proksaj>andreask: hello omelkam had to go out for a while <proksaj>I have set osds as OUT <proksaj>andreask: still all pgs are in peering state <proksaj>andreask: like this HEALTH_WARN 256 pgs peering; 256 pgs stuck inactive; 256 pgs stuck unclean; <proksaj>pg 3.f is stuck inactive for 23516.037183, current state peering, last acting [1] <andreask>hmm ... and a ceph pg _pg_ query of this pg now? <proksaj>andreask: http://pastebin.com/FWzWmUVW <andreask>proksaj: no idea, sorry ... having all osds started should do the trick <andreask>but if this stucks again, I'd wait for a dev here in the channel or post to the mailinglist <proksaj>andreask: sadly it does not <andreask>proksaj: have you already restarted the mons? <proksaj>andreask: yes <omelkam>andreask: thank you for your time, i will create post to mailing list with that
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
