OK, after couple of days I can sum up cause of my downed OSDs.
First, may be minor, cause was inconsistency of the ownership in
/var/lib/ceph - recursive chown ceph:ceph fix this.
Major problem was in ceph.conf. According to the Gentoo Linux guide I
used incorrect naming for mon server by indexes in the config file. The
correct version uses host names:
[global]
fsid = ...
ms bind ipv6 = true
public network = xxxx:xxxx:2:1612::/64
mon initial members = node1
mon host = [xxxx:xxxx:2:1612::50]:6789
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
...
[mon.node1]
host = node1
mon addr = [xxxx:xxxx:2:1612::50]:6789
Other side is Gentoo broken init script.
I thank all of you for help.
MS:
Dne 30.12.2015 v 00:46 Ing. Martin Samek napsal(a):
I made small leap ahead.
Not it seem to be some problem with autentication may be.
2015-12-30 00:43:34.041121 7f72773fd700 0 cephx server osd.0: couldn't find
entity name: osd.0
2015-12-30 00:44:12.835810 7f7277bfe700 0 mon.ares1@0(leader).data_health(1)
update_stats avail 53% total 28662 MB, used 13414 MB, avail 15247 MB
2015-12-30 00:43:34.040272 7fd71445d700 1 -- [xxxx:xxxx:2:1612::60]:6800/23279
--> [xxxx:xxxx:2:1612::50]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0
0x7fd6f9818080 con 0x7fd7239a3680
2015-12-30 00:43:34.040986 7fd71445d700 1 -- [xxxx:xxxx:2:1612::60]:6800/23279
<== mon.0 [xxxx:xxxx:2:1612::50]:6789/0 3 ==== auth_reply(proto 2 -1 (1)
Operation not permitted) v1 ==== 24+0+0 (1161288937 0 0) 0x7fd6f9818580 con
0x7fd7239a3680
2015-12-30 00:43:34.041265 7fd726e9a800 1 -- [xxxx:xxxx:2:1612::60]:6800/23279
mark_down 0x7fd7239a3680 -- 0x7fd7239b7000
2015-12-30 00:43:34.041977 7fd726e9a800 1 journal close
/var/lib/ceph/osd/ceph-0/journal
2015-12-30 00:43:34.043931 7fd726e9a800 -1 ** ERROR: osd init failed: (1)
Operation not permitted
MS:
Dne 30.12.2015 v 00:31 Jan Schermer napsal(a):
Just try putting something like the following in ceph.conf:
[global]
mon_host =xxxx:xxxx:2:1612::50 xxxx:xxxx:2:1612::30
mon_initial_members = node-1 node-2
Also, I just noticed you have two MONs? It should always be an odd
number. Not sure if they can ever get quorum now?
Jan
On 30 Dec 2015, at 00:15, Ing. Martin Samek <samek...@fel.cvut.cz
<mailto:samek...@fel.cvut.cz>> wrote:
I'm deploying ceph cluster manually following different guides. I
didn't use ceph-deploy yet.
MS:
Dne 30.12.2015 v 00:13 Somnath Roy napsal(a):
It should be monitor host names..If you are deploying with
ceph-deploy it should be added in the conf file automatically..How
are you creating your cluster ?
Did you change conf file after installing ?
*From:*Ing. Martin Samek [mailto:samek...@fel.cvut.cz]
*Sent:*Tuesday, December 29, 2015 3:09 PM
*To:*Jan Schermer
*Cc:*Somnath Roy;ceph-users@lists.ceph.com
*Subject:*Re: [ceph-users] My OSDs are down and not coming UP
Hi,
No, never. It is my first attempt, first ceph cluster i try ever run.
im not sure, if "mon initial members" should contain mon servers
ids or hostnames ?
MS:
Dne 30.12.2015 v 00:04 Jan Schermer napsal(a):
Has the cluster ever worked?
Are you sure that "mon initial members = 0" is correct? How do
the OSDs know where to look for MONs?
Jan
On 29 Dec 2015, at 21:41, Ing. Martin Samek
<samek...@fel.cvut.cz> wrote:
Hi,
network is OK, all nodes are in one VLAN, in one switch, in
one rack.
tracepath6 node2
1?: [LOCALHOST] 0.030ms pmtu 1500
1: node2 0.634ms
reached
1: node2 0.296ms
reached
Resume: pmtu 1500 hops 1 back 64
tracepath6 node3
1?: [LOCALHOST] 0.022ms pmtu 1500
1: node3 0.643ms
reached
1: node3 1.065ms
reached
Resume: pmtu 1500 hops 1 back 64
There is no firewall installed or configured.
Martin
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
====================================
Ing. Martin Samek
ICT systems engineer
FELK Admin
Czech Technical University in Prague
Faculty of Electrical Engineering
Department of Control Engineering
Karlovo namesti 13/E, 121 35 Prague
Czech Republic
e-mail: samek...@fel.cvut.cz
phone: +420 22435 7599
mobile: +420 605 285 125
====================================
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
