Hi,
I'm sure I'm doing something wrong, I hope someone can enlighten me...
I'm encountering many issues when I restart a ceph server (any ceph server).
This is on CentOS 7.2, ceph-0.94.6-0.el7.x86_64.
Firt : I have disabled abrt. I don't need abrt.
But when I restart, I see these logs in the systemd-udevd journal :
Apr 21 18:00:14 ceph4._snip_ python[1109]: detected unhandled Python exception
in '/usr/sbin/ceph-disk'
Apr 21 18:00:14 ceph4._snip_ python[1109]: can't communicate with ABRT daemon,
is it running? [Errno 2] No such file or directory
Apr 21 18:00:14 ceph4._snip_ python[1174]: detected unhandled Python exception
in '/usr/sbin/ceph-disk'
Apr 21 18:00:14 ceph4._snip_ python[1174]: can't communicate with ABRT daemon,
is it running? [Errno 2] No such file or directory
How could I possibly debug these exceptions ?
Could that be related to the osd hook that I'm using to put the SSDs in another
root in the crush map (that hook is a bash script, but it's calling another
helper python script that I made and which is trying to use megacli to identify
the SSDs on a non-jbod controller... tricky thing.) ?
Then, I see these kind of errors for most if not all drives :
Apr 21 18:00:47 ceph4._snip_ systemd-udevd[876]: '/usr/sbin/ceph-disk-activate
/dev/sdt1'(err) '2016-04-21 18:00:47.115322 7fc408ff9700 0 -- :/885104093 >>
__MON_IP__:6789/0 pipe(0x7fc400008280 sd=6 :0 s=1 pgs=0 cs=0 l=1
c=0x7fc400012670).fault'
Apr 21 18:00:50 ceph4._snip_ systemd-udevd[876]: '/usr/sbin/ceph-disk-activate
/dev/sdt1'(err) '2016-04-21 18:00:50.115543 7fc408ef8700 0 -- :/885104093 >>
__MON_IP__:6789/0 pipe(0x7fc400000c00 sd=6 :0 s=1 pgs=0 cs=0 l=1
c=0x7fc40000e1d0).fault'
Apr 21 18:00:52 ceph4._snip_ systemd-udevd[876]: '/usr/sbin/ceph-disk-activate
/dev/sdt1'(out) 'failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf
--name=osd.113 --keyring=/var/lib/ceph/osd/ceph-113/keyring osd crush
create-or-move -- 113 1.81 host=ceph4 root=default''
Apr 21 18:00:52 ceph4._snip_ systemd-udevd[876]: '/usr/sbin/ceph-disk-activate
/dev/sdt1'(err) 'ceph-disk: Error: ceph osd start failed: Command
'['/usr/sbin/service', 'ceph', '--cluster', 'ceph', 'start', 'osd.113']'
returned non-zero exit status 1'
Apr 21 18:00:52 ceph4._snip_ systemd-udevd[876]: '/usr/sbin/ceph-disk-activate
/dev/sdt1' [1257] exit with return code 1
Apr 21 18:00:52 ceph4._snip_ systemd-udevd[876]: adding watch on '/dev/sdt1'
Apr 21 18:00:52 ceph4._snip_ systemd-udevd[876]: created db file
'/run/udev/data/b65:49' for
'/devices/pci0000:00/0000:00:07.0/0000:03:00.0/host2/target2:2:6/2:2:6:0/block/sdt/sdt1'
Apr 21 18:00:52 ceph4._snip_ systemd-udevd[876]: passed unknown number of bytes
to netlink monitor 0x7f4cec2f3240
Apr 21 18:00:52 ceph4._snip_ systemd-udevd[876]: seq 2553 processed with 0
Please note that at that time of the boot, I think there is still no network as
the interfaces are brought up later according to the network journal :
Apr 21 18:02:16 ceph4._snip_ network[2904]: Bringing up interface p2p1: [ OK
]
Apr 21 18:02:19 ceph4._snip_ network[2904]: Bringing up interface p2p2: [ OK
]
=> too bad for the OSD startups... I have to say I also disabled
NetworkManager, and I'm using static network configuration files... but I don't
know why the ceph init script would be called before network is up... ?
But even if I had network, I'm having another issue : I'm wondering wether I'm
hitting deadlocks somewhere...
Apr 21 18:01:10 ceph4._snip_ systemd-udevd[779]: worker [792]
/devices/pci0000:00/0000:00:07.0/0000:03:00.0/host2/target2:2:0/2:2:0:0/block/sdn/sdn2
is taking a long time
(...)
Apr 21 18:01:54 ceph4._snip_ systemd-udevd[792]: '/usr/sbin/ceph-disk
activate-journal /dev/sdn2'(err) 'SG_IO: bad/missing sense data, sb[]: 70 00
05 00'
Apr 21 18:01:54 ceph4._snip_ systemd-udevd[792]: '/usr/sbin/ceph-disk
activate-journal /dev/sdn2'(err) ' 00 00 00 0b 00 00 00 00 20 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00'
Apr 21 18:01:54 ceph4._snip_ systemd-udevd[792]: '/usr/sbin/ceph-disk
activate-journal /dev/sdn2'(err) ''
Apr 21 18:01:54 ceph4._snip_ systemd-udevd[792]: '/usr/sbin/ceph-disk
activate-journal /dev/sdn2'(out) '=== osd.107 === '
Apr 21 18:01:54 ceph4._snip_ systemd-udevd[792]: '/usr/sbin/ceph-disk
activate-journal /dev/sdn2'(err) '2016-04-21 18:01:54.707669 7f95801ac700 0 --
:/2141879112 >> __MON_IP__:6789/0 pipe(0x7f957c05f710 sd=4 :0 s=1 pgs=0 cs=0
l=1 c=0x7f957c05bb40).fault'
(...)
Apr 21 18:02:12 ceph4._snip_ systemd-udevd[792]: '/usr/sbin/ceph-disk
activate-journal /dev/sdn2'(err) '2016-04-21 18:02:12.709053 7f95801ac700 0 --
:/2141879112 >> __MON_IP__:6789/0 pipe(0x7f9570008280 sd=4 :0 s=1 pgs=0 cs=0
l=1 c=0x7f95700056a0).fault'
Apr 21 18:02:16 ceph4._snip_ systemd-udevd[792]: '/usr/sbin/ceph-disk
activate-journal /dev/sdn2'(err) 'create-or-move updated item name 'osd.107'
weight 1.81 at location {host=ceph4,root=default} to crush map'
Apr 21 18:02:16 ceph4._snip_ systemd-udevd[792]: '/usr/sbin/ceph-disk
activate-journal /dev/sdn2'(out) 'Starting Ceph osd.107 on ceph4...'
Apr 21 18:02:16 ceph4._snip_ systemd-udevd[792]: '/usr/sbin/ceph-disk
activate-journal /dev/sdn2'(err) 'Running as unit
ceph-osd.107.1461254514.449704730.service.'
Apr 21 18:02:16 ceph4._snip_ systemd-udevd[792]: '/usr/sbin/ceph-disk
activate-journal /dev/sdn2' [1138] exit with return code 0
Apr 21 18:02:16 ceph4._snip_ systemd-udevd[792]: adding watch on '/dev/sdn2'
Apr 21 18:02:16 ceph4._snip_ systemd-udevd[792]: created db file
'/run/udev/data/b8:210' for
'/devices/pci0000:00/0000:00:07.0/0000:03:00.0/host2/target2:2:0/2:2:0:0/block/sdn/sdn2'
If I look at this specific osd journal, I see
Apr 21 18:02:16 ceph4._snip_ systemd[3137]: Executing: /bin/bash -c 'ulimit -n
32768; /usr/bin/ceph-osd -i 107 --pid-file /var/run/ceph/osd.107.pid -c
/etc/ceph/ceph.conf --cluster ceph -f'
Apr 21 18:02:16 ceph4._snip_ bash[3137]: 2016-04-21 18:02:16.147602
7f109f916880 -1 ** ERROR: unable to open OSD superblock on
/var/lib/ceph/osd/ceph-107: (2) No such file or directory
Apr 21 18:02:16 ceph4._snip_ systemd[1]: Child 3137 belongs to
ceph-osd.107.1461254514.449704730.service
I'm assuming this just means that the partition was not mounted correctly
because ceph-disk failed, and that the ceph OSD daemon died...?
After the boot... no OSD is up.
And if I run ceph-disk activate-all manually after the node booted (and gives
me ssh access, which indeed takes long)... everything gets up.
Any idea(s) ?
Thanks
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
