Re: [ceph-users] Upgrade to 0.80.7-0.el6 from 0.80.1-0.el6, OSD crashes on startup

2014-11-13 Thread Joshua McClintock
AH! Sorry for the false alarm, I clearly have a hard drive problem ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 ata2.00: BMDMA stat 0x24 ata2.00: failed command: READ DMA ata2.00: cmd c8/00:08:38:bd:70/00:00:00:00:00/ef tag 0 dma 4096 in res 51/40:00:3f:bd:70/40:00:2

Re: [ceph-users] Upgrade to 0.80.7-0.el6 from 0.80.1-0.el6, OSD crashes on startup

2014-11-13 Thread Sage Weil
Hmm, looks like leveldb is hitting a problem. Is there anything in the kernel log (dmesg) that suggests a disk or file system problem? Are you able to, say, tar up the current/omap directory without problems? This is a single OSD, right? None of the others have been upgraded yet? sage On T

Re: [ceph-users] Upgrade to 0.80.7-0.el6 from 0.80.1-0.el6, OSD crashes on startup

2014-11-13 Thread Joshua McClintock
[root@ceph-node20 ~]# ls /var/lib/ceph/osd/us-west01-0/current 0.10_head 0.1a_head 0.23_head 0.2c_head 0.37_head 0.3_head 0.b_head 1.10_head 1.1b_head 1.24_head 1.2c_head 1.3a_head 1.b_head 2.16_head 2.1_head 2.2a_head 2.32_head 2.3a_head 2.a_head omap 0.11_head 0.1d_head

Re: [ceph-users] Upgrade to 0.80.7-0.el6 from 0.80.1-0.el6, OSD crashes on startup

2014-11-13 Thread Sage Weil
On Thu, 13 Nov 2014, Joshua McClintock wrote: > I upgraded my mons to the latest version and they appear to work, I then > upgraded my mds and it seems fine.   > I then upgraded one OSD node and the OSD fails to start with the following > dump, any help is appreciated: > > --- begin dump of recent

[ceph-users] Upgrade to 0.80.7-0.el6 from 0.80.1-0.el6, OSD crashes on startup

2014-11-13 Thread Joshua McClintock
I upgraded my mons to the latest version and they appear to work, I then upgraded my mds and it seems fine. I then upgraded one OSD node and the OSD fails to start with the following dump, any help is appreciated: --- begin dump of recent events --- 0> 2014-11-13 18:20:15.625793 7fbd973ce7a