On Thu, 13 Nov 2014, Joshua McClintock wrote:
> I upgraded my mons to the latest version and they appear to work, I then
> upgraded my mds and it seems fine.
> I then upgraded one OSD node and the OSD fails to start with the following
> dump, any help is appreciated:
>
> --- begin dump of recent events ---
>
> 0> 2014-11-13 18:20:15.625793 7fbd973ce7a0 -1 *** Caught signal
> (Aborted) **
>
> in thread 7fbd973ce7a0
>
>
> ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)
> 1: /usr/bin/ceph-osd() [0x9bd2a1]
> 2: (()+0xf710) [0x7fbd96373710]
> 3: (gsignal()+0x35) [0x7fbd95245925]
> 4: (abort()+0x175) [0x7fbd95247105]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7fbd95affa5d]
> 6: (()+0xbcbe6) [0x7fbd95afdbe6]
> 7: (()+0xbcc13) [0x7fbd95afdc13]
> 8: (()+0xbcd0e) [0x7fbd95afdd0e]
> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x7f2) [0xafbe22]
> 10: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&,
> ceph::buffer::list*)+0x4ea) [0x7f729a]
> 11: (OSD::load_pgs()+0x18f1) [0x64f2b1]
> 12: (OSD::init()+0x22c0) [0x6536f0]
> 13: (main()+0x35bc) [0x5fe39c]
> 14: (__libc_start_main()+0xfd) [0x7fbd95231d1d]
> 15: /usr/bin/ceph-osd() [0x5f9e49]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
> interpret this.
Hey, this looks like a different report we saw recently off-list! In that
case, they were upgrading from 0.80.4 to 0.80.7. Opening #10105.
You only restarting a single OSD? I would hold off on restarting any more
for the time being.
Can you attach the output from
ls /var/lib/ceph/osd/ceph-NNN/current
and
ceph-kvstore-tool /var/lib/ceph/osd/ceph-NNN/current/omap list
(you may need to install the ceph-tests rpm to get ceph-kvstore-tool).
Thanks!
sage
>
>
> --- logging levels ---
>
> 0/ 5 none
>
> 0/ 1 lockdep
>
> 0/ 1 context
>
> 1/ 1 crush
>
> 1/ 5 mds
>
> 1/ 5 mds_balancer
>
> 1/ 5 mds_locker
>
> 1/ 5 mds_log
>
> 1/ 5 mds_log_expire
>
> 1/ 5 mds_migrator
>
> 0/ 1 buffer
>
> 0/ 1 timer
>
> 0/ 1 filer
>
> 0/ 1 striper
>
> 0/ 1 objecter
>
> 0/ 5 rados
>
> 0/ 5 rbd
>
> 0/ 5 journaler
>
> 0/ 5 objectcacher
>
> 0/ 5 client
>
> 0/ 5 osd
>
> 0/ 5 optracker
>
> 0/ 5 objclass
>
> 1/ 3 filestore
>
> 1/ 3 keyvaluestore
>
> 1/ 3 journal
>
> 0/ 5 ms
>
> 1/ 5 mon
>
> 0/10 monc
>
> 1/ 5 paxos
>
> 0/ 5 tp
>
> 1/ 5 auth
>
> 1/ 5 crypto
>
> 1/ 1 finisher
>
> 1/ 5 heartbeatmap
>
> 1/ 5 perfcounter
>
> 1/ 5 rgw
>
> 1/ 5 javaclient
>
> 1/ 5 asok
>
> 1/ 1 throttle
>
> -2/-2 (syslog threshold)
>
> -1/-1 (stderr threshold)
>
> max_recent 10000
>
> max_new 1000
>
> log_file /var/log/ceph/us-west01-osd.0.log
>
> --- end dump of recent events ---
>
>
> _______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
