I was able to catch on the crashes
pn 951901 opn 0) v3 ==== 982012+0+0 (3546980463 0 0) 0x5e2ea00 con 0x3701420
-3> 2013-06-10 15:45:47.944085 7f27a5038700 20 mon.3@2(peon) e1 have
connection
-2> 2013-06-10 15:45:47.944092 7f27a5038700 20 mon.3@2(peon) e1 ms_dispatch
existing session MonSession: mon.1 10.198.141.202:6789/0 is openallow * for
mon.1 10.198.141.202:6789/0
-1> 2013-06-10 15:45:47.944100 7f27a5038700 20 mon.3@2(peon) e1 caps allow
*
0> 2013-06-10 15:45:47.993974 7f27a5038700 -1 mon/Paxos.cc: In function
'void Paxos::handle_begin(MMonPaxos*)' thread 7f27a5038700 time 2013-06-10
15:45:47.944106
mon/Paxos.cc: 554: FAILED assert(begin->last_committed == last_committed)
ceph version 0.61.3 (92b1e398576d55df8e5888dd1a9545ed3fd99532)
1: (Paxos::handle_begin(MMonPaxos*)+0x9cf) [0x4ddc3f]
2: (Paxos::dispatch(PaxosServiceMessage*)+0x25b) [0x4e269b]
3: (Monitor::_ms_dispatch(Message*)+0x10dc) [0x4b79ec]
4: (Monitor::ms_dispatch(Message*)+0x32) [0x4d1902]
5: (DispatchQueue::entry()+0x3f1) [0x6abae1]
6: (DispatchQueue::DispatchThread::entry()+0xd) [0x639bad]
7: (()+0x7e9a) [0x7f27aa62de9a]
8: (clone()+0x6d) [0x7f27a92c8ccd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
1/ 1 ms
20/20 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-1/-1 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
2013-06-10 15:45:50.148146 7f279d9df700 1 -- 10.198.141.203:6789/0 >> :/0
pipe(0x3830a00 sd=869 :6789 s=0 pgs=0 cs=0 l=0).accept sd=869
10.198.141.34:34566/0
2013-06-10 15:45:50.148355 7f279d9df700 10 mon.3@2(peon) e1
ms_verify_authorizer 10.198.141.34:6803/10847 osd protocol 0
terminate called after throwing an instance of 'ceph::FailedAssertion'
*** Caught signal (Aborted) **
in thread 7f27a5038700
ceph version 0.61.3 (92b1e398576d55df8e5888dd1a9545ed3fd99532)
1: ceph-mon() [0x59255a]
2: (()+0xfcb0) [0x7f27aa635cb0]
3: (gsignal()+0x35) [0x7f27a920b425]
4: (abort()+0x17b) [0x7f27a920eb8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f27a9b5d69d]
6: (()+0xb5846) [0x7f27a9b5b846]
7: (()+0xb5873) [0x7f27a9b5b873]
8: (()+0xb596e) [0x7f27a9b5b96e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df)
[0x649d6f]
10: (Paxos::handle_begin(MMonPaxos*)+0x9cf) [0x4ddc3f]
11: (Paxos::dispatch(PaxosServiceMessage*)+0x25b) [0x4e269b]
12: (Monitor::_ms_dispatch(Message*)+0x10dc) [0x4b79ec]
13: (Monitor::ms_dispatch(Message*)+0x32) [0x4d1902]
14: (DispatchQueue::entry()+0x3f1) [0x6abae1]
15: (DispatchQueue::DispatchThread::entry()+0xd) [0x639bad]
16: (()+0x7e9a) [0x7f27aa62de9a]
17: (clone()+0x6d) [0x7f27a92c8ccd]
2013-06-10 15:45:51.878383 7f27a5038700 -1 *** Caught signal (Aborted) **
in thread 7f27a5038700
ceph version 0.61.3 (92b1e398576d55df8e5888dd1a9545ed3fd99532)
1: ceph-mon() [0x59255a]
2: (()+0xfcb0) [0x7f27aa635cb0]
3: (gsignal()+0x35) [0x7f27a920b425]
4: (abort()+0x17b) [0x7f27a920eb8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f27a9b5d69d]
6: (()+0xb5846) [0x7f27a9b5b846]
7: (()+0xb5873) [0x7f27a9b5b873]
8: (()+0xb596e) [0x7f27a9b5b96e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df)
[0x649d6f]
10: (Paxos::handle_begin(MMonPaxos*)+0x9cf) [0x4ddc3f]
11: (Paxos::dispatch(PaxosServiceMessage*)+0x25b) [0x4e269b]
12: (Monitor::_ms_dispatch(Message*)+0x10dc) [0x4b79ec]
13: (Monitor::ms_dispatch(Message*)+0x32) [0x4d1902]
14: (DispatchQueue::entry()+0x3f1) [0x6abae1]
15: (DispatchQueue::DispatchThread::entry()+0xd) [0x639bad]
16: (()+0x7e9a) [0x7f27aa62de9a]
17: (clone()+0x6d) [0x7f27a92c8ccd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
--- begin dump of recent events ---
-2> 2013-06-10 15:45:50.148146 7f279d9df700 1 -- 10.198.141.203:6789/0 >>
:/0 pipe(0x3830a00 sd=869 :6789 s=0 pgs=0 cs=0 l=0).accept sd=869
10.198.141.34:34566/0
-1> 2013-06-10 15:45:50.148355 7f279d9df700 10 mon.3@2(peon) e1
ms_verify_authorizer 10.198.141.34:6803/10847 osd protocol 0
0> 2013-06-10 15:45:51.878383 7f27a5038700 -1 *** Caught signal (Aborted)
**
in thread 7f27a5038700
ceph version 0.61.3 (92b1e398576d55df8e5888dd1a9545ed3fd99532)
1: ceph-mon() [0x59255a]
2: (()+0xfcb0) [0x7f27aa635cb0]
3: (gsignal()+0x35) [0x7f27a920b425]
4: (abort()+0x17b) [0x7f27a920eb8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f27a9b5d69d]
6: (()+0xb5846) [0x7f27a9b5b846]
7: (()+0xb5873) [0x7f27a9b5b873]
8: (()+0xb596e) [0x7f27a9b5b96e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df)
[0x649d6f]
10: (Paxos::handle_begin(MMonPaxos*)+0x9cf) [0x4ddc3f]
11: (Paxos::dispatch(PaxosServiceMessage*)+0x25b) [0x4e269b]
12: (Monitor::_ms_dispatch(Message*)+0x10dc) [0x4b79ec]
13: (Monitor::ms_dispatch(Message*)+0x32) [0x4d1902]
14: (DispatchQueue::entry()+0x3f1) [0x6abae1]
15: (DispatchQueue::DispatchThread::entry()+0xd) [0x639bad]
16: (()+0x7e9a) [0x7f27aa62de9a]
17: (clone()+0x6d) [0x7f27a92c8ccd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
1/ 1 ms
20/20 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-1/-1 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
Nelson Jeppesen
Disney Technology Solutions and Services
Phone 206-588-5001
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
