At a glance that looks like the bug fixed by just-merged
https://github.com/ceph/ceph/pull/16421
On Thu, Jul 20, 2017 at 1:02 PM Roger Brown <rogerpbr...@gmail.com> wrote:
> I'm on Luminous 12.1.1 and noticed I have flapping OSDs. Even with `ceph
> osd set nodown`, the OSDs will catch signal Aborted and sometimes
> Segmentation fault 2-5 minutes after starting. I verified hosts can talk to
> eachother on the cluster network. I've rebooted the hosts. I'm running out
> of ideas. Please advise.
>
> Tally of crashes:
> roger@osd1:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog{,.1} |
> awk '{print $9}' | sort | uniq -c | sort -nr
> 100 (Segmentation
> 77 (Aborted)
> roger@osd2:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog{,.1} |
> awk '{print $9}' | sort | uniq -c | sort -nr
> 77 (Aborted)
> 13 (Segmentation
> roger@osd3:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog{,.1} |
> awk '{print $9}' | sort | uniq -c | sort -nr
> 86 (Aborted)
> 3 (Segmentation
>
> First crash observed Jul 19:
> roger@osd1:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog.1 | head
> -1
> Jul 19 10:07:12 osd1 ceph-osd[13491]: *** Caught signal (Aborted) **
> roger@osd2:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog.1 | head
> -1
> Jul 19 10:07:36 osd2 ceph-osd[13937]: *** Caught signal (Aborted) **
> roger@osd3:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog.1 | head
> -1
> Jul 19 16:07:12 osd3 ceph-osd[8807]: *** Caught signal (Aborted) **
>
> Crashes started with Luminous 12.1.0:
> roger@osd1:~$ sudo grep 'Jul 19 10:07:12.*ceph version' /var/log/syslog.1
> | head -1
> Jul 19 10:07:12 osd1 ceph-osd[13491]: ceph version 12.1.0
> (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
> roger@osd2:~$ sudo grep 'Jul 19 10:07:36.*ceph version' /var/log/syslog.1
> | head -1
> Jul 19 10:07:36 osd2 ceph-osd[13937]: ceph version 12.1.0
> (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
> roger@osd3:~$ sudo grep 'Jul 19 16:07:12.*ceph version' /var/log/syslog.1
> | head -1
> Jul 19 16:07:12 osd3 ceph-osd[8807]: ceph version 12.1.0
> (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
>
> Representative example from osd1 logs:
> Jul 20 13:42:18 osd1 ceph-osd[4035]: *** Caught signal (Segmentation
> fault) **
> Jul 20 13:42:18 osd1 ceph-osd[4035]: in thread 7f52960e7700
> thread_name:msgr-worker-2
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 2017-07-20 13:42:18.658076
> 7f529bf85c80 -1 osd.3 3444 log_to_monitors {default=true}
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 2017-07-20 13:42:18.662695
> 7f52968e8700 -1 failed to decode message of type 70 v3:
> buffer::malformed_input: void
> osd_peer_stat_t::decode(ceph::buffer::list::iterator&) no longer understand
> old encoding version 1 < struct_compat
> Jul 20 13:42:18 osd1 ceph-osd[4035]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 1: (()+0xa257a4) [0x55bc98fe27a4]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 2: (()+0x11390) [0x7f529a468390]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 3:
> (cephx_verify_authorizer(CephContext*, KeyStore*,
> ceph::buffer::list::iterator&, CephXServiceTicketInfo&,
> ceph::buffer::list&)+0x496) [0x55bc991b0ca6]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 4:
> (CephxAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*,
> ceph::buffer::list&, ceph::buffer::list&, EntityName&, unsigned long&,
> AuthCapsInfo&, CryptoKey&, unsigned long*)+0x31a) [0x55bc991a2cda]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 5:
> (OSD::ms_verify_authorizer(Connection*, int, int, ceph::buffer::list&,
> ceph::buffer::list&, bool&, CryptoKey&)+0xf9) [0x55bc98a2c759]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 6:
> (AsyncConnection::handle_connect_msg(ceph_msg_connect&,
> ceph::buffer::list&, ceph::buffer::list&)+0x228) [0x55bc99271108]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 7:
> (AsyncConnection::_process_connection()+0x1e07) [0x55bc99276a57]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 8:
> (AsyncConnection::process()+0x1ae8) [0x55bc9927b978]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 9: (EventCenter::process_events(int,
> std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08)
> [0x55bc990c6148]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 10: (()+0xb0d0d8) [0x55bc990ca0d8]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 11: (()+0xb8c80) [0x7f5299d6fc80]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 12: (()+0x76ba) [0x7f529a45e6ba]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 13: (clone()+0x6d) [0x7f52994d53dd]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 2017-07-20 13:42:18.662763
> 7f52960e7700 -1 *** Caught signal (Segmentation fault) **
> Jul 20 13:42:18 osd1 ceph-osd[4035]: in thread 7f52960e7700
> thread_name:msgr-worker-2
> Jul 20 13:42:18 osd1 ceph-osd[4035]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 1: (()+0xa257a4) [0x55bc98fe27a4]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 2: (()+0x11390) [0x7f529a468390]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 3:
> (cephx_verify_authorizer(CephContext*, KeyStore*,
> ceph::buffer::list::iterator&, CephXServiceTicketInfo&,
> ceph::buffer::list&)+0x496) [0x55bc991b0ca6]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 4:
> (CephxAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*,
> ceph::buffer::list&, ceph::buffer::list&, EntityName&, unsigned long&,
> AuthCapsInfo&, CryptoKey&, unsigned long*)+0x31a) [0x55bc991a2cda]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 5:
> (OSD::ms_verify_authorizer(Connection*, int, int, ceph::buffer::list&,
> ceph::buffer::list&, bool&, CryptoKey&)+0xf9) [0x55bc98a2c759]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 6:
> (AsyncConnection::handle_connect_msg(ceph_msg_connect&,
> ceph::buffer::list&, ceph::buffer::list&)+0x228) [0x55bc99271108]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 7:
> (AsyncConnection::_process_connection()+0x1e07) [0x55bc99276a57]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 8:
> (AsyncConnection::process()+0x1ae8) [0x55bc9927b978]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 9: (EventCenter::process_events(int,
> std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08)
> [0x55bc990c6148]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 10: (()+0xb0d0d8) [0x55bc990ca0d8]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 11: (()+0xb8c80) [0x7f5299d6fc80]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 12: (()+0x76ba) [0x7f529a45e6ba]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 13: (clone()+0x6d) [0x7f52994d53dd]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Jul 20 13:42:18 osd1 ceph-osd[4035]: -18> 2017-07-20 13:42:18.658076
> 7f529bf85c80 -1 osd.3 3444 log_to_monitors {default=true}
> Jul 20 13:42:18 osd1 ceph-osd[4035]: -5> 2017-07-20 13:42:18.662695
> 7f52968e8700 -1 failed to decode message of type 70 v3:
> buffer::malformed_input: void
> osd_peer_stat_t::decode(ceph::buffer::list::iterator&) no longer understand
> old encoding version 1 < struct_compat
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 0> 2017-07-20 13:42:18.662763
> 7f52960e7700 -1 *** Caught signal (Segmentation fault) **
> Jul 20 13:42:18 osd1 ceph-osd[4035]: in thread 7f52960e7700
> thread_name:msgr-worker-2
> Jul 20 13:42:18 osd1 ceph-osd[4035]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 1: (()+0xa257a4) [0x55bc98fe27a4]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 2: (()+0x11390) [0x7f529a468390]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 3:
> (cephx_verify_authorizer(CephContext*, KeyStore*,
> ceph::buffer::list::iterator&, CephXServiceTicketInfo&,
> ceph::buffer::list&)+0x496) [0x55bc991b0ca6]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 4:
> (CephxAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*,
> ceph::buffer::list&, ceph::buffer::list&, EntityName&, unsigned long&,
> AuthCapsInfo&, CryptoKey&, unsigned long*)+0x31a) [0x55bc991a2cda]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 5:
> (OSD::ms_verify_authorizer(Connection*, int, int, ceph::buffer::list&,
> ceph::buffer::list&, bool&, CryptoKey&)+0xf9) [0x55bc98a2c759]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 6:
> (AsyncConnection::handle_connect_msg(ceph_msg_connect&,
> ceph::buffer::list&, ceph::buffer::list&)+0x228) [0x55bc99271108]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 7:
> (AsyncConnection::_process_connection()+0x1e07) [0x55bc99276a57]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 8:
> (AsyncConnection::process()+0x1ae8) [0x55bc9927b978]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 9: (EventCenter::process_events(int,
> std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08)
> [0x55bc990c6148]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 10: (()+0xb0d0d8) [0x55bc990ca0d8]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 11: (()+0xb8c80) [0x7f5299d6fc80]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 12: (()+0x76ba) [0x7f529a45e6ba]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: 13: (clone()+0x6d) [0x7f52994d53dd]
> Jul 20 13:42:18 osd1 ceph-osd[4035]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Jul 20 13:42:18 osd1 systemd[1]: ceph-osd@3.service: Main process exited,
> code=killed, status=11/SEGV
> Jul 20 13:42:18 osd1 systemd[1]: ceph-osd@3.service: Unit entered failed
> state.
> Jul 20 13:42:18 osd1 systemd[1]: ceph-osd@3.service: Failed with result
> 'signal'.
> Jul 20 13:42:38 osd1 systemd[1]: ceph-osd@3.service: Service hold-off
> time over, scheduling restart.
> Jul 20 13:42:38 osd1 systemd[1]: Stopped Ceph object storage daemon osd.3.
> Jul 20 13:42:38 osd1 systemd[1]: Starting Ceph object storage daemon
> osd.3...
> Jul 20 13:42:39 osd1 systemd[1]: Started Ceph object storage daemon osd.3.
> Jul 20 13:42:39 osd1 ceph-osd[4130]: starting osd.3 at - osd_data
> /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
> Jul 20 13:43:02 osd1 sshd[3497]: Received disconnect from 192.168.0.7 port
> 55258:11: disconnected by user
> Jul 20 13:43:02 osd1 sshd[3497]: Disconnected from 192.168.0.7 port 55258
> Jul 20 13:43:02 osd1 sshd[3466]: pam_unix(sshd:session): session closed
> for user roger
> Jul 20 13:43:02 osd1 systemd-logind[1393]: Removed session 10.
> Jul 20 13:44:53 osd1 ceph-osd[4130]: 2017-07-20 13:44:53.540934
> 7f303995dc80 -1 osd.3 3444 log_to_monitors {default=true}
> Jul 20 13:45:33 osd1 ceph-osd[4130]: 2017-07-20 13:45:33.544688
> 7f30302de700 -1 osd.3 3458 heartbeat_check: no reply from
> 192.168.0.26:6801 osd.0 since back 2017-07-20 13:45:12.643355 front
> 2017-07-20 13:45:12.643355 (cutoff 2017-07-20 13:45:13.544686)
> Jul 20 13:46:01 osd1 ceph-osd[4130]:
> /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static void
> osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*,
> char**, uint32_t*)' thread 7f3033abf700 time 2017-07-20 13:46:01.429584
> Jul 20 13:46:01 osd1 ceph-osd[4130]:
> /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end)
> Jul 20 13:46:01 osd1 ceph-osd[4130]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x102) [0x55f2073ffb72]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 2:
> (std::enable_if<denc_traits<osd_reqid_t,
> void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous,
> void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void>
> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 3:
> (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 4: (decode_message(CephContext*,
> int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&,
> ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8)
> [0x55f20743a5d8]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 5:
> (AsyncConnection::process()+0x1d4e) [0x55f207656bde]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 6: (EventCenter::process_events(int,
> std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08)
> [0x55f2074a1148]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 7: (()+0xb0d0d8) [0x55f2074a50d8]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 8: (()+0xb8c80) [0x7f3037747c80]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 9: (()+0x76ba) [0x7f3037e366ba]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 10: (clone()+0x6d) [0x7f3036ead3dd]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 2017-07-20 13:46:01.434169
> 7f3033abf700 -1 /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static
> void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*,
> char**, uint32_t*)' thread 7f3033abf700 time 2017-07-20 13:46:01.429584
> Jul 20 13:46:01 osd1 ceph-osd[4130]:
> /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end)
> Jul 20 13:46:01 osd1 ceph-osd[4130]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x102) [0x55f2073ffb72]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 2:
> (std::enable_if<denc_traits<osd_reqid_t,
> void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous,
> void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void>
> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 3:
> (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 4: (decode_message(CephContext*,
> int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&,
> ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8)
> [0x55f20743a5d8]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 5:
> (AsyncConnection::process()+0x1d4e) [0x55f207656bde]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 6: (EventCenter::process_events(int,
> std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08)
> [0x55f2074a1148]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 7: (()+0xb0d0d8) [0x55f2074a50d8]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 8: (()+0xb8c80) [0x7f3037747c80]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 9: (()+0x76ba) [0x7f3037e366ba]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 10: (clone()+0x6d) [0x7f3036ead3dd]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 0> 2017-07-20 13:46:01.434169
> 7f3033abf700 -1 /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static
> void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*,
> char**, uint32_t*)' thread 7f3033abf700 time 2017-07-20 13:46:01.429584
> Jul 20 13:46:01 osd1 ceph-osd[4130]:
> /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end)
> Jul 20 13:46:01 osd1 ceph-osd[4130]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x102) [0x55f2073ffb72]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 2:
> (std::enable_if<denc_traits<osd_reqid_t,
> void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous,
> void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void>
> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 3:
> (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 4: (decode_message(CephContext*,
> int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&,
> ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8)
> [0x55f20743a5d8]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 5:
> (AsyncConnection::process()+0x1d4e) [0x55f207656bde]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 6: (EventCenter::process_events(int,
> std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08)
> [0x55f2074a1148]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 7: (()+0xb0d0d8) [0x55f2074a50d8]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 8: (()+0xb8c80) [0x7f3037747c80]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 9: (()+0x76ba) [0x7f3037e366ba]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 10: (clone()+0x6d) [0x7f3036ead3dd]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Jul 20 13:46:01 osd1 ceph-osd[4130]: *** Caught signal (Aborted) **
> Jul 20 13:46:01 osd1 ceph-osd[4130]: in thread 7f3033abf700
> thread_name:msgr-worker-2
> Jul 20 13:46:01 osd1 ceph-osd[4130]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 1: (()+0xa257a4) [0x55f2073bd7a4]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 2: (()+0x11390) [0x7f3037e40390]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 3: (gsignal()+0x38) [0x7f3036ddb428]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 4: (abort()+0x16a) [0x7f3036ddd02a]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 5: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x28e) [0x55f2073ffcfe]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 6:
> (std::enable_if<denc_traits<osd_reqid_t,
> void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous,
> void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void>
> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 7:
> (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 8: (decode_message(CephContext*,
> int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&,
> ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8)
> [0x55f20743a5d8]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 9:
> (AsyncConnection::process()+0x1d4e) [0x55f207656bde]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 10:
> (EventCenter::process_events(int, std::chrono::duration<unsigned long,
> std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 11: (()+0xb0d0d8) [0x55f2074a50d8]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 12: (()+0xb8c80) [0x7f3037747c80]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 13: (()+0x76ba) [0x7f3037e366ba]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 14: (clone()+0x6d) [0x7f3036ead3dd]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 2017-07-20 13:46:01.481468
> 7f3033abf700 -1 *** Caught signal (Aborted) **
> Jul 20 13:46:01 osd1 ceph-osd[4130]: in thread 7f3033abf700
> thread_name:msgr-worker-2
> Jul 20 13:46:01 osd1 ceph-osd[4130]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 1: (()+0xa257a4) [0x55f2073bd7a4]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 2: (()+0x11390) [0x7f3037e40390]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 3: (gsignal()+0x38) [0x7f3036ddb428]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 4: (abort()+0x16a) [0x7f3036ddd02a]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 5: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x28e) [0x55f2073ffcfe]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 6:
> (std::enable_if<denc_traits<osd_reqid_t,
> void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous,
> void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void>
> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 7:
> (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 8: (decode_message(CephContext*,
> int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&,
> ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8)
> [0x55f20743a5d8]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 9:
> (AsyncConnection::process()+0x1d4e) [0x55f207656bde]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 10:
> (EventCenter::process_events(int, std::chrono::duration<unsigned long,
> std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 11: (()+0xb0d0d8) [0x55f2074a50d8]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 12: (()+0xb8c80) [0x7f3037747c80]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 13: (()+0x76ba) [0x7f3037e366ba]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 14: (clone()+0x6d) [0x7f3036ead3dd]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 0> 2017-07-20 13:46:01.481468
> 7f3033abf700 -1 *** Caught signal (Aborted) **
> Jul 20 13:46:01 osd1 ceph-osd[4130]: in thread 7f3033abf700
> thread_name:msgr-worker-2
> Jul 20 13:46:01 osd1 ceph-osd[4130]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 1: (()+0xa257a4) [0x55f2073bd7a4]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 2: (()+0x11390) [0x7f3037e40390]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 3: (gsignal()+0x38) [0x7f3036ddb428]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 4: (abort()+0x16a) [0x7f3036ddd02a]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 5: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x28e) [0x55f2073ffcfe]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 6:
> (std::enable_if<denc_traits<osd_reqid_t,
> void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous,
> void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void>
> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 7:
> (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 8: (decode_message(CephContext*,
> int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&,
> ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8)
> [0x55f20743a5d8]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 9:
> (AsyncConnection::process()+0x1d4e) [0x55f207656bde]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 10:
> (EventCenter::process_events(int, std::chrono::duration<unsigned long,
> std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 11: (()+0xb0d0d8) [0x55f2074a50d8]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 12: (()+0xb8c80) [0x7f3037747c80]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 13: (()+0x76ba) [0x7f3037e366ba]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: 14: (clone()+0x6d) [0x7f3036ead3dd]
> Jul 20 13:46:01 osd1 ceph-osd[4130]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Jul 20 13:46:01 osd1 systemd[1]: ceph-osd@3.service: Main process exited,
> code=killed, status=6/ABRT
> Jul 20 13:46:01 osd1 systemd[1]: ceph-osd@3.service: Unit entered failed
> state.
> Jul 20 13:46:01 osd1 systemd[1]: ceph-osd@3.service: Failed with result
> 'signal'.
> Jul 20 13:46:21 osd1 systemd[1]: ceph-osd@3.service: Service hold-off
> time over, scheduling restart.
> Jul 20 13:46:21 osd1 systemd[1]: Stopped Ceph object storage daemon osd.3.
> Jul 20 13:46:21 osd1 systemd[1]: Starting Ceph object storage daemon
> osd.3...
> Jul 20 13:46:22 osd1 systemd[1]: Started Ceph object storage daemon osd.3.
> Jul 20 13:46:22 osd1 ceph-osd[4223]: starting osd.3 at - osd_data
> /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
> Jul 20 13:48:39 osd1 ceph-osd[4223]: *** Caught signal (Segmentation
> fault) **
> Jul 20 13:48:39 osd1 ceph-osd[4223]: in thread 7f10b1baa700
> thread_name:msgr-worker-2
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 2017-07-20 13:48:39.470084
> 7f10b7a48c80 -1 osd.3 3460 log_to_monitors {default=true}
> Jul 20 13:48:39 osd1 ceph-osd[4223]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 1: (()+0xa257a4) [0x55cead5be7a4]
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 2: (()+0x11390) [0x7f10b5f2b390]
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 3:
> (cephx_verify_authorizer(CephContext*, KeyStore*,
> ceph::buffer::list::iterator&, CephXServiceTicketInfo&,
> ceph::buffer::list&)+0x496) [0x55cead78cca6]
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 4:
> (CephxAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*,
> ceph::buffer::list&, ceph::buffer::list&, EntityName&, unsigned long&,
> AuthCapsInfo&, CryptoKey&, unsigned long*)+0x31a) [0x55cead77ecda]
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 5:
> (OSD::ms_verify_authorizer(Connection*, int, int, ceph::buffer::list&,
> ceph::buffer::list&, bool&, CryptoKey&)+0xf9) [0x55cead008759]
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 6:
> (AsyncConnection::handle_connect_msg(ceph_msg_connect&,
> ceph::buffer::list&, ceph::buffer::list&)+0x228) [0x55cead84d108]
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 7:
> (AsyncConnection::_process_connection()+0x1e07) [0x55cead852a57]
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 7:
> (AsyncConnection::_process_connection()+0x1e07) [0x55cead852a57]
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 8:
> (AsyncConnection::process()+0x1ae8) [0x55cead857978]
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 9: (EventCenter::process_events(int,
> std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08)
> [0x55cead6a2148]
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 10: (()+0xb0d0d8) [0x55cead6a60d8]
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 11: (()+0xb8c80) [0x7f10b5832c80]
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 12: (()+0x76ba) [0x7f10b5f216ba]
> Jul 20 13:48:39 osd1 ceph-osd[4223]: 13: (clone()+0x6d) [0x7f10b4f983dd]
> Jul 20 13:48:39 osd1 ceph-osd[4223]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Jul 20 13:48:39 osd1 systemd[1]: ceph-osd@3.service: Main process exited,
> code=killed, status=11/SEGV
> Jul 20 13:48:39 osd1 systemd[1]: ceph-osd@3.service: Unit entered failed
> state.
> Jul 20 13:48:39 osd1 systemd[1]: ceph-osd@3.service: Failed with result
> 'signal'.
> Jul 20 13:48:59 osd1 systemd[1]: ceph-osd@3.service: Service hold-off
> time over, scheduling restart.
> Jul 20 13:48:59 osd1 systemd[1]: Stopped Ceph object storage daemon osd.3.
> Jul 20 13:48:59 osd1 systemd[1]: Starting Ceph object storage daemon
> osd.3...
> Jul 20 13:49:00 osd1 systemd[1]: Started Ceph object storage daemon osd.3.
> Jul 20 13:49:00 osd1 ceph-osd[4314]: starting osd.3 at - osd_data
> /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
> Jul 20 13:51:15 osd1 ceph-osd[4314]: 2017-07-20 13:51:15.595553
> 7feabcf96c80 -1 osd.3 3460 log_to_monitors {default=true}
> Jul 20 13:51:54 osd1 ceph-osd[4314]: 2017-07-20 13:51:54.599050
> 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from
> 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front
> 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:34.599047)
> Jul 20 13:51:55 osd1 ceph-osd[4314]: 2017-07-20 13:51:55.599219
> 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from
> 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front
> 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:35.599217)
> Jul 20 13:51:56 osd1 ceph-osd[4314]: 2017-07-20 13:51:56.599336
> 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from
> 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front
> 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:36.599335)
> Jul 20 13:51:57 osd1 ceph-osd[4314]: 2017-07-20 13:51:57.599445
> 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from
> 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front
> 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:37.599443)
> Jul 20 13:51:58 osd1 ceph-osd[4314]: 2017-07-20 13:51:58.599563
> 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from
> 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front
> 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:38.599562)
> Jul 20 13:52:26 osd1 ceph-osd[4314]:
> /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static void
> osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*,
> char**, uint32_t*)' thread 7feab78f9700 time 2017-07-20 13:52:26.501284
> Jul 20 13:52:26 osd1 ceph-osd[4314]:
> /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end)
> Jul 20 13:52:26 osd1 ceph-osd[4314]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x102) [0x5565a2421b72]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 2:
> (std::enable_if<denc_traits<osd_reqid_t,
> void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous,
> void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void>
> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 3:
> (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 4: (decode_message(CephContext*,
> int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&,
> ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8)
> [0x5565a245c5d8]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 5:
> (AsyncConnection::process()+0x1d4e) [0x5565a2678bde]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 6: (EventCenter::process_events(int,
> std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08)
> [0x5565a24c3148]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 7: (()+0xb0d0d8) [0x5565a24c70d8]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 8: (()+0xb8c80) [0x7feabad80c80]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 9: (()+0x76ba) [0x7feabb46f6ba]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 10: (clone()+0x6d) [0x7feaba4e63dd]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 2017-07-20 13:52:26.505919
> 7feab78f9700 -1 /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static
> void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*,
> char**, uint32_t*)' thread 7feab78f9700 time 2017-07-20 13:52:26.501284
> Jul 20 13:52:26 osd1 ceph-osd[4314]:
> /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end)
> Jul 20 13:52:26 osd1 ceph-osd[4314]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x102) [0x5565a2421b72]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 2:
> (std::enable_if<denc_traits<osd_reqid_t,
> void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous,
> void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void>
> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 3:
> (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 4: (decode_message(CephContext*,
> int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&,
> ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8)
> [0x5565a245c5d8]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 5:
> (AsyncConnection::process()+0x1d4e) [0x5565a2678bde]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 6: (EventCenter::process_events(int,
> std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08)
> [0x5565a24c3148]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 7: (()+0xb0d0d8) [0x5565a24c70d8]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 8: (()+0xb8c80) [0x7feabad80c80]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 9: (()+0x76ba) [0x7feabb46f6ba]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 10: (clone()+0x6d) [0x7feaba4e63dd]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 0> 2017-07-20 13:52:26.505919
> 7feab78f9700 -1 /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static
> void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*,
> char**, uint32_t*)' thread 7feab78f9700 time 2017-07-20 13:52:26.501284
> Jul 20 13:52:26 osd1 ceph-osd[4314]:
> /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end)
> Jul 20 13:52:26 osd1 ceph-osd[4314]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x102) [0x5565a2421b72]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 2:
> (std::enable_if<denc_traits<osd_reqid_t,
> void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous,
> void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void>
> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 3:
> (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 4: (decode_message(CephContext*,
> int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&,
> ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8)
> [0x5565a245c5d8]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 5:
> (AsyncConnection::process()+0x1d4e) [0x5565a2678bde]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 6: (EventCenter::process_events(int,
> std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08)
> [0x5565a24c3148]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 7: (()+0xb0d0d8) [0x5565a24c70d8]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 8: (()+0xb8c80) [0x7feabad80c80]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 9: (()+0x76ba) [0x7feabb46f6ba]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 10: (clone()+0x6d) [0x7feaba4e63dd]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Jul 20 13:52:26 osd1 ceph-osd[4314]: *** Caught signal (Aborted) **
> Jul 20 13:52:26 osd1 ceph-osd[4314]: in thread 7feab78f9700
> thread_name:msgr-worker-1
> Jul 20 13:52:26 osd1 ceph-osd[4314]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 1: (()+0xa257a4) [0x5565a23df7a4]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 2: (()+0x11390) [0x7feabb479390]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 3: (gsignal()+0x38) [0x7feaba414428]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 4: (abort()+0x16a) [0x7feaba41602a]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 5: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x28e) [0x5565a2421cfe]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 6:
> (std::enable_if<denc_traits<osd_reqid_t,
> void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous,
> void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void>
> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 7:
> (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 8: (decode_message(CephContext*,
> int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&,
> ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8)
> [0x5565a245c5d8]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 9:
> (AsyncConnection::process()+0x1d4e) [0x5565a2678bde]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 10:
> (EventCenter::process_events(int, std::chrono::duration<unsigned long,
> std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 11: (()+0xb0d0d8) [0x5565a24c70d8]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 12: (()+0xb8c80) [0x7feabad80c80]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 13: (()+0x76ba) [0x7feabb46f6ba]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 14: (clone()+0x6d) [0x7feaba4e63dd]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 2017-07-20 13:52:26.554188
> 7feab78f9700 -1 *** Caught signal (Aborted) **
> Jul 20 13:52:26 osd1 ceph-osd[4314]: in thread 7feab78f9700
> thread_name:msgr-worker-1
> Jul 20 13:52:26 osd1 ceph-osd[4314]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 1: (()+0xa257a4) [0x5565a23df7a4]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 2: (()+0x11390) [0x7feabb479390]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 3: (gsignal()+0x38) [0x7feaba414428]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 4: (abort()+0x16a) [0x7feaba41602a]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 5: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x28e) [0x5565a2421cfe]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 6:
> (std::enable_if<denc_traits<osd_reqid_t,
> void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous,
> void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void>
> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 7:
> (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 8: (decode_message(CephContext*,
> int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&,
> ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8)
> [0x5565a245c5d8]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 9:
> (AsyncConnection::process()+0x1d4e) [0x5565a2678bde]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 10:
> (EventCenter::process_events(int, std::chrono::duration<unsigned long,
> std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 11: (()+0xb0d0d8) [0x5565a24c70d8]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 12: (()+0xb8c80) [0x7feabad80c80]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 13: (()+0x76ba) [0x7feabb46f6ba]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 14: (clone()+0x6d) [0x7feaba4e63dd]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 0> 2017-07-20 13:52:26.554188
> 7feab78f9700 -1 *** Caught signal (Aborted) **
> Jul 20 13:52:26 osd1 ceph-osd[4314]: in thread 7feab78f9700
> thread_name:msgr-worker-1
> Jul 20 13:52:26 osd1 ceph-osd[4314]: ceph version 12.1.1
> (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 1: (()+0xa257a4) [0x5565a23df7a4]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 2: (()+0x11390) [0x7feabb479390]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 3: (gsignal()+0x38) [0x7feaba414428]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 4: (abort()+0x16a) [0x7feaba41602a]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 5: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x28e) [0x5565a2421cfe]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 6:
> (std::enable_if<denc_traits<osd_reqid_t,
> void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous,
> void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void>
> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 7:
> (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 8: (decode_message(CephContext*,
> int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&,
> ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8)
> [0x5565a245c5d8]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 9:
> (AsyncConnection::process()+0x1d4e) [0x5565a2678bde]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 10:
> (EventCenter::process_events(int, std::chrono::duration<unsigned long,
> std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 11: (()+0xb0d0d8) [0x5565a24c70d8]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 12: (()+0xb8c80) [0x7feabad80c80]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 13: (()+0x76ba) [0x7feabb46f6ba]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: 14: (clone()+0x6d) [0x7feaba4e63dd]
> Jul 20 13:52:26 osd1 ceph-osd[4314]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Jul 20 13:52:26 osd1 systemd[1]: ceph-osd@3.service: Main process exited,
> code=killed, status=6/ABRT
> Jul 20 13:52:26 osd1 systemd[1]: ceph-osd@3.service: Unit entered failed
> state.
> Jul 20 13:52:26 osd1 systemd[1]: ceph-osd@3.service: Failed with result
> 'signal'.
> Jul 20 13:52:46 osd1 systemd[1]: ceph-osd@3.service: Service hold-off
> time over, scheduling restart.
> Jul 20 13:52:46 osd1 systemd[1]: Stopped Ceph object storage daemon osd.3.
> Jul 20 13:52:46 osd1 systemd[1]: Starting Ceph object storage daemon
> osd.3...
> Jul 20 13:52:47 osd1 systemd[1]: Started Ceph object storage daemon osd.3.
> Jul 20 13:52:47 osd1 ceph-osd[4406]: starting osd.3 at - osd_data
> /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
