Hi,
we are also seeing this (I've also posted to the issue tracker). It only
affects clusters upgraded from Luminous, not new ones.
Also, it's not about re-using OSDs. Deleting any OSD seems to trigger this
bug for all new OSDs on upgraded clusters.
We are still using the pre-Luminous way to remove OSDs, i.e.:
* ceph osd down/stop service
* ceph osd crush remove
* ceph osd auth del
* ceph osd rm
Paul
2018-06-08 22:14 GMT+02:00 Michael Kuriger <mk7...@dexyp.com>:
> Hi everyone,
> I appreciate the suggestions. However, this is still an issue. I've tried
> adding the OSD using ceph-deploy, and manually from the OSD host. I'm not
> able to start newly added OSDs at all, even if I use a new ID. It seems the
> OSD is added to CEPH but I cannot start it. OSDs that existed prior to the
> upgrade to mimic are working fine. Here is a copy of an OSD log entry.
>
> osd.58 0 failed to load OSD map for epoch 378084, got 0 bytes
>
> fsid 1ce494ac-a218-4141-9d4f-295e6fa12f2a
> last_changed 2018-06-05 15:40:50.179880
> created 0.000000
> 0: 10.3.71.36:6789/0 mon.ceph-mon3
> 1: 10.3.74.109:6789/0 mon.ceph-mon2
> 2: 10.3.74.214:6789/0 mon.ceph-mon1
>
> -91> 2018-06-08 12:48:20.697 7fada058e700 1 -- 10.3.56.69:6800/1807239
> <== mon.0 10.3.71.36:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1
> ==== 194+0+0 (645793352 0 0) 0x559f7a3dafc0 con 0x559f7994ec00
> -90> 2018-06-08 12:48:20.697 7fada058e700 10 monclient:
> _check_auth_rotating have uptodate secrets (they expire after 2018-06-08
> 12:47:50.699337)
> -89> 2018-06-08 12:48:20.698 7fadbc9d7140 10 monclient:
> wait_auth_rotating done
> -88> 2018-06-08 12:48:20.698 7fadbc9d7140 10 monclient: _send_command 1
> [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["58"]}]
> -87> 2018-06-08 12:48:20.698 7fadbc9d7140 10 monclient:
> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
> -86> 2018-06-08 12:48:20.698 7fadbc9d7140 1 -- 10.3.56.69:6800/1807239
> --> 10.3.71.36:6789/0 -- mon_command({"prefix": "osd crush
> set-device-class", "class": "hdd", "ids": ["58"]} v 0) v1 -- 0x559f793e73c0
> con 0
> -85> 2018-06-08 12:48:20.700 7fadabaa4700 5 -- 10.3.56.69:6800/1807239
> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
> >> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH
> pgs=25741 cs=1 l=1). rx mon.0 seq 8 0x559f793e73c0
> mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd",
> "ids": ["58"]}]=0 osd.58 already set to class hdd. set-device-class item id
> 58 name 'osd.58' device_class 'hdd': no change. v378738) v1
> -84> 2018-06-08 12:48:20.701 7fada058e700 1 -- 10.3.56.69:6800/1807239
> <== mon.0 10.3.71.36:6789/0 8 ==== mon_command_ack([{"prefix": "osd crush
> set-device-class", "class": "hdd", "ids": ["58"]}]=0 osd.58 already set to
> class hdd. set-device-class item id 58 name 'osd.58' device_class 'hdd': no
> change. v378738) v1 ==== 211+0+0 (4063854475 0 0) 0x559f793e73c0 con
> 0x559f7994ec00
> -83> 2018-06-08 12:48:20.701 7fada058e700 10 monclient:
> handle_mon_command_ack 1 [{"prefix": "osd crush set-device-class", "class":
> "hdd", "ids": ["58"]}]
> -82> 2018-06-08 12:48:20.701 7fada058e700 10 monclient: _finish_command
> 1 = 0 osd.58 already set to class hdd. set-device-class item id 58 name
> 'osd.58' device_class 'hdd': no change.
> -81> 2018-06-08 12:48:20.701 7fadbc9d7140 10 monclient: _send_command 2
> [{"prefix": "osd crush create-or-move", "id": 58, "weight":0.5240, "args":
> ["host=sacephnode12", "root=default"]}]
> -80> 2018-06-08 12:48:20.701 7fadbc9d7140 10 monclient:
> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
> -79> 2018-06-08 12:48:20.701 7fadbc9d7140 1 -- 10.3.56.69:6800/1807239
> --> 10.3.71.36:6789/0 -- mon_command({"prefix": "osd crush
> create-or-move", "id": 58, "weight":0.5240, "args": ["host=sacephnode12",
> "root=default"]} v 0) v1 -- 0x559f793e7600 con 0
> -78> 2018-06-08 12:48:20.703 7fadabaa4700 5 -- 10.3.56.69:6800/1807239
> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
> >> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH
> pgs=25741 cs=1 l=1). rx mon.0 seq 9 0x559f793e7600
> mon_command_ack([{"prefix": "osd crush create-or-move", "id": 58,
> "weight":0.5240, "args": ["host=sacephnode12", "root=default"]}]=0
> create-or-move updated item name 'osd.58' weight 0.524 at location
> {host=sacephnode12,root=default} to crush map v378738) v1
> -77> 2018-06-08 12:48:20.703 7fada058e700 1 -- 10.3.56.69:6800/1807239
> <== mon.0 10.3.71.36:6789/0 9 ==== mon_command_ack([{"prefix": "osd crush
> create-or-move", "id": 58, "weight":0.5240, "args": ["host=sacephnode12",
> "root=default"]}]=0 create-or-move updated item name 'osd.58' weight 0.524
> at location {host=sacephnode12,root=default} to crush map v378738) v1
> ==== 258+0+0 (1998484028 0 0) 0x559f793e7600 con 0x559f7994ec00
> -76> 2018-06-08 12:48:20.703 7fada058e700 10 monclient:
> handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 58,
> "weight":0.5240, "args": ["host=sacephnode12", "root=default"]}]
> -75> 2018-06-08 12:48:20.703 7fada058e700 10 monclient: _finish_command
> 2 = 0 create-or-move updated item name 'osd.58' weight 0.524 at location
> {host=sacephnode12,root=default} to crush map
> -74> 2018-06-08 12:48:20.703 7fadbc9d7140 0 osd.58 0 done with init,
> starting boot process
> -73> 2018-06-08 12:48:20.703 7fadbc9d7140 10 monclient: _renew_subs
> -72> 2018-06-08 12:48:20.703 7fadbc9d7140 10 monclient:
> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
> -71> 2018-06-08 12:48:20.703 7fadbc9d7140 1 -- 10.3.56.69:6800/1807239
> --> 10.3.71.36:6789/0 -- mon_subscribe({mgrmap=0+,osd_pg_creates=0+}) v3
> -- 0x559f79408e00 con 0
> -70> 2018-06-08 12:48:20.703 7fadbc9d7140 1 osd.58 0 start_boot
> -69> 2018-06-08 12:48:20.703 7fadbc9d7140 10 monclient: get_version
> osdmap req 0x559f797667a0
> -68> 2018-06-08 12:48:20.703 7fadbc9d7140 10 monclient:
> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
> -67> 2018-06-08 12:48:20.703 7fadbc9d7140 1 -- 10.3.56.69:6800/1807239
> --> 10.3.71.36:6789/0 -- mon_get_version(what=osdmap handle=1) v1 --
> 0x559f79434b40 con 0
> -66> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command status hook 0x559f793f0700
> -65> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command flush_journal hook 0x559f793f0700
> -64> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command dump_ops_in_flight hook 0x559f793f0700
> -63> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command ops hook 0x559f793f0700
> -62> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command dump_blocked_ops hook 0x559f793f0700
> -61> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command dump_historic_ops hook 0x559f793f0700
> -60> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command dump_historic_slow_ops hook 0x559f793f0700
> -59> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command dump_historic_ops_by_duration hook 0x559f793f0700
> -58> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command dump_op_pq_state hook 0x559f793f0700
> -57> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command dump_blacklist hook 0x559f793f0700
> -56> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command dump_watchers hook 0x559f793f0700
> -55> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command dump_reservations hook 0x559f793f0700
> -54> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command get_latest_osdmap hook 0x559f793f0700
> -53> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command heap hook 0x559f793f0700
> -52> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command set_heap_property hook 0x559f793f0700
> -51> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command get_heap_property hook 0x559f793f0700
> -50> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command dump_objectstore_kv_stats hook 0x559f793f0700
> -49> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command dump_scrubs hook 0x559f793f0700
> -48> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command calc_objectstore_db_histogram hook 0x559f793f0700
> -47> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command flush_store_cache hook 0x559f793f0700
> -46> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command dump_pgstate_history hook 0x559f793f0700
> -45> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command compact hook 0x559f793f0700
> -44> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command get_mapped_pools hook 0x559f793f0700
> -43> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command smart hook 0x559f793f0700
> -42> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command list_devices hook 0x559f793f0700
> -41> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command setomapval hook 0x559f79767280
> -40> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command rmomapkey hook 0x559f79767280
> -39> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command setomapheader hook 0x559f79767280
> -38> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command getomap hook 0x559f79767280
> -37> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command truncobj hook 0x559f79767280
> -36> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command injectdataerr hook 0x559f79767280
> -35> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command injectmdataerr hook 0x559f79767280
> -34> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command set_recovery_delay hook 0x559f79767280
> -33> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command trigger_scrub hook 0x559f79767280
> -32> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
> register_command injectfull hook 0x559f79767280
> -31> 2018-06-08 12:48:20.704 7fadabaa4700 5 -- 10.3.56.69:6800/1807239
> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
> >> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH
> pgs=25741 cs=1 l=1). rx mon.0 seq 10 0x559f7958f8c0 mgrmap(e 201) v1
> -30> 2018-06-08 12:48:20.704 7fadabaa4700 5 -- 10.3.56.69:6800/1807239
> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
> >> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH
> pgs=25741 cs=1 l=1). rx mon.0 seq 11 0x559f79434b40
> mon_get_version_reply(handle=1 version=378738) v2
> -29> 2018-06-08 12:48:20.704 7fada058e700 1 -- 10.3.56.69:6800/1807239
> <== mon.0 10.3.71.36:6789/0 10 ==== mgrmap(e 201) v1 ==== 1776+0+0
> (412200892 0 0) 0x559f7958f8c0 con 0x559f7994ec00
> -28> 2018-06-08 12:48:20.704 7fada058e700 4 mgrc handle_mgr_map Got
> map version 201
> -27> 2018-06-08 12:48:20.704 7fada058e700 4 mgrc handle_mgr_map Active
> mgr is now 10.3.74.109:6801/1015
> -26> 2018-06-08 12:48:20.704 7fada058e700 4 mgrc reconnect Starting
> new session with 10.3.74.109:6801/1015
> -25> 2018-06-08 12:48:20.706 7fadac2a5700 2 -- 10.3.56.69:6800/1807239
> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1
> s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got
> newly_acked_seq 0 vs out_seq 0
> -24> 2018-06-08 12:48:20.706 7fada058e700 1 -- 10.3.56.69:6800/1807239
> --> 10.3.74.109:6801/1015 -- mgropen(unknown.58) v3 -- 0x559f79a9c000 con
> 0
> -23> 2018-06-08 12:48:20.706 7fada058e700 1 -- 10.3.56.69:6800/1807239
> <== mon.0 10.3.71.36:6789/0 11 ==== mon_get_version_reply(handle=1
> version=378738) v2 ==== 24+0+0 (2329122009 0 0) 0x559f79434b40 con
> 0x559f7994ec00
> -22> 2018-06-08 12:48:20.706 7fada058e700 10 monclient:
> handle_get_version_reply finishing 0x559f797667a0 version 378738
> -21> 2018-06-08 12:48:20.706 7fad96a13700 5 osd.58 0 heartbeat:
> osd_stat(1.0 GiB used, 536 GiB avail, 537 GiB total, peers [] op hist [])
> -20> 2018-06-08 12:48:20.706 7fad96a13700 -1 osd.58 0 waiting for
> initial osdmap
> -19> 2018-06-08 12:48:20.706 7fad96a13700 10 monclient: _renew_subs
> -18> 2018-06-08 12:48:20.706 7fad96a13700 10 monclient:
> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
> -17> 2018-06-08 12:48:20.706 7fad96a13700 1 -- 10.3.56.69:6800/1807239
> --> 10.3.71.36:6789/0 -- mon_subscribe({osdmap=378084}) v3 --
> 0x559f7a3b8400 con 0
> -16> 2018-06-08 12:48:20.707 7fadac2a5700 5 -- 10.3.56.69:6800/1807239
> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=201245 cs=1 l=1). rx
> mgr.44007797 seq 1 0x559f79435860 mgrconfigure(period=5, threshold=5) v2
> -15> 2018-06-08 12:48:20.708 7fada058e700 1 -- 10.3.56.69:6800/1807239
> <== mgr.44007797 10.3.74.109:6801/1015 1 ==== mgrconfigure(period=5,
> threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x559f79435860 con
> 0x559f79950a00
> -14> 2018-06-08 12:48:20.708 7fada058e700 4 mgrc handle_mgr_configure
> stats_period=5
> -13> 2018-06-08 12:48:20.708 7fada058e700 4 mgrc handle_mgr_configure
> updated stats threshold: 5
> -12> 2018-06-08 12:48:20.708 7fadabaa4700 5 -- 10.3.56.69:6800/1807239
> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
> >> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH
> pgs=25741 cs=1 l=1). rx mon.0 seq 12 0x559f79aba000 osd_map(378085..378085
> src has 378085..378738 +gap_removed_snaps) v4
> -11> 2018-06-08 12:48:20.708 7fada058e700 1 -- 10.3.56.69:6800/1807239
> --> 10.3.74.109:6801/1015 -- mgrreport(unknown.58 +54-0 packed 742) v6 --
> 0x559f79a9c300 con 0
> -10> 2018-06-08 12:48:20.708 7fada058e700 1 -- 10.3.56.69:6800/1807239
> --> 10.3.74.109:6801/1015 -- pg_stats(0 pgs tid 0 v 0) v1 --
> 0x559f7958f600 con 0
> -9> 2018-06-08 12:48:20.708 7fada058e700 1 -- 10.3.56.69:6800/1807239
> <== mon.0 10.3.71.36:6789/0 12 ==== osd_map(378085..378085 src has
> 378085..378738 +gap_removed_snaps) v4 ==== 33348+0+0 (2799879432 0 0)
> 0x559f79aba000 con 0x559f7994ec00
> -8> 2018-06-08 12:48:20.708 7fada058e700 3 osd.58 0 handle_osd_map
> epochs [378085,378085], i have 0, src has [378085,378738]
> -7> 2018-06-08 12:48:20.709 7fadabaa4700 5 -- 10.3.56.69:6800/1807239
> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
> >> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH
> pgs=25741 cs=1 l=1). rx mon.0 seq 13 0x559f79abaa00 osd_map(378086..378125
> src has 378085..378738) v4
> -6> 2018-06-08 12:48:20.709 7fada058e700 -1 osd.58 0 failed to load
> OSD map for epoch 378084, got 0 bytes
> -5> 2018-06-08 12:48:20.710 7fadac2a5700 1 -- 10.3.56.69:6800/1807239
> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
> cs=1 l=1).read_bulk peer close file descriptor 38
> -4> 2018-06-08 12:48:20.710 7fadac2a5700 1 -- 10.3.56.69:6800/1807239
> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
> cs=1 l=1).read_until read failed
> -3> 2018-06-08 12:48:20.710 7fadac2a5700 1 -- 10.3.56.69:6800/1807239
> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
> cs=1 l=1).process read tag failed
> -2> 2018-06-08 12:48:20.710 7fadac2a5700 1 -- 10.3.56.69:6800/1807239
> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
> cs=1 l=1).fault on lossy channel, failing
> -1> 2018-06-08 12:48:20.710 7fadac2a5700 2 -- 10.3.56.69:6800/1807239
> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
> cs=1 l=1)._stop
> 0> 2018-06-08 12:48:20.711 7fada058e700 -1 /home/jenkins-build/build/
> workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/
> AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/
> release/13.2.0/rpm/el7/BUILD/ceph-13.2.0/src/osd/OSD.h: In function
> 'OSDMapRef OSDService::get_map(epoch_t)' thread 7fada058e700 time
> 2018-06-08 12:48:20.710675
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_
> 64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/
> centos7/MACHINE_SIZE/huge/release/13.2.0/rpm/el7/BUILD/ceph-13.2.0/src/osd/OSD.h:
> 828: FAILED assert(ret)
>
> ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic
> (stable)
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0xff) [0x7fadb3e1753f]
> 2: (()+0x286727) [0x7fadb3e17727]
> 3: (OSDService::get_map(unsigned int)+0x4a) [0x559f76fe4dda]
> 4: (OSD::handle_osd_map(MOSDMap*)+0x1020) [0x559f76f921f0]
> 5: (OSD::_dispatch(Message*)+0xa1) [0x559f76f94d21]
> 6: (OSD::ms_dispatch(Message*)+0x56) [0x559f76f95066]
> 7: (DispatchQueue::entry()+0xb5a) [0x7fadb3e8d74a]
> 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fadb3f2df2d]
> 9: (()+0x7e25) [0x7fadb0afde25]
> 10: (clone()+0x6d) [0x7fadafbf134d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
> to interpret this.
>
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 1/ 5 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 rbd_mirror
> 0/ 5 rbd_replay
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 1/ 5 osd
> 0/ 5 optracker
> 0/ 5 objclass
> 1/ 3 filestore
> 1/ 3 journal
> 0/ 5 ms
> 1/ 5 mon
> 0/10 monc
> 1/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 1 reserver
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/ 5 rgw_sync
> 1/10 civetweb
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> 0/ 0 refs
> 1/ 5 xio
> 1/ 5 compressor
> 1/ 5 bluestore
> 1/ 5 bluefs
> 1/ 3 bdev
> 1/ 5 kstore
> 4/ 5 rocksdb
> 4/ 5 leveldb
> 4/ 5 memdb
> 1/ 5 kinetic
> 1/ 5 fuse
> 1/ 5 mgr
> 1/ 5 mgrc
> 1/ 5 dpdk
> 1/ 5 eventtrace
> -2/-2 (syslog threshold)
> -1/-1 (stderr threshold)
> max_recent 10000
> max_new 1000
> log_file /var/log/ceph/ceph-osd.58.log
> --- end dump of recent events ---
> 2018-06-08 12:48:20.717 7fada058e700 -1 *** Caught signal (Aborted) **
> in thread 7fada058e700 thread_name:ms_dispatch
>
> ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic
> (stable)
> 1: (()+0x8e1870) [0x559f774af870]
> 2: (()+0xf5e0) [0x7fadb0b055e0]
> 3: (gsignal()+0x37) [0x7fadafb2e1f7]
> 4: (abort()+0x148) [0x7fadafb2f8e8]
> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x25d) [0x7fadb3e1769d]
> 6: (()+0x286727) [0x7fadb3e17727]
> 7: (OSDService::get_map(unsigned int)+0x4a) [0x559f76fe4dda]
> 8: (OSD::handle_osd_map(MOSDMap*)+0x1020) [0x559f76f921f0]
> 9: (OSD::_dispatch(Message*)+0xa1) [0x559f76f94d21]
> 10: (OSD::ms_dispatch(Message*)+0x56) [0x559f76f95066]
> 11: (DispatchQueue::entry()+0xb5a) [0x7fadb3e8d74a]
> 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fadb3f2df2d]
> 13: (()+0x7e25) [0x7fadb0afde25]
> 14: (clone()+0x6d) [0x7fadafbf134d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
> to interpret this.
>
> --- begin dump of recent events ---
> 0> 2018-06-08 12:48:20.717 7fada058e700 -1 *** Caught signal
> (Aborted) **
> in thread 7fada058e700 thread_name:ms_dispatch
>
> ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic
> (stable)
> 1: (()+0x8e1870) [0x559f774af870]
> 2: (()+0xf5e0) [0x7fadb0b055e0]
> 3: (gsignal()+0x37) [0x7fadafb2e1f7]
> 4: (abort()+0x148) [0x7fadafb2f8e8]
> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x25d) [0x7fadb3e1769d]
> 6: (()+0x286727) [0x7fadb3e17727]
> 7: (OSDService::get_map(unsigned int)+0x4a) [0x559f76fe4dda]
> 8: (OSD::handle_osd_map(MOSDMap*)+0x1020) [0x559f76f921f0]
> 9: (OSD::_dispatch(Message*)+0xa1) [0x559f76f94d21]
> 10: (OSD::ms_dispatch(Message*)+0x56) [0x559f76f95066]
> 11: (DispatchQueue::entry()+0xb5a) [0x7fadb3e8d74a]
> 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fadb3f2df2d]
> 13: (()+0x7e25) [0x7fadb0afde25]
> 14: (clone()+0x6d) [0x7fadafbf134d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
> to interpret this.
>
>
> Mike Kuriger
>
> >> CEPH team,
> >> Is there a solution yet for adding OSDs in mimic - specifically
> re-using old IDs? I was looking over this BUG report -
> https://urldefense.proofpoint.com/v2/url?u=https-3A__
> tracker.ceph.com_issues_24423&d=DwIFaQ&c=5m9CfXHY6NXqkS7nN5n23w&r=
> 5r9bhr1JAPRaUcJcU-FfGg&m=0PCKiecm216R95S_krqboYMskCBoolGysrvgHZo8LEM&s=
> hfI2uudTfY0lGtBI6iIXvZWvNpme4xwBJe2SWx0_N3I&e= and my issue is similar.
> I removed a bunch of OSD's after upgrading to mimic and I'm not able to
> re-add them using the new volume format. I haven't tried manually adding
> them using 'never used' IDs. I'll try that now but was hoping there would
> be a fix.
> >>
> >> Thanks!
> >>
> >> Mike Kuriger
> >>
> >> _______________________________________________
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.
> ceph.com_listinfo.cgi_ceph-2Dusers-2Dceph.com&d=DwIFaQ&c=
> 5m9CfXHY6NXqkS7nN5n23w&r=5r9bhr1JAPRaUcJcU-FfGg&m=0PCKiecm216R95S_
> krqboYMskCBoolGysrvgHZo8LEM&s=2aoWc5hTz041_26Stz6zPtLiB5zGFw2GbX3TPjsvieI
> &e=
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
