I think this is the relevant section of the debug log. There's no AUTH_NONE error which would make things easy. You can see the same "Invalid argument" error that I'm seeing in the mgr debug output. The malformed request feels like a compatibility or protocol communication issue.
2019-01-04 13:41:58.972 7f88950f5700 10 mon.07@1(peon) e27 ms_verify_authorizer 192.168.253.148:0/3301807723 client protocol 0 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon) e27 _ms_dispatch new session 0x40a58c0 MonSession(client.? 192.168.253.148:0/3301807723 is open , features 0x3ffddff8ffa4fffb (luminous)) fea$ures 0x3ffddff8ffa4fffb 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697 preprocess_query auth(proto 0 26 bytes epoch 0) v1 from client.? 192.168.253.148:0/3301807723 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697 prep_auth() blob_size=26 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697 AuthMonitor::assign_global_id m=auth(proto 0 26 bytes epoch 0) v1 mon=1/3 last_allocated=12307825 max_global_id=12353896 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697 next_global_id should be 12307828 2019-01-04 13:41:58.972 7f8890143700 2 mon.07@1(peon) e27 send_reply 0x5449180 0x4ee1c00 auth_reply(proto 2 0 (0) Success) v1 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697 preprocess_query auth(proto 2 2 bytes epoch 0) v1 from client.? 192.168.253.148:0/3301807723 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697 prep_auth() blob_size=2 2019-01-04 13:41:58.972 7f8890143700 0 mon.07@1(peon).auth v87697 caught error when trying to handle auth request, probably malformed request 2019-01-04 13:41:58.972 7f8890143700 2 mon.07@1(peon) e27 send_reply 0x30dc500 0x5caa280 auth_reply(proto 2 -22 (22) Invalid argument) v1 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon) e27 ms_handle_reset 0x4102a00 192.168.253.148:0/3301807723 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon) e27 reset/close on session client.? 192.168.253.148:0/3301807723 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon) e27 remove_session 0x40a58c0 client.? 192.168.253.148:0/3301807723 features 0x3ffddff8ffa4fffb On Fri, Jan 4, 2019 at 12:32 PM Gregory Farnum <gfar...@redhat.com> wrote: > You can also get more data by checking what the monitor logs for that > manager on the connect attempt (if you turn up its debug mon or debug > ms settings). If one of your managers is behaving, I'd examine its > configuration file and compare to the others. For instance, that > "Invalid argument" might mean the manager is trying to use "AUTH_NONE" > (no CephX) and the monitors aren't allowing that. > -Greg > > On Fri, Jan 4, 2019 at 6:26 AM Randall Smith <rbsm...@adams.edu> wrote: > > > > Greetings, > > > > I'm upgrading my cluster from luminous to mimic. I've upgraded my > monitors and am attempting to upgrade the mgrs. Unfortunately, after an > upgrade the mgr daemon exits immediately with error code 1. > > > > I've tried running ceph-mgr in debug mode to try to see what's happening > but the output (below) is a bit cryptic for me. It looks like > authentication might be failing but it was working prior to the upgrade. > > > > I do have "auth supported = cephx" in the global section of ceph.conf. > > > > What do I need to do to fix this? > > > > Thanks. > > > > /usr/bin/ceph-mgr -f --cluster ceph --id 8 --setuser ceph --setgroup > ceph -d --debug_ms 5 > > 2019-01-04 07:01:38.457 7f808f83f700 2 Event(0x30c42c0 nevent=5000 > time_id=1).set_owner idx=0 owner=140190140331776 > > 2019-01-04 07:01:38.457 7f808f03e700 2 Event(0x30c4500 nevent=5000 > time_id=1).set_owner idx=1 owner=140190131939072 > > 2019-01-04 07:01:38.457 7f808e83d700 2 Event(0x30c4e00 nevent=5000 > time_id=1).set_owner idx=2 owner=140190123546368 > > 2019-01-04 07:01:38.457 7f809dd5b380 1 Processor -- start > > 2019-01-04 07:01:38.477 7f809dd5b380 1 -- - start start > > 2019-01-04 07:01:38.481 7f809dd5b380 1 -- - --> 192.168.253.147:6789/0 > -- auth(proto 0 26 bytes epoch 0) v1 -- 0x32a6780 con 0 > > 2019-01-04 07:01:38.481 7f809dd5b380 1 -- - --> 192.168.253.148:6789/0 > -- auth(proto 0 26 bytes epoch 0) v1 -- 0x32a6a00 con 0 > > 2019-01-04 07:01:38.481 7f808e83d700 1 -- 192.168.253.148:0/1359135487 > learned_addr learned my addr 192.168.253.148:0/1359135487 > > 2019-01-04 07:01:38.481 7f808e83d700 2 -- 192.168.253.148:0/1359135487 > >> 192.168.253.148:6789/0 conn(0x332d500 :-1 > s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got > newly_a$ > > ked_seq 0 vs out_seq 0 > > 2019-01-04 07:01:38.481 7f808f03e700 2 -- 192.168.253.148:0/1359135487 > >> 192.168.253.147:6789/0 conn(0x332ce00 :-1 > s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got > newly_a$ > > ked_seq 0 vs out_seq 0 > > 2019-01-04 07:01:38.481 7f808f03e700 5 -- 192.168.253.148:0/1359135487 > >> 192.168.253.147:6789/0 conn(0x332ce00 :-1 > s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1 > seq > > 1 0x30c5440 mon_map magic: 0 v1 > > 2019-01-04 07:01:38.481 7f808e83d700 5 -- 192.168.253.148:0/1359135487 > >> 192.168.253.148:6789/0 conn(0x332d500 :-1 > s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2 > seq > > 1 0x30c5680 mon_map magic: 0 v1 > > 2019-01-04 07:01:38.481 7f808f03e700 5 -- 192.168.253.148:0/1359135487 > >> 192.168.253.147:6789/0 conn(0x332ce00 :-1 > s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1 > seq > > 2 0x32a6780 auth_reply(proto 2 0 (0) Success) v1 > > 2019-01-04 07:01:38.481 7f808e83d700 5 -- 192.168.253.148:0/1359135487 > >> 192.168.253.148:6789/0 conn(0x332d500 :-1 > s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2 > seq > > 2 0x32a6a00 auth_reply(proto 2 0 (0) Success) v1 > > 2019-01-04 07:01:38.481 7f808e03c700 1 -- 192.168.253.148:0/1359135487 > <== mon.1 192.168.253.147:6789/0 1 ==== mon_map magic: 0 v1 ==== 370+0+0 > (3034216899 0 0) 0x30c5440 con 0x332ce00 > > 2019-01-04 07:01:38.481 7f808e03c700 1 -- 192.168.253.148:0/1359135487 > <== mon.2 192.168.253.148:6789/0 1 ==== mon_map magic: 0 v1 ==== 370+0+0 > (3034216899 0 0) 0x30c5680 con 0x332d500 > > 2019-01-04 07:01:38.481 7f808e03c700 1 -- 192.168.253.148:0/1359135487 > <== mon.1 192.168.253.147:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) > v1 ==== 33+0+0 (3430158761 0 0) 0x32a6780 con 0x33$ > > ce00 > > 2019-01-04 07:01:38.481 7f808e03c700 1 -- 192.168.253.148:0/1359135487 > --> 192.168.253.147:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- > 0x32a6f00 con 0 > > 2019-01-04 07:01:38.481 7f808e03c700 1 -- 192.168.253.148:0/1359135487 > <== mon.2 192.168.253.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) > v1 ==== 33+0+0 (3242503871 0 0) 0x32a6a00 con 0x33$ > > d500 > > 2019-01-04 07:01:38.481 7f808e03c700 1 -- 192.168.253.148:0/1359135487 > --> 192.168.253.148:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- > 0x32a6780 con 0 > > 2019-01-04 07:01:38.481 7f808f03e700 5 -- 192.168.253.148:0/1359135487 > >> 192.168.253.147:6789/0 conn(0x332ce00 :-1 > s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1 > seq > > 3 0x32a6f00 auth_reply(proto 2 -22 (22) Invalid argument) v1 > > 2019-01-04 07:01:38.481 7f808e03c700 1 -- 192.168.253.148:0/1359135487 > <== mon.1 192.168.253.147:6789/0 3 ==== auth_reply(proto 2 -22 (22) > Invalid argument) v1 ==== 24+0+0 (882932531 0 0) 0x32a6f$ > > 0 con 0x332ce00 > > 2019-01-04 07:01:38.481 7f808e03c700 1 -- 192.168.253.148:0/1359135487 > >> 192.168.253.147:6789/0 conn(0x332ce00 :-1 s=STATE_OPEN pgs=74172 cs=1 > l=1).mark_down > > 2019-01-04 07:01:38.481 7f808e03c700 2 -- 192.168.253.148:0/1359135487 > >> 192.168.253.147:6789/0 conn(0x332ce00 :-1 s=STATE_OPEN pgs=74172 cs=1 > l=1)._stop > > 2019-01-04 07:01:38.481 7f808e83d700 5 -- 192.168.253.148:0/1359135487 > >> 192.168.253.148:6789/0 conn(0x332d500 :-1 > s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2 > seq > > 3 0x32a6780 auth_reply(proto 2 -22 (22) Invalid argument) v1 > > 2019-01-04 07:01:38.481 7f808e03c700 1 -- 192.168.253.148:0/1359135487 > <== mon.2 192.168.253.148:6789/0 3 ==== auth_reply(proto 2 -22 (22) > Invalid argument) v1 ==== 24+0+0 (1359424806 0 0) 0x32a6$ > > 80 con 0x332d500 > > 2019-01-04 07:01:38.481 7f808e03c700 1 -- 192.168.253.148:0/1359135487 > >> 192.168.253.148:6789/0 conn(0x332d500 :-1 s=STATE_OPEN pgs=74275 cs=1 > l=1).mark_down > > 2019-01-04 07:01:38.481 7f808e03c700 2 -- 192.168.253.148:0/1359135487 > >> 192.168.253.148:6789/0 conn(0x332d500 :-1 s=STATE_OPEN pgs=74275 cs=1 > l=1)._stop > > > > 2019-01-04 07:01:38.481 7f809dd5b380 1 -- 192.168.253.148:0/1359135487 > shutdown_connections > > 2019-01-04 07:01:38.481 7f809dd5b380 5 -- 192.168.253.148:0/1359135487 > shutdown_connections mark down 192.168.253.148:6789/0 0x332d500 > > 2019-01-04 07:01:38.481 7f809dd5b380 5 -- 192.168.253.148:0/1359135487 > shutdown_connections mark down 192.168.253.147:6789/0 0x332ce00 > > 2019-01-04 07:01:38.481 7f809dd5b380 5 -- 192.168.253.148:0/1359135487 > shutdown_connections delete 0x332ce00 > > 2019-01-04 07:01:38.481 7f809dd5b380 5 -- 192.168.253.148:0/1359135487 > shutdown_connections delete 0x332d500 > > 2019-01-04 07:01:38.485 7f809dd5b380 1 -- 192.168.253.148:0/1359135487 > shutdown_connections > > 2019-01-04 07:01:38.485 7f809dd5b380 1 -- 192.168.253.148:0/1359135487 > wait complete. > > 2019-01-04 07:01:38.485 7f809dd5b380 1 -- 192.168.253.148:0/1359135487 > >> 192.168.253.148:0/1359135487 conn(0x332c000 :-1 s=STATE_NONE pgs=0 > cs=0 l=0).mark_down > > 2019-01-04 07:01:38.485 7f809dd5b380 2 -- 192.168.253.148:0/1359135487 > >> 192.168.253.148:0/1359135487 conn(0x332c000 :-1 s=STATE_NONE pgs=0 > cs=0 l=0)._stop > > failed to fetch mon config (--no-mon-config to skip) > > > > -- > > Randall Smith > > Computing Services > > Adams State University > > http://www.adams.edu/ > > 719-587-7741 > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Randall Smith Computing Services Adams State University http://www.adams.edu/ 719-587-7741
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
