Hi Dan,
We ended up upgrading all mons+mgrs to 14.2.9 and the message stopped and the
PG stats reappeared, as expected. Marcello started the OSD restarts this
morning.
I think it would have been much less stressful to get the cluster onto 12.2.13
before the nautilus upgrade, and much easier to get help should anything go
wrong. Something to bear in mind for the future.
Cheers,
Tom
> -----Original Message-----
> From: Dan van der Ster <d...@vanderster.com>
> Sent: 18 May 2020 10:18
> To: Byrne, Thomas (STFC,RAL,SC) <tom.by...@stfc.ac.uk>
> Cc: ceph-users@ceph.io; Armand Pilon, Marcello (STFC,RAL,SC)
> <marcello.armand-pi...@stfc.ac.uk>
> Subject: Re: [ceph-users] Luminous to Nautilus mon upgrade oddity - failed to
> decode mgrstat state; luminous dev version? buffer::end_of_buffer
>
> Hi Tom,
>
> Did you get past this? It looks like the mon is confused how to decode because
> of your non-standard release.
> (So I imaging that running all 14.2.9 mons would get past it, but if you're
> being
> cautious this should be reproduceable on your test cluster).
>
> -- Dan
>
>
> On Wed, May 13, 2020 at 12:07 PM Thomas Byrne - UKRI STFC
> <tom.by...@stfc.ac.uk> wrote:
> >
> > Hi all,
> >
> > We're upgrading a cluster from luminous to nautilus. The monitors and
> managers are running a non-release version of luminous (12.2.12-642-
> g5ff3e8e) and we're upgrading them to 14.2.9.
> >
> > We've upgraded one monitor and it's happily in quorum as a peon.
> However, when a ceph status hits the nautilus mon it has trouble talking to
> the
> manager apparently, and it returns a status output with no pg stats and
> garbage usage numbers. From the mon log:
> >
> > 2020-05-13 10:41:43.121 7fa1e6fdf700 0 mon.ceph-mon5@4(peon) e25
> > handle_command mon_command({"prefix": "status"} v 0) v1
> > 2020-05-13 10:41:43.121 7fa1e6fdf700 0 log_channel(audit) log [DBG] :
> > from='client.? v1:130.246.x.x:0/3261311028' entity='client.admin'
> > cmd=[{"prefix": "status"}]: dispatch
> > 2020-05-13 10:41:43.443 7fa1e6fdf700 -1 mon.ceph-
> mon5@4(peon).mgrstat
> > failed to decode mgrstat state; luminous dev version?
> > buffer::end_of_buffer
> > 2020-05-13 10:41:44.397 7fa1e6fdf700 1 mon.ceph-mon5@4(peon) e25
> > dropping unexpected mon_health( e 0 r 0 ) v1
> >
> > Is this expected for a luminous to nautilus upgrade or could this be due to
> the odd luminous version we are running, or something else entirely?
> >
> > Cheers,
> > Tom
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> > email to ceph-users-le...@ceph.io
This email and any attachments are intended solely for the use of the named
recipients. If you are not the intended recipient you must not use, disclose,
copy or distribute this email or any of its attachments and should notify the
sender immediately and delete this email from your system. UK Research and
Innovation (UKRI) has taken every reasonable precaution to minimise risk of
this email or any attachments containing viruses or malware but the recipient
should carry out its own virus and malware checks before opening the
attachments. UKRI does not accept any liability for any losses or damages which
the recipient may sustain due to presence of any viruses. Opinions, conclusions
or other information in this message and attachments that are not related
directly to UKRI business are solely those of the author and do not represent
the views of UKRI.
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
