Dear all,
as promised, here I am following up on this thread.
After a some days since the last message on this thread, most of the
insecure connections from clients "magically" disappeared and the latest
ones were from not-yet upgraded openstack nova-compute nodes.
Now that all nova nodes are completely up-to-date and rebooted at the
same ceph distribution level, we were able to remove the allow_insecure
from MONs and all things seems fine.
Thank you again for the insightful suggestions.
Il 19/05/2025 21:14, Eugen Block ha scritto:
Just a quick update: I set auth_allow_insecure_global_id_reclaim to
false because all the client sessions we had showed either new_ok or
reclaim_ok in global_id_status. No complaints so far. :-)
Zitat von Eugen Block <ebl...@nde.ag>:
The mon sessions dump also shows the global_id_status, this could
help identify the clients to focus on. I would recommend to inspect
this output:
ceph tell mon.<MON> sessions | jq -r '.[] |
.name,.socket_addr.addr,.global_id_status'
and check each client.
While looking at our own mon sessions, it seems like I could probably
disable auth_allow_insecure_global_id_reclaim, looks like all older
clients are either new_ok or reclaim_ok. And we still have plenty of
"con_features_release" entries with "luminous", just FYI. I don't
really recall doing anything special with our clients (also OpenStack
servers), tbh. We upgraded them along with the Ceph versions, and now
it looks okay. Maybe a reboot after an upgrade suffices, I don't know.
Here's the list of possible global_id_status entries with a short
comment:
https://github.com/ceph/ceph/blob/df7b75547aab3bb055fff16588cd697e593dc19a/src/auth/AuthServiceHandler.h#L28
Zitat von Sergio Rabellino <rabell...@di.unito.it>:
Probably yes, but the list it's too long, so in the middle there's a
"..." : that's why I searched for a ceph command able to return all
the results for session properties without ellipsing.
For this purpose, in my search I found the command "ceph daemon
mon-name sessions" were I saw the "luminous" word that in my mind
was "wrong" from my top post of this thread.
Il 16/05/2025 14:56, Eugen Block ha scritto:
If auth_expose_insecure_global_id_reclaim is true, do you see the
client in 'ceph health detail'?
Zitat von Sergio Rabellino <rabell...@di.unito.it>:
this is the output by now that allows normal operation. Changing
the third breaks client access (and maybe other bad behaviour)
root@juju-XXX7ca-YY62-lxd-0:~# for i in $(ceph config ls | grep
global_id); do echo "Config: $i"; ceph config get mon
$i; done
Config: mon_warn_on_insecure_global_id_reclaim
true
Config: mon_warn_on_insecure_global_id_reclaim_allowed
true
Config: auth_allow_insecure_global_id_reclaim
true
Config: auth_expose_insecure_global_id_reclaim
true
We could disable the warnings, but is this affordable ?
Il 16/05/2025 13:45, Eugen Block ha scritto:
It's been a while since I dealt with all that global_id stuff:
https://docs.ceph.com/en/octopus/rados/operations/health-checks/#auth-insecure-global-id-reclaim
Can you share this output?
for i in $(ceph config ls | grep global_id); do echo "Config:
$i"; ceph config get mon $i; done
Config: mon_warn_on_insecure_global_id_reclaim
true
Config: mon_warn_on_insecure_global_id_reclaim_allowed
false
Config: auth_allow_insecure_global_id_reclaim
true
Config: auth_expose_insecure_global_id_reclaim
true
We still allow older clients, but we're aware and don't want the
warning.
Zitat von Sergio Rabellino <rabell...@di.unito.it>:
Sorry to bother the list with old questions again, I'm a little
new into this list.
But if 'ceph daemon mon.name sessions' does not reports the
actual protocol used, how I can find this info ?
Both the configs you asked for are "true": are you saying that
I'm concerned on null problem ?
If so (and it may be), why if we remove the allow of
insecure_global_id on mon(s) all the VMs stop talking with the
underlying volumes ?
Bests.
Il 15/05/2025 11:39, Eugen Block ha scritto:
The output of 'ceph features' (or the mon session output) does
not reflect that it's an older client. It just represents a
collection of supported features. This has been discussed
multiple times on this list. This is not unusual.
Regarding the insecure global_id, does your cluster warn about
it? Can you share these configs?
ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed
ceph config get mon mon_warn_on_insecure_global_id_reclaim
Zitat von Sergio Rabellino <rabell...@di.unito.it>:
Thanks for your prompt reply, dump follows:
# ceph osd dump |grep require_osd_release
require_osd_release octopus
We are aware of this setting and change it on release change
(we did in the last days mimic->nautilus->octopus).
The very strange thing it's that we upgraded luminous->mimic
more than two years ago.
About librados, I get:
# rados -v
ceph version 15.2.17
(8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)
so it seems fine to me.
Il 15/05/2025 11:01, Sinan Polat ha scritto:
What is the output of:
ceph osd dump | grep require_osd_release
Have you upgraded OpenStack (librados) as well?
Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino
<rabell...@di.unito.it <mailto:rabell...@di.unito.it>>:
Dear list,
we are upgrading our ceph infrastructure from mimic to
octopus
(please
be kind, we known that we are working with "old" tools,
but these
ceph
releases are tied to our openstack installation needs) and
_*all*_
the
ceph actors (mon/mgr/osd/rgw - no mds as we do not serve
filesystem)
were upgraded fine as follow:
> # ceph -v
> ceph version 15.2.17
(8a82819d84cf884bd39c17e3236e0632ac146dc4)
> octopus (stable)
We're using a ubuntu/juju orchestrator for managing ceph (and
openstack
too).
It seems all ok, but if I ask a mon to dump the sessions,
we get
that_all of them_ are on luminous:
> "con_features": 4540138292840890367,
> "con_features_hex": "3f01cfb8ffedffff",
> "con_features_release": "luminous",
We found this oddity when we tried to unset the "insecure
global_id
reclaim" flag and all things got broken, so we had to
reactivate
the flag.
All ceph network layers are "closed", so we're not so
urged to remove
the flag, but we would like to understand if this is a known
problem or
an error done by us.
Any hints ?
thanks in advance.
-- ing. Sergio Rabellino
Università degli Studi di Torino
Dipartimento di Informatica
Tecnico di Ricerca
Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603
C.so Svizzera , 185 - 10149 - Torino
<http://www.di.unito.it>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-le...@ceph.io
<mailto:ceph-users-le...@ceph.io>
--
ing. Sergio Rabellino
Università degli Studi di Torino
Dipartimento di Informatica
Tecnico di Ricerca
Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603
C.so Svizzera , 185 - 10149 - Torino
<http://www.di.unito.it>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
--
ing. Sergio Rabellino
Università degli Studi di Torino
Dipartimento di Informatica
Tecnico di Ricerca
Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603
C.so Svizzera , 185 - 10149 - Torino
<http://www.di.unito.it>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
--
ing. Sergio Rabellino
Università degli Studi di Torino
Dipartimento di Informatica
Tecnico di Ricerca
Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603
C.so Svizzera , 185 - 10149 - Torino
<http://www.di.unito.it>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
--
ing. Sergio Rabellino
Università degli Studi di Torino
Dipartimento di Informatica
Tecnico di Ricerca
Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603
C.so Svizzera , 185 - 10149 - Torino
<http://www.di.unito.it>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
--
ing. Sergio Rabellino
Università degli Studi di Torino
Dipartimento di Informatica
Tecnico di Ricerca
Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603
C.so Svizzera , 185 - 10149 - Torino
<http://www.di.unito.it>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
