On Saturday, Apr 05, 2025 at 1:27 PM, Eugen Block <ebl...@nde.ag
(mailto:ebl...@nde.ag)> wrote:
I don't see the module error in the logs you provided. Has it cleared?
A mgr failover is often helpful in such cases, basically it's the
first thing I suggest since two or three years.
Zitat von Jeremy Hansen <jer...@skidrow.la>:
> Thank you so much for the detailed instructions. Here’s logs from
> the failover to a new node.
>
> Apr 05 20:06:08 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:06:08] "GET /metrics HTTP/1.1" 200 -
""
> "Prometheus/2.33.4"
> Apr 05 20:06:08 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:08.568+0000 7efcbd800700 0 [prometheus INFO cherrypy.access.139625005087208] ::ffff:192.168.47.72 - - [05/Apr/2025:20:06:08] "GET /metrics HTTP/1.1" 200 -
""
> "Prometheus/2.33.4"
> Apr 05 20:06:10 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:06:10] "GET /metrics HTTP/1.1" 200 -
""
> "Prometheus/2.33.4"
> Apr 05 20:06:10 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:10.702+0000 7efcb4c00700 0 [prometheus INFO cherrypy.access.139625005087208] ::ffff:192.168.47.71 - - [05/Apr/2025:20:06:10] "GET /metrics HTTP/1.1" 200 -
""
> "Prometheus/2.33.4"
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.027+0000 7efd95200700 1 mgr
handle_mgr_map
> Activating!
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.028+0000 7efd95200700 1 mgr handle_mgr_map I am
now
> activating
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.040+0000 7efc27e00700 0 [dashboard INFO root]
Stopping
> engine...
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.040+0000 7efc27e00700 0 [dashboard INFO root]
Stopped
> engine...
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc27e00700 0 [prometheus INFO root]
Stopping
> engine...
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc27e00700 0 [prometheus INFO root]
Stopped
> engine
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE
Bus
> STOPPING
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE
Bus
> STOPPING
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.053+0000 7efc3a400700 0 [dashboard INFO root]
Engine
> stopped.
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283))
shut
> down
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283))
shut
> down
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE
Bus
> STOPPED
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE
Bus
> STOPPED
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO root]
Engine
> stopped.
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.056+0000 7efc27e00700 0 [alerts DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.057+0000 7efc27e00700 0 [alerts INFO
root]
> Init
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.057+0000 7efc27e00700 1 mgr load Constructed class from
module:
> alerts
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efc27e00700 0 [balancer DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efc27e00700 1 mgr load Constructed class from
module:
> balancer
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efcfac00700 0 [alerts INFO
root]
> Starting
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.060+0000 7efc27e00700 0 [cephadm DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.060+0000 7efcb4c00700 0 [balancer INFO
root]
> Starting
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Optimize
plan
> auto_2025-04-05_20:06:12
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Mode upmap, max
misplaced
> 0.050000
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try
again
> later
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.212+0000 7efc27e00700 1 mgr load Constructed class from
module:
> cephadm
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.213+0000 7efc27e00700 0 [crash DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.213+0000 7efc27e00700 1 mgr load Constructed class from
module:
> crash
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.215+0000 7efc27e00700 0 [dashboard DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.215+0000 7efc27e00700 1 mgr load Constructed class from
module:
> dashboard
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.216+0000 7efc23800700 0 [dashboard INFO access_control] Loading user roles
DB
> version=2
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.217+0000 7efc23800700 0 [dashboard INFO sso] Loading SSO
DB
> version=1
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.217+0000 7efc23800700 0 [dashboard INFO root] server: ssl=yes
host=::
> port=8443
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.218+0000 7efc27e00700 0 [devicehealth DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.218+0000 7efc27e00700 1 mgr load Constructed class from
module:
> devicehealth
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc27e00700 0 [insights DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc27e00700 1 mgr load Constructed class from
module:
> insights
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc20600700 0 [insights INFO root] Reset curr health slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:06:12 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc21a00700 0 [devicehealth INFO
root]
> Starting
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.222+0000 7efc27e00700 0 [iostat DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.222+0000 7efc27e00700 1 mgr load Constructed class from
module:
> iostat
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.224+0000 7efc27e00700 0 [mirroring DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.227+0000 7efc21a00700 0 [devicehealth INFO root]
Check
> health
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.235+0000 7efc27e00700 1 mgr load Constructed class from
module:
> mirroring
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.236+0000 7efc27e00700 0 [nfs DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.236+0000 7efc27e00700 1 mgr load Constructed class from
module:
> nfs
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.237+0000 7efc27e00700 0 [orchestrator DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.237+0000 7efc27e00700 1 mgr load Constructed class from
module:
> orchestrator
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc27e00700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc27e00700 1 mgr load Constructed class from
module:
> pg_autoscaler
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc10200700 0 [pg_autoscaler INFO
root]
> _maybe_adjust
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.240+0000 7efc27e00700 0 [progress DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.240+0000 7efc27e00700 1 mgr load Constructed class from
module:
> progress
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.247+0000 7efc27e00700 0 [prometheus DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.248+0000 7efc27e00700 1 mgr load Constructed class from
module:
> prometheus
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.252+0000 7efc0ee00700 0 [progress INFO
root]
> Loading...
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.252+0000 7efc27e00700 0 [rbd_support DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7efcf2127710>, <progress.module.GhostEvent object at 0x7efcf2127cc0>, <progress.module.GhostEvent object at 0x7efcf2127208>, <progress.module.GhostEvent object at 0x7efcf2127f28>, <progress.module.GhostEvent object at 0x7efcf21270f0>, <progress.module.GhostEvent object at 0x7efcf2127eb8>, <progress.module.GhostEvent object at 0x7efcf2127b38>, <progress.module.GhostEvent object at 0x7efcf2127a20>, <progress.module.GhostEvent object at 0x7efcf2127da0>, <progress.module.GhostEvent object at 0x7efcf2127f98>, <progress.module.GhostEvent object at 0x7efcf2127780>, <progress.module.GhostEvent object at 0x7efcf2127320>, <progress.module.GhostEvent object at 0x7efcf2127588>, <progress.module.GhostEvent object at 0x7efcf2127518>, <progress.module.GhostEvent object at 0x7efcf2127c50>, <progress.module.GhostEvent object at 0x7efcf21279e8>, <progress.module.GhostEvent object at 0x7efcf2127d68>, <progress.module.GhostEvent object at 0x7efcf2127470>, <progress.module.GhostEvent object at 0x7efcf2127940>, <progress.module.GhostEvent object at 0x7efcf2127048>, <progress.module.GhostEvent object at 0x7efcf2127390>, <progress.module.GhostEvent object at 0x7efcf21276d8>, <progress.module.GhostEvent object at 0x7efcf2127a90>, <progress.module.GhostEvent object at 0x7efcf2127ba8>, <progress.module.GhostEvent object at 0x7efcf2127e48>, <progress.module.GhostEvent object at 0x7efcf21272e8>, <progress.module.GhostEvent object at 0x7efcf2127080>, <progress.module.GhostEvent object at 0x7efcf2127b00>, <progress.module.GhostEvent object at 0x7efcf21279b0>, <progress.module.GhostEvent object at 0x7efcf20c1588>, <progress.module.GhostEvent object at 0x7efcf20c1ac8>, <progress.module.GhostEvent object at 0x7efcf20c1550>, <progress.module.GhostEvent object at 0x7efcf20c17f0>, <progress.module.GhostEvent object at 0x7efcf20c1d68>, <progress.module.GhostEvent object at 0x7efcf20c1b70>, <progress.module.GhostEvent object at 0x7efcf20c1c50>, <progress.module.GhostEvent object at 0x7efcf20c1860>, <progress.module.GhostEvent object at 0x7efcf20c1908>, <progress.module.GhostEvent object at 0x7efcf20c14e0>, <progress.module.GhostEvent object at 0x7efcf20c1cf8>, <progress.module.GhostEvent object at 0x7efcf20c1470>, <progress.module.GhostEvent object at 0x7efcf20c1d30>, <progress.module.GhostEvent object at 0x7efcf20c1e80>, <progress.module.GhostEvent object at 0x7efcf20c1e10>, <progress.module.GhostEvent object at 0x7efcf20c1eb8>, <progress.module.GhostEvent object at 0x7efcf20c1f28>, <progress.module.GhostEvent object at 0x7efcf20c1668>, <progress.module.GhostEvent object at 0x7efcf20c16a0>, <progress.module.GhostEvent object at 0x7efcf20c1ba8>, <progress.module.GhostEvent object at 0x7efcf20c1c18>]
historic
> events
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] Loaded
OSDMap,
> ready.
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] Processing OSDMap
change
> 27179..27179
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.256+0000 7efc0d600700 0 [prometheus INFO root] server_addr: ::
server_port:
> 9283
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.256+0000 7efc0d600700 0 [prometheus INFO root]
Cache
> enabled
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.256+0000 7efc08600700 0 [prometheus INFO root] starting metric
collection
> thread
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.258+0000 7efc0d600700 0 [prometheus INFO root]
Starting
> engine...
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE
Bus
> STARTING
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.258+0000 7efc0d600700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE
Bus
> STARTING
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.259+0000 7efc07200700 0 [rbd_support INFO root] recovery
thread
> starting
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.259+0000 7efc07200700 0 [rbd_support INFO root]
starting
> setup
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.259+0000 7efc27e00700 1 mgr load Constructed class from
module:
> rbd_support
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.301+0000 7efc27e00700 0 [restful DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.301+0000 7efc27e00700 1 mgr load Constructed class from
module:
> restful
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.303+0000 7efc07200700 0 [rbd_support INFO root]
MirrorSnapshotScheduleHandler:
> load_schedules
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.305+0000 7efc07200700 0 [rbd_support INFO root] load_schedules:
csxyz,
> start_after=
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.307+0000 7efc04a00700 0 [restful INFO root] server_addr: ::
server_port:
> 8003
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.310+0000 7efc27e00700 0 [snap_schedule DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.312+0000 7efc04a00700 0 [restful WARNING root] server not running: no
certificate
> configured
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.73' (ECDSA) to the list of
known
> hosts.
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.319+0000 7efc23800700 0 [dashboard INFO root] Configured CherryPy,
starting
> engine...
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.72' (ECDSA) to the list of
known
> hosts.
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.71' (ECDSA) to the list of
known
> hosts.
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.361+0000 7efbf8200700 0 [rbd_support INFO root]
MirrorSnapshotScheduleHandler:
> starting
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.369+0000 7efbf7800700 0 [rbd_support INFO root]
PerfHandler:
> starting
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.371+0000 7efc07200700 0 [rbd_support INFO root] load_task_task:
csxyz,
> start_after=
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.389+0000 7efc27e00700 1 mgr load Constructed class from
module:
> snap_schedule
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.390+0000 7efc27e00700 0 [status DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.390+0000 7efc27e00700 1 mgr load Constructed class from
module:
> status
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.391+0000 7efc27e00700 0 [telemetry DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.392+0000 7efc27e00700 1 mgr load Constructed class from
module:
> telemetry
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.393+0000 7efc27e00700 0 [volumes DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5)
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.409+0000 7efc27e00700 0 [volumes INFO volumes.fs.async_job] queuing job for
volume
> 'cssnfsxyz'
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.410+0000 7efc27e00700 0 [volumes INFO volumes.fs.async_job] queuing job for
volume
> 'cssnfsxyz'
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.410+0000 7efc27e00700 1 mgr load Constructed class from
module:
> volumes
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]:
CherryPy
> Checker:
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an
empty
> config.
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]:
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE Serving
on
> http://:::9283
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.413+0000 7efc0d600700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Serving
on
> http://:::9283
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE
Bus
> STARTED
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.413+0000 7efc0d600700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE
Bus
> STARTED
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.413+0000 7efc0d600700 0 [prometheus INFO root]
Engine
> started.
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17)
File
> exists
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17)
File
> exists
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17)
File
> exists
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17)
File
> exists
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17)
File
> exists
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17)
File
> exists
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17)
File
> exists
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17)
File
> exists
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17)
File
> exists
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17)
File
> exists
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.431+0000 7efbd8400700 0 [rbd_support INFO root]
TaskHandler:
> starting
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.432+0000 7efc07200700 0 [rbd_support INFO root]
TrashPurgeScheduleHandler:
> load_schedules
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.434+0000 7efc07200700 0 [rbd_support INFO root] load_schedules:
csxyz,
> start_after=
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.445+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFS
->
> /api/cephfs
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.448+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFsUi
->
> /ui-api/cephfs
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.450+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiUi
->
> /ui-api/iscsi
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.450+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Iscsi
->
> /api/iscsi
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.450+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiTarget
->
> /api/iscsi/target
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.451+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaCluster
->
> /api/nfs-ganesha/cluster
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.452+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaExports
->
> /api/nfs-ganesha/export
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.453+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaUi
->
> /ui-api/nfs-ganesha
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.453+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rbd
->
> /api/block/image
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.455+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdStatus
->
> /ui-api/block/rbd
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.455+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdSnapshot
->
> /api/block/image/{image_spec}/snap
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.456+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdTrash
->
> /api/block/image/trash
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.457+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdNamespace
->
> /api/block/pool/{pool_name}/namespace
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.458+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Pool
->
> /api/pool
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.460+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PoolUi
->
> /ui-api/pool
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.461+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RBDPool
->
> /api/pool
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.462+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Orchestrator
->
> /ui-api/orchestrator
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.462+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Service
->
> /api/service
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.463+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroring
->
> /api/block/mirroring
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.463+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringSummary
->
> /api/block/mirroring/summary
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.464+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolMode
->
> /api/block/mirroring/pool
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.464+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolBootstrap
->
> /api/block/mirroring/pool/{pool_name}/bootstrap
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.464+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolPeer
->
> /api/block/mirroring/pool/{pool_name}/peer
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.465+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringStatus
->
> /ui-api/block/mirroring
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.466+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rgw
->
> /ui-api/rgw
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.466+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwDaemon
->
> /api/rgw/daemon
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.466+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwSite
->
> /api/rgw/site
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.467+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwBucket
->
> /api/rgw/bucket
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.468+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwUser
->
> /api/rgw/user
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.472+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Auth
->
> /api/auth
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.472+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Cluster
->
> /api/cluster
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.473+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ClusterConfiguration
->
> /api/cluster_conf
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.474+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRule
->
> /api/crush_rule
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.476+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRuleUi
->
> /ui-api/crush_rule
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.476+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Daemon
->
> /api/daemon
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.477+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Docs
->
> /docs
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.477+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfile
->
> /api/erasure_code_profile
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.478+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfileUi
->
> /ui-api/erasure_code_profile
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.478+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FrontendLogging
->
> /ui-api/logging
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.478+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Grafana
->
> /api/grafana
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.479+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Host
->
> /api/host
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.480+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HostUi
->
> /ui-api/host
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.481+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Health
->
> /api/health
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.481+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HomeController
->
> /
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.481+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LangsController
->
> /ui-api/langs
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.482+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LoginController
->
> /ui-api/login
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.482+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Logs
->
> /api/logs
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.482+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrModules
->
> /api/mgr/module
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.483+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Monitor
->
> /api/monitor
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.484+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Osd
->
> /api/osd
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.486+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdUi
->
> /ui-api/osd
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.487+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdFlagsController
->
> /api/osd/flags
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.488+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MdsPerfCounter
->
> /api/perf_counters/mds
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.488+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MonPerfCounter
->
> /api/perf_counters/mon
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.489+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdPerfCounter
->
> /api/perf_counters/osd
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.489+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwPerfCounter
->
> /api/perf_counters/rgw
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.490+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirrorPerfCounter
->
> /api/perf_counters/rbd-mirror
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.490+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrPerfCounter
->
> /api/perf_counters/mgr
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.491+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: TcmuRunnerPerfCounter
->
> /api/perf_counters/tcmu-runner
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.491+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PerfCounters
->
> /api/perf_counters
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.491+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusReceiver
->
> /api/prometheus_receiver
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.492+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Prometheus
->
> /api/prometheus
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.493+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusNotifications
->
> /api/prometheus/notifications
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.493+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Role
->
> /api/role
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.494+0000 7efbd7a00700 0 [rbd_support INFO root]
TrashPurgeScheduleHandler:
> starting
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.495+0000 7efc07200700 0 [rbd_support INFO root]
setup
> complete
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.495+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Scope
->
> /ui-api/scope
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.496+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Saml2
->
> /auth/saml2
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.496+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Settings
->
> /api/settings
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.497+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: StandardSettings
->
> /ui-api/standard_settings
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.498+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Summary
->
> /api/summary
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.498+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Task
->
> /api/task
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.498+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Telemetry
->
> /api/telemetry
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.499+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: User
->
> /api/user
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.500+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserPasswordPolicy
->
> /api/user
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.500+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserChangePassword
->
> /api/user/{username}
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.501+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FeatureTogglesEndpoint
->
> /api/feature_toggles
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.501+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MessageOfTheDay
->
> /ui-api/motd
> Apr 05 20:06:12 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.710+0000 7efc23800700 0 [dashboard INFO dashboard.module]
Engine
> started.
> Apr 05 20:06:13 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:13.039+0000 7efc27400700 0 log_channel(cluster) log [DBG] : pgmap v3: 609 pgs: 609 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109
TiB
> avail
> Apr 05 20:06:13 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:06:13] "GET /metrics HTTP/1.1" 200 39714
""
> "Prometheus/2.33.4"
> Apr 05 20:06:13 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:13.287+0000 7efc02200700 0 [prometheus INFO cherrypy.access.139624858224400] ::ffff:192.168.47.73 - - [05/Apr/2025:20:06:13] "GET /metrics HTTP/1.1" 200 39714
""
> "Prometheus/2.33.4"
> Apr 05 20:06:14 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:14.037+0000 7efc26000700 0 log_channel(cluster) log [DBG] : pgmap v4: 609 pgs: 609 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109
TiB
> avail
> Apr 05 20:06:16 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:16.038+0000 7efc26000700 0 log_channel(cluster) log [DBG] : pgmap v5: 609 pgs: 609 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109
TiB
> avail
> Apr 05 20:06:17 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.106+0000 7efc1fc00700 0 [insights INFO root] Received health check update >
pending
> Apr 05 20:06:17 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.149+0000 7efc20600700 0 [insights INFO root] Applying 1 health updates to slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:06:17 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.149+0000 7efc20600700 0 [insights INFO root] Maybe flushing slot key health_history/2025-04-05_20 next flush None checks check count 0
needed
> False
> Apr 05 20:06:17 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.256+0000 7efc0ee00700 0 [progress INFO root] Processing OSDMap
change
> 27179..27179
> Apr 05 20:06:17 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.518+0000 7efce9400700 0 [cephadm INFO root] Detected new or changed devices
on
> cn01.ceph.xyz.corp
> Apr 05 20:06:17 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.518+0000 7efce9400700 0 log_channel(cephadm) log [INF] : Detected new or changed devices
on
> cn01.ceph.xyz.corp
> Apr 05 20:06:17 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.611+0000 7efcd7c00700 0 [cephadm INFO root] Detected new or changed devices
on
> cn03.ceph.xyz.corp
> Apr 05 20:06:17 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.611+0000 7efcd7c00700 0 log_channel(cephadm) log [INF] : Detected new or changed devices
on
> cn03.ceph.xyz.corp
> Apr 05 20:06:17 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.726+0000 7efccf000700 0 [cephadm INFO root] Detected new or changed devices
on
> cn02.ceph.xyz.corp
> Apr 05 20:06:17 cn02.ceph.xyz.corp
>
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.726+0000 7efccf000700 0 log_channel(cephadm) log [INF] : Detected new or changed devices
on
> cn02.ceph.xyz.corp
>
> > On Saturday, Apr 05, 2025 at 4:51 AM, Eugen Block <ebl...@nde.ag
> > (mailto:ebl...@nde.ag)> wrote:
> > Please don't drop the list from your response.
> >
> > The orchestrator might be down, but you should have an active mgr
> > (ceph -s). On the host running the mgr, you can just run 'cephadm logs
> > --name mgr.<your_mgr>', or 'journalctl -u
> > ceph-<ceph_fsid>@mgr.<your_mgr>'.
> > To get fresh logs, you can just run 'ceph mgr fail', look on which
> > host a different mgr takes over and gather logs from there.
> >
> > Zitat von Jeremy Hansen <jer...@skidrow.la>:
> >
> > > Thanks for the clarification on version jumps. I interpreted the
> > > docs to mean that I had to incrementally step through each release.
> > > This will save a lot of time.
> > >
> > > Since the orchestrator is down and I’m not sure how to trigger a
> > > failure, I don’t know how to gather those logs. I apologize. I
> > > appreciate your assistance.
> > >
> > > -jeremy
> > >
> > > > On Saturday, Apr 05, 2025 at 4:01 AM, Eugen Block <ebl...@nde.ag
> > > > (mailto:ebl...@nde.ag)> wrote:
> > > > It might be a different issue, can you paste the entire stack trace
> > > > from the mgr when it's failing?
> > > >
> > > > Also, you could go directly from Pacific to Reef, there's no need to
> > > > upgrade to Quincy (which is EOL). And I would also recommend to
> > > > upgrade to a latest major version, e.g. 17.2.8, not .0,
otherwise you
> > > > might hit bugs which have already been fixed in later releases.
> > > >
> > > > Zitat von Jeremy Hansen <jer...@skidrow.la>:
> > > >
> > > > > I ran in to the “Error ENOENT: Module not found” issue with the
> > > > > orchestrator. I see the note in the cephadm upgrade docs
but I don’t
> > > > > quite know what action to take to fix this:
> > > > >
> > > > > ceph versions
> > > > > {
> > > > > "mon": {
> > > > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
> > > > > pacific (stable)": 3
> > > > > },
> > > > > "mgr": {
> > > > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
> > > > > pacific (stable)": 1,
> > > > > "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e)
> > > > > quincy (stable)": 2
> > > > > },
> > > > > "osd": {
> > > > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
> > > > > pacific (stable)": 15
> > > > > },
> > > > > "mds": {
> > > > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
> > > > > pacific (stable)": 3
> > > > > },
> > > > > "overall": {
> > > > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
> > > > > pacific (stable)": 22,
> > > > > "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e)
> > > > > quincy (stable)": 2
> > > > > }
> > > > > }
> > > > >
> > > > > ceph orch upgrade status
> > > > > Error ENOENT: Module not found
> > > > >
> > > > > ceph config-key get mgr/cephadm/osd_remove_queue
> > > > > []
> > > > >
> > > > > Thanks. I’m trying to upgrade this cluster all the way to current.
> > > > >
> > > > > -jeremy
> > > >
> > > >
> > > > _______________________________________________
> > > > ceph-users mailing list -- ceph-users@ceph.io
> > > > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> >
> >
