I don't see the module error in the logs you provided. Has it cleared?
A mgr failover is often helpful in such cases, basically it's the
first thing I suggest since two or three years.
Zitat von Jeremy Hansen <jer...@skidrow.la>:
Thank you so much for the detailed instructions. Here’s logs from
the failover to a new node.
Apr 05 20:06:08 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:06:08] "GET /metrics HTTP/1.1" 200 - ""
"Prometheus/2.33.4"
Apr 05 20:06:08 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:08.568+0000 7efcbd800700 0 [prometheus INFO cherrypy.access.139625005087208] ::ffff:192.168.47.72 - - [05/Apr/2025:20:06:08] "GET /metrics HTTP/1.1" 200 - ""
"Prometheus/2.33.4"
Apr 05 20:06:10 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:06:10] "GET /metrics HTTP/1.1" 200 - ""
"Prometheus/2.33.4"
Apr 05 20:06:10 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:10.702+0000 7efcb4c00700 0 [prometheus INFO cherrypy.access.139625005087208] ::ffff:192.168.47.71 - - [05/Apr/2025:20:06:10] "GET /metrics HTTP/1.1" 200 - ""
"Prometheus/2.33.4"
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.027+0000 7efd95200700 1 mgr handle_mgr_map
Activating!
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.028+0000 7efd95200700 1 mgr handle_mgr_map I am now
activating
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.040+0000 7efc27e00700 0 [dashboard INFO root] Stopping
engine...
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.040+0000 7efc27e00700 0 [dashboard INFO root] Stopped
engine...
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc27e00700 0 [prometheus INFO root] Stopping
engine...
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc27e00700 0 [prometheus INFO root] Stopped
engine
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE Bus
STOPPING
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Bus
STOPPING
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.053+0000 7efc3a400700 0 [dashboard INFO root] Engine
stopped.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut
down
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut
down
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE Bus
STOPPED
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Bus
STOPPED
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO root] Engine
stopped.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.056+0000 7efc27e00700 0 [alerts DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.057+0000 7efc27e00700 0 [alerts INFO root]
Init
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.057+0000 7efc27e00700 1 mgr load Constructed class from module:
alerts
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efc27e00700 0 [balancer DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efc27e00700 1 mgr load Constructed class from module:
balancer
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efcfac00700 0 [alerts INFO root]
Starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.060+0000 7efc27e00700 0 [cephadm DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.060+0000 7efcb4c00700 0 [balancer INFO root]
Starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Optimize plan
auto_2025-04-05_20:06:12
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Mode upmap, max misplaced
0.050000
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try again
later
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.212+0000 7efc27e00700 1 mgr load Constructed class from module:
cephadm
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.213+0000 7efc27e00700 0 [crash DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.213+0000 7efc27e00700 1 mgr load Constructed class from module:
crash
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.215+0000 7efc27e00700 0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.215+0000 7efc27e00700 1 mgr load Constructed class from module:
dashboard
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.216+0000 7efc23800700 0 [dashboard INFO access_control] Loading user roles DB
version=2
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.217+0000 7efc23800700 0 [dashboard INFO sso] Loading SSO DB
version=1
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.217+0000 7efc23800700 0 [dashboard INFO root] server: ssl=yes host=::
port=8443
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.218+0000 7efc27e00700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.218+0000 7efc27e00700 1 mgr load Constructed class from module:
devicehealth
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc27e00700 0 [insights DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc27e00700 1 mgr load Constructed class from module:
insights
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc20600700 0 [insights INFO root] Reset curr health slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:06:12 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc21a00700 0 [devicehealth INFO root]
Starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.222+0000 7efc27e00700 0 [iostat DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.222+0000 7efc27e00700 1 mgr load Constructed class from module:
iostat
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.224+0000 7efc27e00700 0 [mirroring DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.227+0000 7efc21a00700 0 [devicehealth INFO root] Check
health
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.235+0000 7efc27e00700 1 mgr load Constructed class from module:
mirroring
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.236+0000 7efc27e00700 0 [nfs DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.236+0000 7efc27e00700 1 mgr load Constructed class from module:
nfs
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.237+0000 7efc27e00700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.237+0000 7efc27e00700 1 mgr load Constructed class from module:
orchestrator
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc27e00700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc27e00700 1 mgr load Constructed class from module:
pg_autoscaler
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc10200700 0 [pg_autoscaler INFO root]
_maybe_adjust
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.240+0000 7efc27e00700 0 [progress DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.240+0000 7efc27e00700 1 mgr load Constructed class from module:
progress
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.247+0000 7efc27e00700 0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.248+0000 7efc27e00700 1 mgr load Constructed class from module:
prometheus
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.252+0000 7efc0ee00700 0 [progress INFO root]
Loading...
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.252+0000 7efc27e00700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7efcf2127710>, <progress.module.GhostEvent object at 0x7efcf2127cc0>, <progress.module.GhostEvent object at 0x7efcf2127208>, <progress.module.GhostEvent object at 0x7efcf2127f28>, <progress.module.GhostEvent object at 0x7efcf21270f0>, <progress.module.GhostEvent object at 0x7efcf2127eb8>, <progress.module.GhostEvent object at 0x7efcf2127b38>, <progress.module.GhostEvent object at 0x7efcf2127a20>, <progress.module.GhostEvent object at 0x7efcf2127da0>, <progress.module.GhostEvent object at 0x7efcf2127f98>, <progress.module.GhostEvent object at 0x7efcf2127780>, <progress.module.GhostEvent object at 0x7efcf2127320>, <progress.module.GhostEvent object at 0x7efcf2127588>, <progress.module.GhostEvent object at 0x7efcf2127518>, <progress.module.GhostEvent object at 0x7efcf2127c50>, <progress.module.GhostEvent object at 0x7efcf21279e8>, <progress.module.GhostEvent object at 0x7efcf2127d68>, <progress.module.GhostEvent object at 0x7efcf2127470>, <progress.module.GhostEvent object at 0x7efcf2127940>, <progress.module.GhostEvent object at 0x7efcf2127048>, <progress.module.GhostEvent object at 0x7efcf2127390>, <progress.module.GhostEvent object at 0x7efcf21276d8>, <progress.module.GhostEvent object at 0x7efcf2127a90>, <progress.module.GhostEvent object at 0x7efcf2127ba8>, <progress.module.GhostEvent object at 0x7efcf2127e48>, <progress.module.GhostEvent object at 0x7efcf21272e8>, <progress.module.GhostEvent object at 0x7efcf2127080>, <progress.module.GhostEvent object at 0x7efcf2127b00>, <progress.module.GhostEvent object at 0x7efcf21279b0>, <progress.module.GhostEvent object at 0x7efcf20c1588>, <progress.module.GhostEvent object at 0x7efcf20c1ac8>, <progress.module.GhostEvent object at 0x7efcf20c1550>, <progress.module.GhostEvent object at 0x7efcf20c17f0>, <progress.module.GhostEvent object at 0x7efcf20c1d68>, <progress.module.GhostEvent object at 0x7efcf20c1b70>, <progress.module.GhostEvent object at 0x7efcf20c1c50>, <progress.module.GhostEvent object at 0x7efcf20c1860>, <progress.module.GhostEvent object at 0x7efcf20c1908>, <progress.module.GhostEvent object at 0x7efcf20c14e0>, <progress.module.GhostEvent object at 0x7efcf20c1cf8>, <progress.module.GhostEvent object at 0x7efcf20c1470>, <progress.module.GhostEvent object at 0x7efcf20c1d30>, <progress.module.GhostEvent object at 0x7efcf20c1e80>, <progress.module.GhostEvent object at 0x7efcf20c1e10>, <progress.module.GhostEvent object at 0x7efcf20c1eb8>, <progress.module.GhostEvent object at 0x7efcf20c1f28>, <progress.module.GhostEvent object at 0x7efcf20c1668>, <progress.module.GhostEvent object at 0x7efcf20c16a0>, <progress.module.GhostEvent object at 0x7efcf20c1ba8>, <progress.module.GhostEvent object at 0x7efcf20c1c18>] historic
events
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] Loaded OSDMap,
ready.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] Processing OSDMap change
27179..27179
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.256+0000 7efc0d600700 0 [prometheus INFO root] server_addr: :: server_port:
9283
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.256+0000 7efc0d600700 0 [prometheus INFO root] Cache
enabled
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.256+0000 7efc08600700 0 [prometheus INFO root] starting metric collection
thread
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.258+0000 7efc0d600700 0 [prometheus INFO root] Starting
engine...
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE Bus
STARTING
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.258+0000 7efc0d600700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Bus
STARTING
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.259+0000 7efc07200700 0 [rbd_support INFO root] recovery thread
starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.259+0000 7efc07200700 0 [rbd_support INFO root] starting
setup
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.259+0000 7efc27e00700 1 mgr load Constructed class from module:
rbd_support
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.301+0000 7efc27e00700 0 [restful DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.301+0000 7efc27e00700 1 mgr load Constructed class from module:
restful
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.303+0000 7efc07200700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler:
load_schedules
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.305+0000 7efc07200700 0 [rbd_support INFO root] load_schedules: csxyz,
start_after=
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.307+0000 7efc04a00700 0 [restful INFO root] server_addr: :: server_port:
8003
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.310+0000 7efc27e00700 0 [snap_schedule DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.312+0000 7efc04a00700 0 [restful WARNING root] server not running: no certificate
configured
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.73' (ECDSA) to the list of known
hosts.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.319+0000 7efc23800700 0 [dashboard INFO root] Configured CherryPy, starting
engine...
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.72' (ECDSA) to the list of known
hosts.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.71' (ECDSA) to the list of known
hosts.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.361+0000 7efbf8200700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler:
starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.369+0000 7efbf7800700 0 [rbd_support INFO root] PerfHandler:
starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.371+0000 7efc07200700 0 [rbd_support INFO root] load_task_task: csxyz,
start_after=
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.389+0000 7efc27e00700 1 mgr load Constructed class from module:
snap_schedule
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.390+0000 7efc27e00700 0 [status DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.390+0000 7efc27e00700 1 mgr load Constructed class from module:
status
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.391+0000 7efc27e00700 0 [telemetry DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.392+0000 7efc27e00700 1 mgr load Constructed class from module:
telemetry
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.393+0000 7efc27e00700 0 [volumes DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.409+0000 7efc27e00700 0 [volumes INFO volumes.fs.async_job] queuing job for volume
'cssnfsxyz'
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.410+0000 7efc27e00700 0 [volumes INFO volumes.fs.async_job] queuing job for volume
'cssnfsxyz'
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.410+0000 7efc27e00700 1 mgr load Constructed class from module:
volumes
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: CherryPy
Checker:
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an empty
config.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]:
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE Serving on
http://:::9283
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.413+0000 7efc0d600700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Serving on
http://:::9283
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE Bus
STARTED
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.413+0000 7efc0d600700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Bus
STARTED
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.413+0000 7efc0d600700 0 [prometheus INFO root] Engine
started.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.431+0000 7efbd8400700 0 [rbd_support INFO root] TaskHandler:
starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.432+0000 7efc07200700 0 [rbd_support INFO root] TrashPurgeScheduleHandler:
load_schedules
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.434+0000 7efc07200700 0 [rbd_support INFO root] load_schedules: csxyz,
start_after=
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.445+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFS ->
/api/cephfs
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.448+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFsUi ->
/ui-api/cephfs
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.450+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiUi ->
/ui-api/iscsi
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.450+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Iscsi ->
/api/iscsi
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.450+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiTarget ->
/api/iscsi/target
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.451+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaCluster ->
/api/nfs-ganesha/cluster
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.452+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaExports ->
/api/nfs-ganesha/export
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.453+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaUi ->
/ui-api/nfs-ganesha
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.453+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rbd ->
/api/block/image
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.455+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdStatus ->
/ui-api/block/rbd
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.455+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdSnapshot ->
/api/block/image/{image_spec}/snap
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.456+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdTrash ->
/api/block/image/trash
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.457+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdNamespace ->
/api/block/pool/{pool_name}/namespace
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.458+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Pool ->
/api/pool
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.460+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PoolUi ->
/ui-api/pool
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.461+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RBDPool ->
/api/pool
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.462+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Orchestrator ->
/ui-api/orchestrator
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.462+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Service ->
/api/service
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.463+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroring ->
/api/block/mirroring
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.463+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringSummary ->
/api/block/mirroring/summary
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.464+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolMode ->
/api/block/mirroring/pool
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.464+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolBootstrap ->
/api/block/mirroring/pool/{pool_name}/bootstrap
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.464+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolPeer ->
/api/block/mirroring/pool/{pool_name}/peer
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.465+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringStatus ->
/ui-api/block/mirroring
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.466+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rgw ->
/ui-api/rgw
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.466+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwDaemon ->
/api/rgw/daemon
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.466+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwSite ->
/api/rgw/site
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.467+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwBucket ->
/api/rgw/bucket
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.468+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwUser ->
/api/rgw/user
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.472+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Auth ->
/api/auth
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.472+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Cluster ->
/api/cluster
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.473+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ClusterConfiguration ->
/api/cluster_conf
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.474+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRule ->
/api/crush_rule
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.476+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRuleUi ->
/ui-api/crush_rule
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.476+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Daemon ->
/api/daemon
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.477+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Docs ->
/docs
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.477+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfile ->
/api/erasure_code_profile
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.478+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfileUi ->
/ui-api/erasure_code_profile
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.478+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FrontendLogging ->
/ui-api/logging
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.478+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Grafana ->
/api/grafana
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.479+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Host ->
/api/host
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.480+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HostUi ->
/ui-api/host
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.481+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Health ->
/api/health
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.481+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HomeController ->
/
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.481+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LangsController ->
/ui-api/langs
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.482+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LoginController ->
/ui-api/login
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.482+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Logs ->
/api/logs
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.482+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrModules ->
/api/mgr/module
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.483+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Monitor ->
/api/monitor
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.484+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Osd ->
/api/osd
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.486+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdUi ->
/ui-api/osd
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.487+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdFlagsController ->
/api/osd/flags
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.488+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MdsPerfCounter ->
/api/perf_counters/mds
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.488+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MonPerfCounter ->
/api/perf_counters/mon
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.489+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdPerfCounter ->
/api/perf_counters/osd
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.489+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwPerfCounter ->
/api/perf_counters/rgw
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.490+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirrorPerfCounter ->
/api/perf_counters/rbd-mirror
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.490+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrPerfCounter ->
/api/perf_counters/mgr
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.491+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: TcmuRunnerPerfCounter ->
/api/perf_counters/tcmu-runner
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.491+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PerfCounters ->
/api/perf_counters
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.491+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusReceiver ->
/api/prometheus_receiver
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.492+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Prometheus ->
/api/prometheus
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.493+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusNotifications ->
/api/prometheus/notifications
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.493+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Role ->
/api/role
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.494+0000 7efbd7a00700 0 [rbd_support INFO root] TrashPurgeScheduleHandler:
starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.495+0000 7efc07200700 0 [rbd_support INFO root] setup
complete
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.495+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Scope ->
/ui-api/scope
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.496+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Saml2 ->
/auth/saml2
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.496+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Settings ->
/api/settings
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.497+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: StandardSettings ->
/ui-api/standard_settings
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.498+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Summary ->
/api/summary
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.498+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Task ->
/api/task
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.498+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Telemetry ->
/api/telemetry
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.499+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: User ->
/api/user
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.500+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserPasswordPolicy ->
/api/user
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.500+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserChangePassword ->
/api/user/{username}
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.501+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FeatureTogglesEndpoint ->
/api/feature_toggles
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.501+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MessageOfTheDay ->
/ui-api/motd
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.710+0000 7efc23800700 0 [dashboard INFO dashboard.module] Engine
started.
Apr 05 20:06:13 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:13.039+0000 7efc27400700 0 log_channel(cluster) log [DBG] : pgmap v3: 609 pgs: 609 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB
avail
Apr 05 20:06:13 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:06:13] "GET /metrics HTTP/1.1" 200 39714 ""
"Prometheus/2.33.4"
Apr 05 20:06:13 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:13.287+0000 7efc02200700 0 [prometheus INFO cherrypy.access.139624858224400] ::ffff:192.168.47.73 - - [05/Apr/2025:20:06:13] "GET /metrics HTTP/1.1" 200 39714 ""
"Prometheus/2.33.4"
Apr 05 20:06:14 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:14.037+0000 7efc26000700 0 log_channel(cluster) log [DBG] : pgmap v4: 609 pgs: 609 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB
avail
Apr 05 20:06:16 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:16.038+0000 7efc26000700 0 log_channel(cluster) log [DBG] : pgmap v5: 609 pgs: 609 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB
avail
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.106+0000 7efc1fc00700 0 [insights INFO root] Received health check update 0
pending
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.149+0000 7efc20600700 0 [insights INFO root] Applying 1 health updates to slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:06:17 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.149+0000 7efc20600700 0 [insights INFO root] Maybe flushing slot key health_history/2025-04-05_20 next flush None checks check count 0 needed
False
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.256+0000 7efc0ee00700 0 [progress INFO root] Processing OSDMap change
27179..27179
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.518+0000 7efce9400700 0 [cephadm INFO root] Detected new or changed devices on
cn01.ceph.xyz.corp
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.518+0000 7efce9400700 0 log_channel(cephadm) log [INF] : Detected new or changed devices on
cn01.ceph.xyz.corp
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.611+0000 7efcd7c00700 0 [cephadm INFO root] Detected new or changed devices on
cn03.ceph.xyz.corp
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.611+0000 7efcd7c00700 0 log_channel(cephadm) log [INF] : Detected new or changed devices on
cn03.ceph.xyz.corp
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.726+0000 7efccf000700 0 [cephadm INFO root] Detected new or changed devices on
cn02.ceph.xyz.corp
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.726+0000 7efccf000700 0 log_channel(cephadm) log [INF] : Detected new or changed devices on
cn02.ceph.xyz.corp
On Saturday, Apr 05, 2025 at 4:51 AM, Eugen Block <ebl...@nde.ag
(mailto:ebl...@nde.ag)> wrote:
Please don't drop the list from your response.
The orchestrator might be down, but you should have an active mgr
(ceph -s). On the host running the mgr, you can just run 'cephadm logs
--name mgr.<your_mgr>', or 'journalctl -u
ceph-<ceph_fsid>@mgr.<your_mgr>'.
To get fresh logs, you can just run 'ceph mgr fail', look on which
host a different mgr takes over and gather logs from there.
Zitat von Jeremy Hansen <jer...@skidrow.la>:
> Thanks for the clarification on version jumps. I interpreted the
> docs to mean that I had to incrementally step through each release.
> This will save a lot of time.
>
> Since the orchestrator is down and I’m not sure how to trigger a
> failure, I don’t know how to gather those logs. I apologize. I
> appreciate your assistance.
>
> -jeremy
>
> > On Saturday, Apr 05, 2025 at 4:01 AM, Eugen Block <ebl...@nde.ag
> > (mailto:ebl...@nde.ag)> wrote:
> > It might be a different issue, can you paste the entire stack trace
> > from the mgr when it's failing?
> >
> > Also, you could go directly from Pacific to Reef, there's no need to
> > upgrade to Quincy (which is EOL). And I would also recommend to
> > upgrade to a latest major version, e.g. 17.2.8, not .0, otherwise you
> > might hit bugs which have already been fixed in later releases.
> >
> > Zitat von Jeremy Hansen <jer...@skidrow.la>:
> >
> > > I ran in to the “Error ENOENT: Module not found” issue with the
> > > orchestrator. I see the note in the cephadm upgrade docs but I don’t
> > > quite know what action to take to fix this:
> > >
> > > ceph versions
> > > {
> > > "mon": {
> > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
> > > pacific (stable)": 3
> > > },
> > > "mgr": {
> > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
> > > pacific (stable)": 1,
> > > "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e)
> > > quincy (stable)": 2
> > > },
> > > "osd": {
> > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
> > > pacific (stable)": 15
> > > },
> > > "mds": {
> > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
> > > pacific (stable)": 3
> > > },
> > > "overall": {
> > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
> > > pacific (stable)": 22,
> > > "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e)
> > > quincy (stable)": 2
> > > }
> > > }
> > >
> > > ceph orch upgrade status
> > > Error ENOENT: Module not found
> > >
> > > ceph config-key get mgr/cephadm/osd_remove_queue
> > > []
> > >
> > > Thanks. I’m trying to upgrade this cluster all the way to current.
> > >
> > > -jeremy
> >
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
