[ceph: root@cn01 /]# ceph orch host ls Error ENOENT: Module not found -jeremy
> On Saturday, Apr 05, 2025 at 1:27 PM, Eugen Block <ebl...@nde.ag > (mailto:ebl...@nde.ag)> wrote: > I don't see the module error in the logs you provided. Has it cleared? > A mgr failover is often helpful in such cases, basically it's the > first thing I suggest since two or three years. > > Zitat von Jeremy Hansen <jer...@skidrow.la>: > > > Thank you so much for the detailed instructions. Here’s logs from > > the failover to a new node. > > > > Apr 05 20:06:08 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > ::ffff:192.168.47.72 - - [05/Apr/2025:20:06:08] "GET /metrics HTTP/1.1" > > 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:06:08 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:08.568+0000 7efcbd800700 0 [prometheus INFO > > cherrypy.access.139625005087208] ::ffff:192.168.47.72 - - > > [05/Apr/2025:20:06:08] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:06:10 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > ::ffff:192.168.47.71 - - [05/Apr/2025:20:06:10] "GET /metrics HTTP/1.1" > > 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:06:10 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:10.702+0000 7efcb4c00700 0 [prometheus INFO > > cherrypy.access.139625005087208] ::ffff:192.168.47.71 - - > > [05/Apr/2025:20:06:10] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.027+0000 7efd95200700 1 mgr handle_mgr_map > > Activating! > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.028+0000 7efd95200700 1 mgr handle_mgr_map I am > > now > > activating > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.040+0000 7efc27e00700 0 [dashboard INFO root] > > Stopping > > engine... > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.040+0000 7efc27e00700 0 [dashboard INFO root] > > Stopped > > engine... > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.041+0000 7efc27e00700 0 [prometheus INFO root] > > Stopping > > engine... > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.041+0000 7efc27e00700 0 [prometheus INFO root] > > Stopped > > engine > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > [05/Apr/2025:20:06:12] ENGINE Bus > > STOPPING > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.041+0000 7efc43000700 0 [prometheus INFO > > cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Bus > > STOPPING > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.053+0000 7efc3a400700 0 [dashboard INFO root] > > Engine > > stopped. > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > [05/Apr/2025:20:06:12] ENGINE HTTP Server > > cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut > > down > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO > > cherrypy.error] [05/Apr/2025:20:06:12] ENGINE HTTP Server > > cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut > > down > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > [05/Apr/2025:20:06:12] ENGINE Bus > > STOPPED > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO > > cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Bus > > STOPPED > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO root] > > Engine > > stopped. > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.056+0000 7efc27e00700 0 [alerts DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.057+0000 7efc27e00700 0 [alerts INFO root] > > Init > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.057+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > alerts > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.058+0000 7efc27e00700 0 [balancer DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.058+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > balancer > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.058+0000 7efcfac00700 0 [alerts INFO root] > > Starting > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.060+0000 7efc27e00700 0 [cephadm DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.060+0000 7efcb4c00700 0 [balancer INFO root] > > Starting > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] > > Optimize plan > > auto_2025-04-05_20:06:12 > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] > > Mode upmap, max misplaced > > 0.050000 > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] > > Some PGs (1.000000) are unknown; try again > > later > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.212+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > cephadm > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.213+0000 7efc27e00700 0 [crash DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.213+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > crash > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.215+0000 7efc27e00700 0 [dashboard DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.215+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > dashboard > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.216+0000 7efc23800700 0 [dashboard INFO > > access_control] Loading user roles DB > > version=2 > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.217+0000 7efc23800700 0 [dashboard INFO sso] > > Loading SSO DB > > version=1 > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.217+0000 7efc23800700 0 [dashboard INFO root] > > server: ssl=yes host=:: > > port=8443 > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.218+0000 7efc27e00700 0 [devicehealth DEBUG > > root] setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.218+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > devicehealth > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.219+0000 7efc27e00700 0 [insights DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.219+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > insights > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.219+0000 7efc20600700 0 [insights INFO root] > > Reset curr health slot key health_history/2025-04-05_20 next flush None > > checks check count Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.219+0000 7efc21a00700 0 [devicehealth INFO root] > > Starting > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.222+0000 7efc27e00700 0 [iostat DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.222+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > iostat > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.224+0000 7efc27e00700 0 [mirroring DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.227+0000 7efc21a00700 0 [devicehealth INFO root] > > Check > > health > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.235+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > mirroring > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.236+0000 7efc27e00700 0 [nfs DEBUG root] setting > > log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.236+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > nfs > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.237+0000 7efc27e00700 0 [orchestrator DEBUG > > root] setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.237+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > orchestrator > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.239+0000 7efc27e00700 0 [pg_autoscaler DEBUG > > root] setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.239+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > pg_autoscaler > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.239+0000 7efc10200700 0 [pg_autoscaler INFO root] > > _maybe_adjust > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.240+0000 7efc27e00700 0 [progress DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.240+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > progress > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.247+0000 7efc27e00700 0 [prometheus DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.248+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > prometheus > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.252+0000 7efc0ee00700 0 [progress INFO root] > > Loading... > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.252+0000 7efc27e00700 0 [rbd_support DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] > > Loaded [<progress.module.GhostEvent object at 0x7efcf2127710>, > > <progress.module.GhostEvent object at 0x7efcf2127cc0>, > > <progress.module.GhostEvent object at 0x7efcf2127208>, > > <progress.module.GhostEvent object at 0x7efcf2127f28>, > > <progress.module.GhostEvent object at 0x7efcf21270f0>, > > <progress.module.GhostEvent object at 0x7efcf2127eb8>, > > <progress.module.GhostEvent object at 0x7efcf2127b38>, > > <progress.module.GhostEvent object at 0x7efcf2127a20>, > > <progress.module.GhostEvent object at 0x7efcf2127da0>, > > <progress.module.GhostEvent object at 0x7efcf2127f98>, > > <progress.module.GhostEvent object at 0x7efcf2127780>, > > <progress.module.GhostEvent object at 0x7efcf2127320>, > > <progress.module.GhostEvent object at 0x7efcf2127588>, > > <progress.module.GhostEvent object at 0x7efcf2127518>, > > <progress.module.GhostEvent object at 0x7efcf2127c50>, > > <progress.module.GhostEvent object at 0x7efcf21279e8>, > > <progress.module.GhostEvent object at 0x7efcf2127d68>, > > <progress.module.GhostEvent object at 0x7efcf2127470>, > > <progress.module.GhostEvent object at 0x7efcf2127940>, > > <progress.module.GhostEvent object at 0x7efcf2127048>, > > <progress.module.GhostEvent object at 0x7efcf2127390>, > > <progress.module.GhostEvent object at 0x7efcf21276d8>, > > <progress.module.GhostEvent object at 0x7efcf2127a90>, > > <progress.module.GhostEvent object at 0x7efcf2127ba8>, > > <progress.module.GhostEvent object at 0x7efcf2127e48>, > > <progress.module.GhostEvent object at 0x7efcf21272e8>, > > <progress.module.GhostEvent object at 0x7efcf2127080>, > > <progress.module.GhostEvent object at 0x7efcf2127b00>, > > <progress.module.GhostEvent object at 0x7efcf21279b0>, > > <progress.module.GhostEvent object at 0x7efcf20c1588>, > > <progress.module.GhostEvent object at 0x7efcf20c1ac8>, > > <progress.module.GhostEvent object at 0x7efcf20c1550>, > > <progress.module.GhostEvent object at 0x7efcf20c17f0>, > > <progress.module.GhostEvent object at 0x7efcf20c1d68>, > > <progress.module.GhostEvent object at 0x7efcf20c1b70>, > > <progress.module.GhostEvent object at 0x7efcf20c1c50>, > > <progress.module.GhostEvent object at 0x7efcf20c1860>, > > <progress.module.GhostEvent object at 0x7efcf20c1908>, > > <progress.module.GhostEvent object at 0x7efcf20c14e0>, > > <progress.module.GhostEvent object at 0x7efcf20c1cf8>, > > <progress.module.GhostEvent object at 0x7efcf20c1470>, > > <progress.module.GhostEvent object at 0x7efcf20c1d30>, > > <progress.module.GhostEvent object at 0x7efcf20c1e80>, > > <progress.module.GhostEvent object at 0x7efcf20c1e10>, > > <progress.module.GhostEvent object at 0x7efcf20c1eb8>, > > <progress.module.GhostEvent object at 0x7efcf20c1f28>, > > <progress.module.GhostEvent object at 0x7efcf20c1668>, > > <progress.module.GhostEvent object at 0x7efcf20c16a0>, > > <progress.module.GhostEvent object at 0x7efcf20c1ba8>, > > <progress.module.GhostEvent object at 0x7efcf20c1c18>] historic > > events > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] > > Loaded OSDMap, > > ready. > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] > > Processing OSDMap change > > 27179..27179 > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.256+0000 7efc0d600700 0 [prometheus INFO root] > > server_addr: :: server_port: > > 9283 > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.256+0000 7efc0d600700 0 [prometheus INFO root] > > Cache > > enabled > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.256+0000 7efc08600700 0 [prometheus INFO root] > > starting metric collection > > thread > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.258+0000 7efc0d600700 0 [prometheus INFO root] > > Starting > > engine... > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > [05/Apr/2025:20:06:12] ENGINE Bus > > STARTING > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.258+0000 7efc0d600700 0 [prometheus INFO > > cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Bus > > STARTING > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.259+0000 7efc07200700 0 [rbd_support INFO root] > > recovery thread > > starting > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.259+0000 7efc07200700 0 [rbd_support INFO root] > > starting > > setup > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.259+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > rbd_support > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.301+0000 7efc27e00700 0 [restful DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.301+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > restful > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.303+0000 7efc07200700 0 [rbd_support INFO root] > > MirrorSnapshotScheduleHandler: > > load_schedules > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.305+0000 7efc07200700 0 [rbd_support INFO root] > > load_schedules: csxyz, > > start_after= > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.307+0000 7efc04a00700 0 [restful INFO root] > > server_addr: :: server_port: > > 8003 > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.310+0000 7efc27e00700 0 [snap_schedule DEBUG > > root] setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.312+0000 7efc04a00700 0 [restful WARNING root] > > server not running: no certificate > > configured > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > Warning: Permanently added '192.168.47.73' (ECDSA) to the list of known > > hosts. > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.319+0000 7efc23800700 0 [dashboard INFO root] > > Configured CherryPy, starting > > engine... > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > Warning: Permanently added '192.168.47.72' (ECDSA) to the list of known > > hosts. > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > Warning: Permanently added '192.168.47.71' (ECDSA) to the list of known > > hosts. > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.361+0000 7efbf8200700 0 [rbd_support INFO root] > > MirrorSnapshotScheduleHandler: > > starting > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.369+0000 7efbf7800700 0 [rbd_support INFO root] > > PerfHandler: > > starting > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.371+0000 7efc07200700 0 [rbd_support INFO root] > > load_task_task: csxyz, > > start_after= > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.389+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > snap_schedule > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.390+0000 7efc27e00700 0 [status DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.390+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > status > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.391+0000 7efc27e00700 0 [telemetry DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.392+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > telemetry > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.393+0000 7efc27e00700 0 [volumes DEBUG root] > > setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.409+0000 7efc27e00700 0 [volumes INFO > > volumes.fs.async_job] queuing job for volume > > 'cssnfsxyz' > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.410+0000 7efc27e00700 0 [volumes INFO > > volumes.fs.async_job] queuing job for volume > > 'cssnfsxyz' > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.410+0000 7efc27e00700 1 mgr load Constructed > > class from module: > > volumes > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > CherryPy > > Checker: > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > The Application mounted at '' has an empty > > config. > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > [05/Apr/2025:20:06:12] ENGINE Serving on > > http://:::9283 > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.413+0000 7efc0d600700 0 [prometheus INFO > > cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Serving on > > http://:::9283 > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > [05/Apr/2025:20:06:12] ENGINE Bus > > STARTED > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.413+0000 7efc0d600700 0 [prometheus INFO > > cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Bus > > STARTED > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.413+0000 7efc0d600700 0 [prometheus INFO root] > > Engine > > started. > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error > > registering admin socket command: (17) File > > exists > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error > > registering admin socket command: (17) File > > exists > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error > > registering admin socket command: (17) File > > exists > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error > > registering admin socket command: (17) File > > exists > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error > > registering admin socket command: (17) File > > exists > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error > > registering admin socket command: (17) File > > exists > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error > > registering admin socket command: (17) File > > exists > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error > > registering admin socket command: (17) File > > exists > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error > > registering admin socket command: (17) File > > exists > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error > > registering admin socket command: (17) File > > exists > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.431+0000 7efbd8400700 0 [rbd_support INFO root] > > TaskHandler: > > starting > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.432+0000 7efc07200700 0 [rbd_support INFO root] > > TrashPurgeScheduleHandler: > > load_schedules > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.434+0000 7efc07200700 0 [rbd_support INFO root] > > load_schedules: csxyz, > > start_after= > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.445+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: CephFS -> > > /api/cephfs > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.448+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: CephFsUi -> > > /ui-api/cephfs > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.450+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: IscsiUi -> > > /ui-api/iscsi > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.450+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Iscsi -> > > /api/iscsi > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.450+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > IscsiTarget -> > > /api/iscsi/target > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.451+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > NFSGaneshaCluster -> > > /api/nfs-ganesha/cluster > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.452+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > NFSGaneshaExports -> > > /api/nfs-ganesha/export > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.453+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > NFSGaneshaUi -> > > /ui-api/nfs-ganesha > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.453+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Rbd -> > > /api/block/image > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.455+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: RbdStatus > > -> > > /ui-api/block/rbd > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.455+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdSnapshot -> > > /api/block/image/{image_spec}/snap > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.456+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: RbdTrash -> > > /api/block/image/trash > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.457+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdNamespace -> > > /api/block/pool/{pool_name}/namespace > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.458+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Pool -> > > /api/pool > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.460+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: PoolUi -> > > /ui-api/pool > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.461+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: RBDPool -> > > /api/pool > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.462+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Orchestrator -> > > /ui-api/orchestrator > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.462+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Service -> > > /api/service > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.463+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdMirroring -> > > /api/block/mirroring > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.463+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdMirroringSummary -> > > /api/block/mirroring/summary > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.464+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdMirroringPoolMode -> > > /api/block/mirroring/pool > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.464+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdMirroringPoolBootstrap -> > > /api/block/mirroring/pool/{pool_name}/bootstrap > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.464+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdMirroringPoolPeer -> > > /api/block/mirroring/pool/{pool_name}/peer > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.465+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdMirroringStatus -> > > /ui-api/block/mirroring > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.466+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Rgw -> > > /ui-api/rgw > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.466+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: RgwDaemon > > -> > > /api/rgw/daemon > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.466+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: RgwSite -> > > /api/rgw/site > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.467+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: RgwBucket > > -> > > /api/rgw/bucket > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.468+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: RgwUser -> > > /api/rgw/user > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.472+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Auth -> > > /api/auth > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.472+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Cluster -> > > /api/cluster > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.473+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > ClusterConfiguration -> > > /api/cluster_conf > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.474+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: CrushRule > > -> > > /api/crush_rule > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.476+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > CrushRuleUi -> > > /ui-api/crush_rule > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.476+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Daemon -> > > /api/daemon > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.477+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Docs -> > > /docs > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.477+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > ErasureCodeProfile -> > > /api/erasure_code_profile > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.478+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > ErasureCodeProfileUi -> > > /ui-api/erasure_code_profile > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.478+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > FrontendLogging -> > > /ui-api/logging > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.478+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Grafana -> > > /api/grafana > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.479+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Host -> > > /api/host > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.480+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: HostUi -> > > /ui-api/host > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.481+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Health -> > > /api/health > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.481+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > HomeController -> > > / > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.481+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > LangsController -> > > /ui-api/langs > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.482+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > LoginController -> > > /ui-api/login > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.482+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Logs -> > > /api/logs > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.482+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: MgrModules > > -> > > /api/mgr/module > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.483+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Monitor -> > > /api/monitor > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.484+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Osd -> > > /api/osd > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.486+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: OsdUi -> > > /ui-api/osd > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.487+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > OsdFlagsController -> > > /api/osd/flags > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.488+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > MdsPerfCounter -> > > /api/perf_counters/mds > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.488+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > MonPerfCounter -> > > /api/perf_counters/mon > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.489+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > OsdPerfCounter -> > > /api/perf_counters/osd > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.489+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RgwPerfCounter -> > > /api/perf_counters/rgw > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.490+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdMirrorPerfCounter -> > > /api/perf_counters/rbd-mirror > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.490+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > MgrPerfCounter -> > > /api/perf_counters/mgr > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.491+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > TcmuRunnerPerfCounter -> > > /api/perf_counters/tcmu-runner > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.491+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > PerfCounters -> > > /api/perf_counters > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.491+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > PrometheusReceiver -> > > /api/prometheus_receiver > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.492+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Prometheus > > -> > > /api/prometheus > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.493+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > PrometheusNotifications -> > > /api/prometheus/notifications > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.493+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Role -> > > /api/role > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.494+0000 7efbd7a00700 0 [rbd_support INFO root] > > TrashPurgeScheduleHandler: > > starting > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.495+0000 7efc07200700 0 [rbd_support INFO root] > > setup > > complete > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.495+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Scope -> > > /ui-api/scope > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.496+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Saml2 -> > > /auth/saml2 > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.496+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Settings -> > > /api/settings > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.497+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > StandardSettings -> > > /ui-api/standard_settings > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.498+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Summary -> > > /api/summary > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.498+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Task -> > > /api/task > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.498+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Telemetry > > -> > > /api/telemetry > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.499+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: User -> > > /api/user > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.500+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > UserPasswordPolicy -> > > /api/user > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.500+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > UserChangePassword -> > > /api/user/{username} > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.501+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > FeatureTogglesEndpoint -> > > /api/feature_toggles > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.501+0000 7efc23800700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > MessageOfTheDay -> > > /ui-api/motd > > Apr 05 20:06:12 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:12.710+0000 7efc23800700 0 [dashboard INFO > > dashboard.module] Engine > > started. > > Apr 05 20:06:13 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:13.039+0000 7efc27400700 0 log_channel(cluster) log > > [DBG] : pgmap v3: 609 pgs: 609 active+clean; 1.1 TiB data, 3.3 TiB used, > > 106 TiB / 109 TiB > > avail > > Apr 05 20:06:13 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > ::ffff:192.168.47.73 - - [05/Apr/2025:20:06:13] "GET /metrics HTTP/1.1" > > 200 39714 "" > > "Prometheus/2.33.4" > > Apr 05 20:06:13 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:13.287+0000 7efc02200700 0 [prometheus INFO > > cherrypy.access.139624858224400] ::ffff:192.168.47.73 - - > > [05/Apr/2025:20:06:13] "GET /metrics HTTP/1.1" 200 39714 "" > > "Prometheus/2.33.4" > > Apr 05 20:06:14 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:14.037+0000 7efc26000700 0 log_channel(cluster) log > > [DBG] : pgmap v4: 609 pgs: 609 active+clean; 1.1 TiB data, 3.3 TiB used, > > 106 TiB / 109 TiB > > avail > > Apr 05 20:06:16 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:16.038+0000 7efc26000700 0 log_channel(cluster) log > > [DBG] : pgmap v5: 609 pgs: 609 active+clean; 1.1 TiB data, 3.3 TiB used, > > 106 TiB / 109 TiB > > avail > > Apr 05 20:06:17 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:17.106+0000 7efc1fc00700 0 [insights INFO root] > > Received health check update 0 > > pending > > Apr 05 20:06:17 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:17.149+0000 7efc20600700 0 [insights INFO root] > > Applying 1 health updates to slot key health_history/2025-04-05_20 next > > flush None checks check count Apr 05 20:06:17 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:17.149+0000 7efc20600700 0 [insights INFO root] > > Maybe flushing slot key health_history/2025-04-05_20 next flush None checks > > check count 0 needed > > False > > Apr 05 20:06:17 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:17.256+0000 7efc0ee00700 0 [progress INFO root] > > Processing OSDMap change > > 27179..27179 > > Apr 05 20:06:17 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:17.518+0000 7efce9400700 0 [cephadm INFO root] > > Detected new or changed devices on > > cn01.ceph.xyz.corp > > Apr 05 20:06:17 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:17.518+0000 7efce9400700 0 log_channel(cephadm) log > > [INF] : Detected new or changed devices on > > cn01.ceph.xyz.corp > > Apr 05 20:06:17 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:17.611+0000 7efcd7c00700 0 [cephadm INFO root] > > Detected new or changed devices on > > cn03.ceph.xyz.corp > > Apr 05 20:06:17 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:17.611+0000 7efcd7c00700 0 log_channel(cephadm) log > > [INF] : Detected new or changed devices on > > cn03.ceph.xyz.corp > > Apr 05 20:06:17 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:17.726+0000 7efccf000700 0 [cephadm INFO root] > > Detected new or changed devices on > > cn02.ceph.xyz.corp > > Apr 05 20:06:17 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > debug 2025-04-05T20:06:17.726+0000 7efccf000700 0 log_channel(cephadm) log > > [INF] : Detected new or changed devices on > > cn02.ceph.xyz.corp > > > > > On Saturday, Apr 05, 2025 at 4:51 AM, Eugen Block <ebl...@nde.ag > > > (mailto:ebl...@nde.ag)> wrote: > > > Please don't drop the list from your response. > > > > > > The orchestrator might be down, but you should have an active mgr > > > (ceph -s). On the host running the mgr, you can just run 'cephadm logs > > > --name mgr.<your_mgr>', or 'journalctl -u > > > ceph-<ceph_fsid>@mgr.<your_mgr>'. > > > To get fresh logs, you can just run 'ceph mgr fail', look on which > > > host a different mgr takes over and gather logs from there. > > > > > > Zitat von Jeremy Hansen <jer...@skidrow.la>: > > > > > > > Thanks for the clarification on version jumps. I interpreted the > > > > docs to mean that I had to incrementally step through each release. > > > > This will save a lot of time. > > > > > > > > Since the orchestrator is down and I’m not sure how to trigger a > > > > failure, I don’t know how to gather those logs. I apologize. I > > > > appreciate your assistance. > > > > > > > > -jeremy > > > > > > > > > On Saturday, Apr 05, 2025 at 4:01 AM, Eugen Block <ebl...@nde.ag > > > > > (mailto:ebl...@nde.ag)> wrote: > > > > > It might be a different issue, can you paste the entire stack trace > > > > > from the mgr when it's failing? > > > > > > > > > > Also, you could go directly from Pacific to Reef, there's no need to > > > > > upgrade to Quincy (which is EOL). And I would also recommend to > > > > > upgrade to a latest major version, e.g. 17.2.8, not .0, otherwise you > > > > > might hit bugs which have already been fixed in later releases. > > > > > > > > > > Zitat von Jeremy Hansen <jer...@skidrow.la>: > > > > > > > > > > > I ran in to the “Error ENOENT: Module not found” issue with the > > > > > > orchestrator. I see the note in the cephadm upgrade docs but I don’t > > > > > > quite know what action to take to fix this: > > > > > > > > > > > > ceph versions > > > > > > { > > > > > > "mon": { > > > > > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) > > > > > > pacific (stable)": 3 > > > > > > }, > > > > > > "mgr": { > > > > > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) > > > > > > pacific (stable)": 1, > > > > > > "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) > > > > > > quincy (stable)": 2 > > > > > > }, > > > > > > "osd": { > > > > > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) > > > > > > pacific (stable)": 15 > > > > > > }, > > > > > > "mds": { > > > > > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) > > > > > > pacific (stable)": 3 > > > > > > }, > > > > > > "overall": { > > > > > > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) > > > > > > pacific (stable)": 22, > > > > > > "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) > > > > > > quincy (stable)": 2 > > > > > > } > > > > > > } > > > > > > > > > > > > ceph orch upgrade status > > > > > > Error ENOENT: Module not found > > > > > > > > > > > > ceph config-key get mgr/cephadm/osd_remove_queue > > > > > > [] > > > > > > > > > > > > Thanks. I’m trying to upgrade this cluster all the way to current. > > > > > > > > > > > > -jeremy > > > > > > > > > > > > > > > _______________________________________________ > > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > > > > > > > > >
signature.asc
Description: PGP signature
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
