rocky linux 9.2
mysql 8.0.32
A single Management Server node with mysql on the same node
1. After rebooting management host I get follow error messages
sudo systemctl status cloudstack-management
Loaded: loaded (/usr/lib/systemd/system/cloudstack-management.service;
enabled; preset: disabled)
Active: active (running) since Tue 2023-09-19 17:35:39 KST; 2min 3s ago
Main PID: 1148 (java)
Tasks: 49 (limit: 408699)
Memory: 1.1G
CPU: 40.653s
CGroup: /system.slice/cloudstack-management.service
└─1148 /usr/bin/java
-Djava.security.properties=/etc/cloudstack/management/java.security.ciphers
-Djava.awt.headless=true -Dcom.sun.management.>
java[1148]: at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
java[1148]: at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
java[1148]: at org.eclipse.jetty.server.Server.start(Server.java:423)
java[1148]: at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
java[1148]: at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
java[1148]: at org.eclipse.jetty.server.Server.doStart(Server.java:387)
java[1148]: at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
java[1148]: at
org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:192)
java[1148]: at
org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:107)
java[1148]: INFO [o.a.c.s.NfsMountManager] (main:null) (logid:) Clean up
mounted NFS mount points used in current session
server.log
2023-09-19 17:35:54,352 ERROR [c.c.c.ClusterManagerImpl] (main:null) (logid:)
Detected that another management node with the same IP 10.0.33.1 is already
running, please check your cluster configuration
2023-09-19 17:35:54,353 ERROR [o.a.c.s.l.CloudStackExtendedLifeCycle]
(main:null) (logid:) Failed to configure ClusterManagerImpl
javax.naming.ConfigurationException: Detected that another management node with
the same IP is already running, please check your cluster configuration
at
com.cloud.cluster.ClusterManagerImpl.checkConflicts(ClusterManagerImpl.java:1245)
at
com.cloud.cluster.ClusterManagerImpl.configure(ClusterManagerImpl.java:1115)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle$3.with(CloudStackExtendedLifeCycle.java:114)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:153)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.configure(CloudStackExtendedLifeCycle.java:110)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:55)
at
org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
at
org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54)
at
org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at
org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155)
at
org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
at
org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:937)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:144)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:121)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:244)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:232)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:116)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:78)
at
org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:70)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:57)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:61)
at
org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at
org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:192)
at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:107)
2. After restart cloudstack-management
2-1 as soon as restart
● cloudstack-management.service - CloudStack Management Server
Loaded: loaded (/usr/lib/systemd/system/cloudstack-management.service;
enabled; preset: disabled)
Active: active (running) since Tue 2023-09-19 17:47:21 KST; 6s ago
Main PID: 1893 (java)
Tasks: 43 (limit: 408699)
Memory: 895.5M
CPU: 19.833s
CGroup: /system.slice/cloudstack-management.service
└─1893 /usr/bin/java
-Djava.security.properties=/etc/cloudstack/management/java.security.ciphers
-Djava.awt.headless=true -Dcom.sun.management.>
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:28 mgmt1.acs.unitm.kr java[1893]: INFO [c.c.s.GsonHelper]
(main:null) (logid:) Default Builder inited.
2-2 after some seconds
● cloudstack-management.service - CloudStack Management Server
Loaded: loaded (/usr/lib/systemd/system/cloudstack-management.service;
enabled; preset: disabled)
Active: active (running) since Tue 2023-09-19 17:47:21 KST; 1min 27s ago
Main PID: 1893 (java)
Tasks: 183 (limit: 408699)
Memory: 1.0G
CPU: 56.129s
CGroup: /system.slice/cloudstack-management.service
└─1893 /usr/bin/java
-Djava.security.properties=/etc/cloudstack/management/java.security.ciphers
-Djava.awt.headless=true -Dcom.sun.management.>
Sep 19 17:47:39 mgmt1.acs.unitm.kr java[1893]: INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-cfbc07ba) (logid:ce3e9ffc) We are good, no orphan m>
Sep 19 17:47:39 mgmt1.acs.unitm.kr java[1893]: INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-cfbc07ba) (logid:ce3e9ffc) No inactive management s>
Sep 19 17:47:41 mgmt1.acs.unitm.kr java[1893]: WARN [c.c.a.AlertManagerImpl]
(Cluster-Notification-1:ctx-d247bd11) (logid:f2700a2b) alertType=[14] dataCent>
Sep 19 17:47:41 mgmt1.acs.unitm.kr java[1893]: WARN [c.c.a.AlertManagerImpl]
(Cluster-Notification-1:ctx-d247bd11) (logid:f2700a2b) No recipients set in gl>
Sep 19 17:47:41 mgmt1.acs.unitm.kr java[1893]: WARN [c.c.c.ClusterManagerImpl]
(Cluster-Notification-1:ctx-d247bd11) (logid:f2700a2b) Notifying management >
Sep 19 17:48:39 mgmt1.acs.unitm.kr java[1893]: INFO [c.c.s.StatsCollector]
(StatsCollector-1:ctx-fb9c4c33) (logid:891ff247) system memory from /proc:
67003>
Sep 19 17:48:39 mgmt1.acs.unitm.kr java[1893]: INFO [c.c.s.StatsCollector]
(StatsCollector-1:ctx-fb9c4c33) (logid:891ff247) free memory from /proc:
6451747>
Sep 19 17:48:39 mgmt1.acs.unitm.kr java[1893]: INFO [c.c.s.StatsCollector]
(StatsCollector-1:ctx-fb9c4c33) (logid:891ff247) used memory from /proc: 1119316
Sep 19 17:48:39 mgmt1.acs.unitm.kr java[1893]: ERROR
[c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:ctx-25684151)
(logid:c61e1671) IOException from : h>
Sep 19 17:48:39 mgmt1.acs.unitm.kr java[1893]: ERROR
[c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:ctx-25684151)
(logid:c61e1671) IOException from : h>
server logs
2023-09-19 17:47:34,671 ERROR [c.c.u.PropertiesUtil] (main:null) (logid:)
Unable to find properties file: commands.properties
2023-09-19 17:47:38,379 ERROR [c.c.c.ClusterServiceServletContainer]
(main:null) (logid:) error initializing cluster service servlet container
java.net.BindException: Address already in use (Bind failed)
at java.base/java.net.PlainSocketImpl.socketBind(Native Method)
at
java.base/java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:452)
at java.base/java.net.ServerSocket.bind(ServerSocket.java:395)
at java.base/java.net.ServerSocket.<init>(ServerSocket.java:257)
at java.base/java.net.ServerSocket.<init>(ServerSocket.java:149)
at
com.cloud.cluster.ClusterServiceServletContainer$ListenerThread.<init>(ClusterServiceServletContainer.java:82)
at
com.cloud.cluster.ClusterServiceServletContainer.start(ClusterServiceServletContainer.java:60)
at
com.cloud.cluster.ClusterServiceServletAdapter.start(ClusterServiceServletAdapter.java:111)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle$1.with(CloudStackExtendedLifeCycle.java:74)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:153)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.startBeans(CloudStackExtendedLifeCycle.java:71)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycleStart.run(CloudStackExtendedLifeCycleStart.java:46)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$1.with(DefaultModuleDefinitionSet.java:104)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:244)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:232)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.startContexts(DefaultModuleDefinitionSet.java:96)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79)
at
org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:70)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:57)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:61)
at
org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at
org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:192)
at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:107)
2023-09-19 17:47:39,549 ERROR [c.c.c.ClusterServiceServletImpl]
(Cluster-Worker-5:ctx-9f9b2994) (logid:66f4952c) IOException from :
http://hostip:9090/clusterservice, method : name=method, value=5
2023-09-19 17:47:39,555 ERROR [c.c.c.ClusterServiceServletImpl]
(Cluster-Worker-5:ctx-9f9b2994) (logid:66f4952c) IOException from :
http://hostip:9090/clusterservice, method : name=method, value=5
....
I saw issue #6814 that I think it is same problem, so I check database;
after reboot first db status
select * from mshost;
state up, port 9090
select * from mshost_peer;
state up
after restart cloudstack-management
always mshost is down state but mshost_peer up state even though
cloudstack-management stoped.
