These logs are from this morning.
I rebuilt my CloudStack environment and now the System VMs have
connectivity, so much so that I can access the VMs via SSH and ping the
management, public, and storage networks through it.
I solved the logs you sent me by simply rebuilding the environment (and
network) from scratch.
The SSL logs I mentioned are reported below:
2025-08-11 20:34:11,995 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Worker-5:[ctx-f152a4c5]) (logid:099f2acb) Cluster PDU
97591085894372 -> 97591085894372 completed. time: 4ms. agent: 0, pdu
seq: 263, pdu ack seq: 0, json:
{"managementServerHostId":1,"managementServerHostUuid":"4cfdc9b0-565e-4920-bc32-d079f456fc8b","managementServerRunId":1754939509540,"collectionTime":"Aug
11, 2025, 8:34:11
PM","sessions":2,"cpuUtilization":0.0,"totalJvmMemoryBytes":1124597760,"freeJvmMemoryBytes":384206344,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":15765252,"jvmStartTime":1754939486632,"availableProcessors":32,"loadAverage":0.17,"totalInit":1591017472,"totalUsed":1012164680,"totalCommitted":1401356288,"pid":217209,"jvmName":"217209@cloudstack-onexbh","jvmVendor":"Ubuntu","jvmVersion":"17.0.15+6-Ubuntu-0ubuntu120.04","osDistribution":"Ubuntu
20.04.6
LTS","agentCount":5,"heapMemoryUsed":741131168,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":30,"threadsRunnableCount":23,"threadsTerminatedCount":0,"threadsTotalCount":1427,"threadsWaitingCount":1317,"systemMemoryTotal":101304950784,"systemMemoryFree":94171963392,"systemMemoryUsed":1912644,"systemMemoryVirtualSize":22671781888,"logInfo":"","systemTotalCpuCycles":43622.244,"systemLoadAverages":[0.17,0.1,0.09],"systemCyclesUsage":[652771,292009,862649072],"dbLocal":true,"usageLocal":false,"systemBootTime":"Aug
8, 2025, 5:24:32 PM","kernelVersion":"5.4.0-216-generic"}
2025-08-11 20:34:11,995 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Worker-5:[ctx-f152a4c5]) (logid:099f2acb) Cluster PDU
97591085894372 -> 97591085894372. agent: 0, pdu seq: 263, pdu ack seq:
0, json:
{"managementServerHostId":1,"managementServerHostUuid":"4cfdc9b0-565e-4920-bc32-d079f456fc8b","managementServerRunId":1754939509540,"collectionTime":"Aug
11, 2025, 8:34:11
PM","sessions":2,"cpuUtilization":0.0,"totalJvmMemoryBytes":1124597760,"freeJvmMemoryBytes":384206344,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":15765252,"jvmStartTime":1754939486632,"availableProcessors":32,"loadAverage":0.17,"totalInit":1591017472,"totalUsed":1012164680,"totalCommitted":1401356288,"pid":217209,"jvmName":"217209@cloudstack-onexbh","jvmVendor":"Ubuntu","jvmVersion":"17.0.15+6-Ubuntu-0ubuntu120.04","osDistribution":"Ubuntu
20.04.6
LTS","agentCount":5,"heapMemoryUsed":741131168,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":30,"threadsRunnableCount":23,"threadsTerminatedCount":0,"threadsTotalCount":1427,"threadsWaitingCount":1317,"systemMemoryTotal":101304950784,"systemMemoryFree":94171963392,"systemMemoryUsed":1912644,"systemMemoryVirtualSize":22671781888,"logInfo":"","systemTotalCpuCycles":43622.244,"systemLoadAverages":[0.17,0.1,0.09],"systemCyclesUsage":[652771,292009,862649072],"dbLocal":true,"usageLocal":false,"systemBootTime":"Aug
8, 2025, 5:24:32 PM","kernelVersion":"5.4.0-216-generic"}
2025-08-11 20:34:11,995 DEBUG [c.c.c.ClusterServiceServletImpl]
(Cluster-Worker-5:[ctx-f152a4c5]) (logid:099f2acb) Executing
ClusterServicePdu with service URL:
https://127.0.0.1:9090/clusterservice
2025-08-11 20:34:11,998 ERROR [c.c.c.ClusterServiceServletImpl]
(Cluster-Worker-5:[ctx-f152a4c5]) (logid:099f2acb) Exception from :
https://127.0.0.1:9090/clusterservice, method : null, exception :
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <127.0.0.1>
doesn't match any of the subject alternative names:
[fe80:0:0:0:5ac2:32ff:fe02:ae4, 10.31.4.50, 10.31.3.1,
cloudstack-onexbh, cloudstack.internal]
at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:507)
at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
at
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at
com.cloud.cluster.ClusterServiceServletImpl.executePostMethod(ClusterServiceServletImpl.java:143)
at
com.cloud.cluster.ClusterServiceServletImpl.execute(ClusterServiceServletImpl.java:106)
at
com.cloud.cluster.ClusterManagerImpl.onSendingClusterPdu(ClusterManagerImpl.java:275)
at
com.cloud.cluster.ClusterManagerImpl$1.runInContext(ClusterManagerImpl.java:235)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2025-08-11 20:34:11,998 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Worker-5:[ctx-f152a4c5]) (logid:099f2acb) Cluster PDU
97591085894372 -> 97591085894372 completed. time: 3ms. agent: 0, pdu
seq: 263, pdu ack seq: 0, json:
{"managementServerHostId":1,"managementServerHostUuid":"4cfdc9b0-565e-4920-bc32-d079f456fc8b","managementServerRunId":1754939509540,"collectionTime":"Aug
11, 2025, 8:34:11
PM","sessions":2,"cpuUtilization":0.0,"totalJvmMemoryBytes":1124597760,"freeJvmMemoryBytes":384206344,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":15765252,"jvmStartTime":1754939486632,"availableProcessors":32,"loadAverage":0.17,"totalInit":1591017472,"totalUsed":1012164680,"totalCommitted":1401356288,"pid":217209,"jvmName":"217209@cloudstack-onexbh","jvmVendor":"Ubuntu","jvmVersion":"17.0.15+6-Ubuntu-0ubuntu120.04","osDistribution":"Ubuntu
20.04.6
LTS","agentCount":5,"heapMemoryUsed":741131168,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":30,"threadsRunnableCount":23,"threadsTerminatedCount":0,"threadsTotalCount":1427,"threadsWaitingCount":1317,"systemMemoryTotal":101304950784,"systemMemoryFree":94171963392,"systemMemoryUsed":1912644,"systemMemoryVirtualSize":22671781888,"logInfo":"","systemTotalCpuCycles":43622.244,"systemLoadAverages":[0.17,0.1,0.09],"systemCyclesUsage":[652771,292009,862649072],"dbLocal":true,"usageLocal":false,"systemBootTime":"Aug
8, 2025, 5:24:32 PM","kernelVersion":"5.4.0-216-generic"}
2025-08-11 20:34:12,063 DEBUG [c.c.s.S.VmStatsCollector]
(StatsCollector-3:[ctx-e9f9863e]) (logid:3a21ac65) VmStatsCollector is
running to process VMs across 3 UP hosts
2025-08-11 20:34:12,067 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(StatsCollector-3:[ctx-e9f9863e]) (logid:3a21ac65) Wait time setting on
com.cloud.agent.api.GetVmStatsCommand is 1800 seconds
2025-08-11 20:34:12,067 DEBUG [c.c.a.m.ClusteredDirectAgentAttache]
(StatsCollector-3:[ctx-e9f9863e]) (logid:3a21ac65) Seq
5-3902650552093245880: Routed from 97591085894372
2025-08-11 20:34:12,068 DEBUG [c.c.a.m.D.Task]
(DirectAgent-90:[ctx-4073f5c6]) (logid:8267a6c4) Seq
5-3902650552093245880: Executing request
2025-08-11 20:34:12,068 DEBUG [c.c.h.v.r.VmwareResource]
(DirectAgent-90:[ctx-4073f5c6]) (logid:3a21ac65) Executing resource
command GetVmStatsCommand: [].
I remain at your disposal and thank you for your support!!
Kayo Henrique
Analista de Infraestrutura e Redes
OneX Data Centers
Em 2025-08-12 07:03, Prashanth Reddy escreveu:
From the logs I don't see an issue with SSL certs itself, we clearly
see management servers are unable to connect to the systemVM's.
For the SSVM and CPVM to work, management servers should be able to ssh
into the systemVM's on the POD IP's ( in case of vmware) assigned. From
the logs we see this as failing as below
Logs for SSVM ( similar logs are seen for CPVM as well)
2025-08-11 11:07:53,220 DEBUG [c.c.h.v.r.VmwareResource]
(DirectAgent-180:[ctx-d02be6b8, 10.42.0.23, job-44/job-138, cmd:
StartCommand]) (logid:2459ff24) VM s-31-VM has been started
successfully with hostname s-31-VM.
2025-08-11 11:07:53,220 DEBUG [c.c.a.r.v.VirtualRoutingResource]
(DirectAgent-180:[ctx-d02be6b8, 10.42.0.23, job-44/job-138, cmd:
StartCommand]) (logid:2459ff24) Trying to connect to 10.42.0.93
2025-08-11 11:07:56,276 DEBUG [c.c.a.r.v.VirtualRoutingResource]
(DirectAgent-180:[ctx-d02be6b8, 10.42.0.23, job-44/job-138, cmd:
StartCommand]) (logid:2459ff24) Could not connect to 10.42.0.93
2025-08-11 11:08:01,276 DEBUG [c.c.a.r.v.VirtualRoutingResource]
(DirectAgent-180:[ctx-d02be6b8, 10.42.0.23, job-44/job-138, cmd:
StartCommand]) (logid:2459ff24) Trying to connect to 10.42.0.93
2025-08-11 11:08:04,340 DEBUG [c.c.a.r.v.VirtualRoutingResource]
(DirectAgent-180:[ctx-d02be6b8, 10.42.0.23, job-44/job-138, cmd:
StartCommand]) (logid:2459ff24) Could not connect to 10.42.0.93
2025-08-11 11:08:09,340 DEBUG [c.c.a.r.v.VirtualRoutingResource]
(DirectAgent-180:[ctx-d02be6b8, 10.42.0.23, job-44/job-138, cmd:
StartCommand]) (logid:2459ff24) Unable to logon to 10.42.0.93
2025-08-11 11:08:09,340 DEBUG [c.c.a.r.v.VirtualRoutingResource]
(DirectAgent-180:[ctx-d02be6b8, 10.42.0.23, job-44/job-138, cmd:
StartCommand]) (logid:2459ff24) Trying to connect to 10.42.0.93
2025-08-11 11:23:52,828 DEBUG [c.c.a.r.v.VirtualRoutingResource]
(DirectAgent-180:[ctx-d02be6b8, 10.42.0.23, job-44/job-138, cmd:
StartCommand]) (logid:2459ff24) Trying to connect to 10.42.0.93
2025-08-11 11:23:55,892 DEBUG [c.c.a.r.v.VirtualRoutingResource]
(DirectAgent-180:[ctx-d02be6b8, 10.42.0.23, job-44/job-138, cmd:
StartCommand]) (logid:2459ff24) Could not connect to 10.42.0.93
2025-08-11 11:24:00,892 DEBUG [c.c.a.r.v.VirtualRoutingResource]
(DirectAgent-180:[ctx-d02be6b8, 10.42.0.23, job-44/job-138, cmd:
StartCommand]) (logid:2459ff24) Unable to logon to 10.42.0.93
2025-08-11 11:24:03,956 ERROR [c.c.u.FileUtil]
(DirectAgent-180:[ctx-d02be6b8, 10.42.0.23, job-44/job-138, cmd:
StartCommand]) (logid:2459ff24) Failed to scp files to system VM due
to, No route to host
2025-08-11 11:24:07,028 ERROR [c.c.u.FileUtil]
(DirectAgent-180:[ctx-d02be6b8, 10.42.0.23, job-44/job-138, cmd:
StartCommand]) (logid:2459ff24) Failed to scp files to system VM due
to, No route to host
2025-08-11 11:24:10,100 ERROR [c.c.u.FileUtil]
(DirectAgent-180:[ctx-d02be6b8, 10.42.0.23, job-44/job-138, cmd:
StartCommand]) (logid:2459ff24) Failed to scp files to system VM due
to, No route to host
2025-08-11 11:24:10,100 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-180:[ctx-d02be6b8, 10.42.0.23, job-44/job-138, cmd:
StartCommand]) (logid:2459ff24) Failed to scp files to system VM.
Patching of systemVM failed
com.cloud.utils.exception.CloudRuntimeException: Failed to scp files to
system VM due to, No route to host
2025-08-11 11:24:10,123 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-91:[ctx-9e73a045, job-44/job-138, ctx-025b0546])
(logid:2459ff24) Seq 1-5696209103693548290: Received: { Ans: , MgmtId:
97591085894372, via: 1(10.42.0.23), Ver: v1, Flags: 110, { StartAnswer
} }
2025-08-11 11:24:10,130 INFO
[c.c.v.ClusteredVirtualMachineManagerImpl]
(Work-Job-Executor-91:[ctx-9e73a045, job-44/job-138, ctx-025b0546])
(logid:2459ff24) Unable to start VM on Host
{"id":1,"name":"10.42.0.23","type":"Routing","uuid":"8733a859-b04f-4341-9ceb-182b7917628f"}
due to Failed to scp files to system VM. Patching of systemVM failed
due to: Failed to scp files to system VM due to, No route to host
2025-08-11 11:24:10,139 DEBUG
[c.c.v.ClusteredVirtualMachineManagerImpl]
(Work-Job-Executor-91:[ctx-9e73a045, job-44/job-138, ctx-025b0546])
(logid:2459ff24) Cleaning up resources for the vm VM instance
{"id":31,"instanceName":"s-31-VM","state":"Starting","type":"SecondaryStorageVm","uuid":"700c9d06-fc6e-40ce-a42f-d59e657771b3"}
in Starting state
From the logs your pod network is using the following vswitch on vmware
- "name":"vSwitch_Storage,402,vmwaresvs"
You can try to ssh into the SystemVM's directly from the management
servers once they are running on vmware and see if that works - You can
ssh to systemVM's on vmware using the steps here The System VM Template
— Apache CloudStack 4.20.1.0
documentation<https://docs.cloudstack.apache.org/en/latest/adminguide/systemvm.html#accessing-system-vms>
The System VM Template — Apache CloudStack 4.20.1.0
documentation<https://docs.cloudstack.apache.org/en/latest/adminguide/systemvm.html#accessing-system-vms>
CloudStack uses several types of system Instances to perform tasks in
the cloud. In general CloudStack manages these system VMs and creates,
starts, and stops them as needed based on scale and immediate needs.
Unlike user VMs, system VMs are expunged on destroying them. However,
the administrator should be aware of them and their roles to assist in
debugging issues. The System VM Template The ...
docs.cloudstack.apache.org
If the ssh to systemVM's is not working , To isolate the issue may be
you can deploy a VM directly on vmware with a nic on the vswitch stated
above and see if you can ssh into the VM from all your management
servers once running.
Thanks
Prashanth
________________________________
From: Kayo Henrique <kayo.henri...@onexdatacenter.com.br>
Sent: Tuesday, August 12, 2025 1:15 AM
To: Users <users@cloudstack.apache.org>
Subject: POSSIBLE SSL ERROR ON SYSTEM VMS - POSSÍVEL ERRO DE SSL NAS
SYSTEM VMS
*IN ENGLISH*
Hello,
I've rebuilt my CloudStack environment to VMware and I'm having a
problem.
It appears my System VMs are powered on, have connectivity, and are
pinging all networks, but the System VM services (SSVM and CPVM) aren't
working.
The evidence images and the management server log file are available at
the link below:
https://drive.onexdatacenter.com.br/s/gRreLjZ4bg5KPHM
I did some research and discovered that it might be related to the SSL
certificate, but I don't quite understand how it works!
I'm here to help!
//
*EM PORTUGUÊS*
Olá,
Refiz meu ambiente de CloudStack para VMware e estou com um problema.
Aparentemente minhas System VMs estão ligadas, com conectividade,
pingando todas as redes, mas os serviços das System VMs (SSVM e CPVM)
não funcionam.
As imagens de evidência e o arquivo de logs do management server estão
presentes no link abaixo:
https://drive.onexdatacenter.com.br/s/gRreLjZ4bg5KPHM
Pesquisei um pouco sobre e descobri que pode estar relacionado ao
certificado SSL, mas não entendi muito bem como funcionaria isso!
Fico à disposição!!
--
Atenciosamente,
Kayo Henrique
Analista de Infraestrutura e Redes
OneX Data Centers
