Hi All,
Just now my cloudstack installation decided to expunge my virtual router from
some unknown reasons and I cannot bring it back either by issuing the
restartNetwork or starting/stopping existing VMs. Please advise
Logs:
2021-10-04 14:05:12,027 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-c472d70c) (logid:63e8f217) Zone 1 is ready to launch
console proxy
2021-10-04 14:05:12,087 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-b62699e4) (logid:ecc9b3f4) Zone 1 is ready to launch
secondary storage VM
2021-10-04 14:05:12,099 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-6:null) (logid:) Ping from 5(s-1-VM)
2021-10-04 14:05:13,271 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-11:null) (logid:) Ping from 6(v-2-VM)
2021-10-04 14:05:13,640 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-434:ctx-d6482f34) (logid:11ace9d1) Seq 1-8645222434691446206:
Response Received:
2021-10-04 14:05:13,641 DEBUG [c.c.a.t.Request] (DirectAgent-434:ctx-d6482f34)
(logid:11ace9d1) Seq 1-8645222434691446206: Processing: { Ans: , MgmtId:
1743772850, via: 1(LAC), Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":"true","wait":"0"}}] }
2021-10-04 14:05:13,641 DEBUG [c.c.a.t.Request]
(API-Job-Executor-21:ctx-149e14b4 job-4966 ctx-631bf242) (logid:11ace9d1) Seq
1-8645222434691446206: Received: { Ans: , MgmtId: 1743772850, via: 1(SG3LAC),
Ver: v1, Flags: 10, { Answer } }
2021-10-04 14:05:13,669 INFO [o.a.c.s.v.VolumeServiceImpl]
(API-Job-Executor-21:ctx-149e14b4 job-4966 ctx-631bf242) (logid:11ace9d1)
Volume 6 is not referred anywhere, remove it from volumes table
2021-10-04 14:05:13,695 DEBUG [c.c.v.VirtualMachineManagerImpl]
(API-Job-Executor-21:ctx-149e14b4 job-4966 ctx-631bf242) (logid:11ace9d1)
Expunged VM[DomainRouter|r-6-VM]
2021-10-04 14:05:13,722 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-21:ctx-149e14b4 job-4966 ctx-631bf242) (logid:11ace9d1)
Asking VirtualRouter to implemenet Ntwk[206|Guest|20]
2021-10-04 14:05:13,737 ERROR [o.c.n.r.d.RouterDeploymentDefinition]
(API-Job-Executor-21:ctx-149e14b4 job-4966 ctx-631bf242) (logid:11ace9d1)
Didn't support redundant virtual router without public network!
2021-10-04 14:05:13,760 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-21:ctx-149e14b4 job-4966) (logid:11ace9d1) Unexpected
exception while executing
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is
unreachable: Can't find all necessary running routers!
[BLOCK of TEXT JAVA Error]
2021-10-04 14:05:13,766 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-21:ctx-149e14b4 job-4966) (logid:11ace9d1) Complete async
job-4966, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Resource
[DataCenter:1] is unreachable: Can't find all necessary running routers!"}
2021-10-04 14:05:13,770 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-21:ctx-149e14b4 job-4966) (logid:11ace9d1) Publish async
job-4966 complete on message bus
2021-10-04 14:05:13,770 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-21:ctx-149e14b4 job-4966) (logid:11ace9d1) Wake up jobs
related to job-4966
2021-10-04 14:05:13,770 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-21:ctx-149e14b4 job-4966) (logid:11ace9d1) Update db status
for job-4966
2021-10-04 14:05:13,774 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-21:ctx-149e14b4 job-4966) (logid:11ace9d1) Wake up jobs
joined with job-4966 and disjoin all subjobs created from job- 4966
2021-10-04 14:05:13,790 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-21:ctx-149e14b4 job-4966) (logid:11ace9d1) Done executing
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for job-4966
