Hi,
One of our host are died (after very high load).
Host has a lot of Virtual Routers running.
We are using advanced networking.
Only way to bring machine alive was hard reset.
After this CloudStack decied to start all of VirtualRouterts,
when server was back online.
Most of VR are started, but couple don't.
Cloudstack want to start this VM, but can't ;(
Here are log from this action.
It's looks like that CS are looking for VR host on Xen host, but he couldn't
find.
Please help how to repair this situation.
Restart network with clean up options ending with the same resoult.
Is there any option to recreate VR?
I tried to remove VR from Infrastructure -> Virtual Router, but there is still
problem
with create new one.
Here are logs from "restart network with clean up options"
WARN [network.element.VpcVirtualRouterElement] (Job-Executor-151:job-6526)
Network Ntwk[200|Public|1] is not associated with any VPC
WARN [network.element.VpcVirtualRouterElement] (Job-Executor-151:job-6526)
Network Ntwk[338|Guest|8] is not associated with any VPC
WARN [network.element.VpcVirtualRouterElement] (Job-Executor-151:job-6526)
Network Ntwk[202|Control|3] is not associated with any VPC
ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-151:job-6526) Failed
to start instance VM[DomainRouter|r-1015-VM]
java.lang.NullPointerException
at
com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
at
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175)
at
com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518)
at
com.cloud.network.NetworkManagerImpl.restartNetwork(NetworkManagerImpl.java:2473)
at
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1381)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
INFO [xen.resource.CitrixResourceBase] (DirectAgent-16:) VM does not exist on
XenServer0d1d5015-4061-4554-96de-406155449296
WARN [cloud.network.NetworkManagerImpl] (Job-Executor-151:job-6526) Failed to
implement network Ntwk[338|Guest|8] elements and resources as a part of network
restart due to
com.cloud.exception.AgentUnavailableException: Resource [Host:9] is
unreachable: Host 9: Unable to start instance due to null
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
at
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175)
at
com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518)
at
com.cloud.network.NetworkManagerImpl.restartNetwork(NetworkManagerImpl.java:2473)
at
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1381)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NullPointerException
at
com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763)
... 23 more
WARN [cloud.network.NetworkServiceImpl] (Job-Executor-151:job-6526) Network
id=338 failed to restart.
WARN [cloud.async.AsyncJobManagerImpl] (Job-Executor-151:job-6526) Unable to
unregister active job 6526 from JMX monitoring
Here are log when I want to start VR (his state is stopeed, I want to start and
got this)
WARN [network.element.VpcVirtualRouterElement] (Job-Executor-153:job-6528)
Network Ntwk[200|Public|1] is not associated with any VPC
WARN [network.element.VpcVirtualRouterElement] (Job-Executor-153:job-6528)
Network Ntwk[338|Guest|8] is not associated with any VPC
WARN [network.element.VpcVirtualRouterElement] (Job-Executor-153:job-6528)
Network Ntwk[202|Control|3] is not associated with any VPC
ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-153:job-6528) Failed
to start instance VM[DomainRouter|r-1015-VM]
java.lang.NullPointerException
at
com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2874)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2824)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(StartRouterCmd.java:103)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Best Regards,
Michael
