Hi, We are usning xenserver from Citrix.
I found some clue. In console to server on this host I see that nfs has error "Stale NFS file handle". How to remount NFS? Maybe this is a problem? But why other VR are started and few of them not... ;( Best regards, Michael > Hi Michael, > if it helps, I would restart the agent on the host machine where VR lives. > it depends on hypervisor but on KVM, following command: service > cloudstack-agent restart > Thanks, > Motty > On Thu, Aug 7, 2014 at 11:52 AM, Michael Lukzak <[email protected]> wrote: >> Hi, >> One of our host are died (after very high load). >> Host has a lot of Virtual Routers running. >> We are using advanced networking. >> Only way to bring machine alive was hard reset. >> After this CloudStack decied to start all of VirtualRouterts, >> when server was back online. >> Most of VR are started, but couple don't. >> Cloudstack want to start this VM, but can't ;( >> Here are log from this action. >> It's looks like that CS are looking for VR host on Xen host, but he >> couldn't find. >> Please help how to repair this situation. >> Restart network with clean up options ending with the same resoult. >> Is there any option to recreate VR? >> I tried to remove VR from Infrastructure -> Virtual Router, but there is >> still problem >> with create new one. >> Here are logs from "restart network with clean up options" >> WARN [network.element.VpcVirtualRouterElement] >> (Job-Executor-151:job-6526) Network Ntwk[200|Public|1] is not associated >> with any VPC >> WARN [network.element.VpcVirtualRouterElement] >> (Job-Executor-151:job-6526) Network Ntwk[338|Guest|8] is not associated >> with any VPC >> WARN [network.element.VpcVirtualRouterElement] >> (Job-Executor-151:job-6526) Network Ntwk[202|Control|3] is not associated >> with any VPC >> ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-151:job-6526) >> Failed to start instance VM[DomainRouter|r-1015-VM] >> java.lang.NullPointerException >> at >> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) >> at >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) >> at >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) >> at >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) >> at >> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) >> at >> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175) >> at >> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518) >> at >> com.cloud.network.NetworkManagerImpl.restartNetwork(NetworkManagerImpl.java:2473) >> at >> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1381) >> at >> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >> at >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92) >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) >> at >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:722) >> INFO [xen.resource.CitrixResourceBase] (DirectAgent-16:) VM does not >> exist on XenServer0d1d5015-4061-4554-96de-406155449296 >> WARN [cloud.network.NetworkManagerImpl] (Job-Executor-151:job-6526) >> Failed to implement network Ntwk[338|Guest|8] elements and resources as a >> part of network restart due to >> com.cloud.exception.AgentUnavailableException: Resource [Host:9] is >> unreachable: Host 9: Unable to start instance due to null >> at >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841) >> at >> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) >> at >> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175) >> at >> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518) >> at >> com.cloud.network.NetworkManagerImpl.restartNetwork(NetworkManagerImpl.java:2473) >> at >> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1381) >> at >> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >> at >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92) >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) >> at >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:722) >> Caused by: java.lang.NullPointerException >> at >> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) >> at >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) >> at >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) >> at >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) >> ... 23 more >> WARN [cloud.network.NetworkServiceImpl] (Job-Executor-151:job-6526) >> Network id=338 failed to restart. >> WARN [cloud.async.AsyncJobManagerImpl] (Job-Executor-151:job-6526) Unable >> to unregister active job 6526 from JMX monitoring >> Here are log when I want to start VR (his state is stopeed, I want to >> start and got this) >> WARN [network.element.VpcVirtualRouterElement] >> (Job-Executor-153:job-6528) Network Ntwk[200|Public|1] is not associated >> with any VPC >> WARN [network.element.VpcVirtualRouterElement] >> (Job-Executor-153:job-6528) Network Ntwk[338|Guest|8] is not associated >> with any VPC >> WARN [network.element.VpcVirtualRouterElement] >> (Job-Executor-153:job-6528) Network Ntwk[202|Control|3] is not associated >> with any VPC >> ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-153:job-6528) >> Failed to start instance VM[DomainRouter|r-1015-VM] >> java.lang.NullPointerException >> at >> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) >> at >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) >> at >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) >> at >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) >> at >> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2874) >> at >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2824) >> at >> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >> at >> org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(StartRouterCmd.java:103) >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) >> at >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:722) >> Best Regards, >> Michael -- Pozdrowienia, Michael Lukzak
