depending on your setup, but what I would do move all your Instances from host machine where the VR lives and reboot. Again, that's what I would do.
you can try this link for XenServer 5 http://blogs.citrix.com/2009/01/26/how-to-configure-an-nfs-based-iso-sr-using-local-storage-on-a-xenserver5-host/ Thanks, On Thu, Aug 7, 2014 at 12:17 PM, Michael Lukzak <[email protected]> wrote: > Hi, > > We are usning xenserver from Citrix. > > I found some clue. In console to server on this host I see that nfs has > error "Stale NFS file handle". > > How to remount NFS? Maybe this is a problem? > But why other VR are started and few of them not... ;( > > Best regards, > Michael > > > > > Hi Michael, > > if it helps, I would restart the agent on the host machine where VR > lives. > > > it depends on hypervisor but on KVM, following command: service > > cloudstack-agent restart > > > > Thanks, > > Motty > > > > On Thu, Aug 7, 2014 at 11:52 AM, Michael Lukzak <[email protected]> wrote: > > >> Hi, > > >> One of our host are died (after very high load). > >> Host has a lot of Virtual Routers running. > > >> We are using advanced networking. > > >> Only way to bring machine alive was hard reset. > > >> After this CloudStack decied to start all of VirtualRouterts, > >> when server was back online. > > >> Most of VR are started, but couple don't. > >> Cloudstack want to start this VM, but can't ;( > >> Here are log from this action. > > >> It's looks like that CS are looking for VR host on Xen host, but he > >> couldn't find. > > >> Please help how to repair this situation. > >> Restart network with clean up options ending with the same resoult. > > >> Is there any option to recreate VR? > >> I tried to remove VR from Infrastructure -> Virtual Router, but there is > >> still problem > >> with create new one. > > > > >> Here are logs from "restart network with clean up options" > >> WARN [network.element.VpcVirtualRouterElement] > >> (Job-Executor-151:job-6526) Network Ntwk[200|Public|1] is not associated > >> with any VPC > >> WARN [network.element.VpcVirtualRouterElement] > >> (Job-Executor-151:job-6526) Network Ntwk[338|Guest|8] is not associated > >> with any VPC > >> WARN [network.element.VpcVirtualRouterElement] > >> (Job-Executor-151:job-6526) Network Ntwk[202|Control|3] is not > associated > >> with any VPC > >> ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-151:job-6526) > >> Failed to start instance VM[DomainRouter|r-1015-VM] > >> java.lang.NullPointerException > >> at > >> > com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) > >> at > >> > com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) > >> at > >> > com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) > >> at > >> > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) > >> at > >> > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) > >> at > >> > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175) > >> at > >> > com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518) > >> at > >> > com.cloud.network.NetworkManagerImpl.restartNetwork(NetworkManagerImpl.java:2473) > >> at > >> > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1381) > >> at > >> > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > >> at > >> > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92) > >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) > >> at > >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > >> at > >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >> at java.lang.Thread.run(Thread.java:722) > >> INFO [xen.resource.CitrixResourceBase] (DirectAgent-16:) VM does not > >> exist on XenServer0d1d5015-4061-4554-96de-406155449296 > >> WARN [cloud.network.NetworkManagerImpl] (Job-Executor-151:job-6526) > >> Failed to implement network Ntwk[338|Guest|8] elements and resources as > a > >> part of network restart due to > >> com.cloud.exception.AgentUnavailableException: Resource [Host:9] is > >> unreachable: Host 9: Unable to start instance due to null > >> at > >> > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841) > >> at > >> > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) > >> at > >> > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175) > >> at > >> > com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518) > >> at > >> > com.cloud.network.NetworkManagerImpl.restartNetwork(NetworkManagerImpl.java:2473) > >> at > >> > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1381) > >> at > >> > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > >> at > >> > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92) > >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) > >> at > >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > >> at > >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >> at java.lang.Thread.run(Thread.java:722) > >> Caused by: java.lang.NullPointerException > >> at > >> > com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) > >> at > >> > com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) > >> at > >> > com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) > >> at > >> > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) > >> ... 23 more > >> WARN [cloud.network.NetworkServiceImpl] (Job-Executor-151:job-6526) > >> Network id=338 failed to restart. > >> WARN [cloud.async.AsyncJobManagerImpl] (Job-Executor-151:job-6526) > Unable > >> to unregister active job 6526 from JMX monitoring > > > >> Here are log when I want to start VR (his state is stopeed, I want to > >> start and got this) > > >> WARN [network.element.VpcVirtualRouterElement] > >> (Job-Executor-153:job-6528) Network Ntwk[200|Public|1] is not associated > >> with any VPC > >> WARN [network.element.VpcVirtualRouterElement] > >> (Job-Executor-153:job-6528) Network Ntwk[338|Guest|8] is not associated > >> with any VPC > >> WARN [network.element.VpcVirtualRouterElement] > >> (Job-Executor-153:job-6528) Network Ntwk[202|Control|3] is not > associated > >> with any VPC > >> ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-153:job-6528) > >> Failed to start instance VM[DomainRouter|r-1015-VM] > >> java.lang.NullPointerException > >> at > >> > com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) > >> at > >> > com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) > >> at > >> > com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) > >> at > >> > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) > >> at > >> > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2874) > >> at > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2824) > >> at > >> > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > >> at > >> > org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(StartRouterCmd.java:103) > >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) > >> at > >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > >> at > >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >> at java.lang.Thread.run(Thread.java:722) > > > >> Best Regards, > >> Michael > > > > -- > Pozdrowienia, > Michael Lukzak
