Hi,
I am running ACS 4.8.0. One cluster with two xenserver 6.5 hosts.
One of the VM's network went unreachable so when I checked VR it was unable to
reach gateway. So I tried to restart VR and did cleanup. But issue is still
same. When it rebooted VR, I saw ping command to gateway which ending with this
,
PING 123.23.24.25 (123.23.24.25): 48 data bytes
/etc/init.d/cloud-early-config: line 574: 2009 Terminated ping -n
-c 3 $GW
But then it did give root login prompt on console. But problem is I am unable
to login to VR anymore since cloudstack just shutdown VR vm from xen and I
can't even log into VR console so I can't check more logs.
Cloudstack shut down VR automatically with this mail alert -
Command: com.cloud.agent.api.Command failed while starting virtual router
Details: Timed out in waiting SSH execution result com.cloud.agent.api.Answer
And on cloudstack management logs I get this error,
2016-08-24 14:32:41,548 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.router.RebootRouterCmd
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to Unable to start
VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not
retrying
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1120)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.cloud.utils.exception.ExecutionException: Unable to start
VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not
retrying
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1084)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
... 18 more
2016-08-24 14:32:41,550 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Complete async
job-17880, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
[Host:1] is unreachable: Host 1: Unable to start instance due to Unable to
start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart,
not retrying"}
2016-08-24 14:32:41,553 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Publish async
job-17880 complete on message bus
Network I am trying to get up is production network, so I need it up asap. What
could be the problem here?
I tried to create new VR, and that seems to reach gateway. Its problem with
only this VR. What if I destroy router, from cloudstack panel? Would it
recreate VR automatically? Or any other suggestion on this issue?
/sonali
