We just started running into an issue with a couple networks in Cloudstack that
we haven’t been able to resolve yet.
The VMs inside the network had no connectivity, so we have tried restarting the
network (with and without the ‘clean up’ option), and also just restarting the
virtual router through cloudstack.
This is part of the management-server.log when we restart the network with the
cleanup option:
2014-08-22 11:25:52,909 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Checking if host: 40 has enough capacity for
requested CPU: 500 and requested RAM: 134217728 , cpuOverprovisioningFactor: 2.0
2014-08-22 11:25:52,914 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Hosts's actual total CPU: 64000 and CPU after
applying overprovisioning: 128000
2014-08-22 11:25:52,914 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Free CPU: 110000 , Requested CPU: 500
2014-08-22 11:25:52,915 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Free RAM: 82959712256 , Requested RAM: 134217728
2014-08-22 11:25:52,915 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Host has enough CPU and RAM available
2014-08-22 11:25:52,916 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Found a suitable host, adding to list: 40
2014-08-22 11:25:52,916 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Host name: hv-40-1.phx, hostId: 41 is in avoid set,
skipping this and trying other available hosts
2014-08-22 11:25:52,932 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Host: 42 has cpu capability (cpu:32, speed:2000) to
support requested CPU: 1 and requested speed: 500
2014-08-22 11:25:52,933 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Checking if host: 42 has enough capacity for
requested CPU: 500 and requested RAM: 134217728 , cpuOverprovisioningFactor: 2.0
2014-08-22 11:25:52,938 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Hosts's actual total CPU: 64000 and CPU after
applying overprovisioning: 128000
2014-08-22 11:25:52,938 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Free CPU: 91000 , Requested CPU: 500
2014-08-22 11:25:52,938 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Free RAM: 27793432576 , Requested RAM: 134217728
2014-08-22 11:25:52,939 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Host has enough CPU and RAM available
2014-08-22 11:25:52,939 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Found a suitable host, adding to list: 42
2014-08-22 11:25:52,939 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a
FirstFitRoutingAllocator) Host Allocator returning 7 suitable hosts
2014-08-22 11:25:52,943 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Checking
suitable pools for volume (Id, Type): (6539,ROOT)
2014-08-22 11:25:52,943 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Volume
has pool already allocated, checking if pool can be reused, poolId: 235
2014-08-22 11:25:52,946 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Planner
need not allocate a pool for this volume since its READY
2014-08-22 11:25:52,946 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Trying
to find a potenial host and associated storage pools from the suitable
host/pool lists for this VM
2014-08-22 11:25:52,946 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Checking
if host: 43 can access any suitable storage pool for volume: ROOT
2014-08-22 11:25:52,949 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Host: 43
cannot access pool: 235
2014-08-22 11:25:52,949 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Checking
if host: 44 can access any suitable storage pool for volume: ROOT
2014-08-22 11:25:52,951 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Host: 44
cannot access pool: 235
2014-08-22 11:25:52,951 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Checking
if host: 45 can access any suitable storage pool for volume: ROOT
2014-08-22 11:25:52,954 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Host: 45
cannot access pool: 235
2014-08-22 11:25:52,954 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Checking
if host: 46 can access any suitable storage pool for volume: ROOT
2014-08-22 11:25:52,957 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Host: 46
cannot access pool: 235
2014-08-22 11:25:52,957 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Checking
if host: 39 can access any suitable storage pool for volume: ROOT
2014-08-22 11:25:52,959 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Host: 39
cannot access pool: 235
2014-08-22 11:25:52,959 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Checking
if host: 40 can access any suitable storage pool for volume: ROOT
2014-08-22 11:25:52,962 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Host: 40
cannot access pool: 235
2014-08-22 11:25:52,962 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Checking
if host: 42 can access any suitable storage pool for volume: ROOT
2014-08-22 11:25:52,964 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Host: 42
cannot access pool: 235
2014-08-22 11:25:52,964 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Could
not find a potential host that has associated storage pools from the suitable
host/pool lists for this VM
2014-08-22 11:25:52,972 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Could
not find suitable Deployment Destination for this VM under any clusters,
returning.
2014-08-22 11:25:52,972 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a)
Searching resources only under specified Cluster: 5
2014-08-22 11:25:52,974 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) The
specified cluster is in avoid set, returning.
2014-08-22 11:25:52,994 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) VM state
transitted from :Starting to Stopped with event: OperationFailedvm's original
host id: null new host id: null host id before state transition: 41
2014-08-22 11:25:53,008 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Hosts's
actual total CPU: 64000 and CPU after applying overprovisioning: 128000
2014-08-22 11:25:53,008 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Hosts's
actual total RAM: 98928826880 and RAM after applying overprovisioning:
98928828416
2014-08-22 11:25:53,009 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) release
cpu from host: 41, old used: 36000,reserved: 0, actual total: 64000, total with
overprovisioning: 128000; new used: 35500,reserved:0; movedfromreserved:
false,moveToReserveredfalse
2014-08-22 11:25:53,009 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) release
mem from host: 41, old used: 47513075712,reserved: 0, total: 98928828416; new
used: 47378857984,reserved:0; movedfromreserved: false,moveToReserveredfalse
2014-08-22 11:25:53,016 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a)
Invocation exception, caused by:
com.cloud.utils.exception.CloudRuntimeException: Unable to start instance
'r-6420-VM' (7ecbca93-f9c9-496a-9ad7-0c57cb63789f), see management server log
for details
2014-08-22 11:25:53,017 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Unable to start
instance 'r-6420-VM' (7ecbca93-f9c9-496a-9ad7-0c57cb63789f), see management
server log for details
2014-08-22 11:25:53,018 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940) Done with run of VM
work job: com.cloud.vm.VmWorkStart for VM 6420, job origin: 156937
2014-08-22 11:25:53,018 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940) Unable to complete
AsyncJobVO {id:156940, userId: 2, accountId: 2, instanceType: null, instanceId:
null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAZFHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc3IAEWphdmEudXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA_QAAAAAAADHcIAAAAEAAAAAF0AA5SZXN0YXJ0TmV0d29ya3QAP3JPMEFCWE55QUJGcVlYWmhMbXhoYm1jdVFtOXZiR1ZoYnMwZ2NvRFZuUHJ1QWdBQldnQUZkbUZzZFdWNGNBRXhw,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 182571079363322, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Fri Aug 22 11:25:03 MST 2014}, job origin:156937
com.cloud.utils.exception.CloudRuntimeException: Unable to start instance
'r-6420-VM' (7ecbca93-f9c9-496a-9ad7-0c57cb63789f), see management server log
for details
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1137)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5190)
at sun.reflect.GeneratedMethodAccessor446.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5335)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-08-22 11:25:53,022 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940) Complete async
job-156940, jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFuZy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAjkpvYiBmYWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgJ3ItNjQyMC1WTScgKDdlY2JjYTkzLWY5YzktNDk2YS05YWQ3LTBjNTdjYjYzNzg5ZiksIHNlZSBtYW5hZ2VtZW50IHNlcnZlciBsb2cgZm9yIGRldGFpbHN1cgAeW0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAAA1zcgAbamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZGVjbGFyaW5nQ2xhc3NxAH4ABEwACGZpbGVOYW1lcQB-AARMAAptZXRob2ROYW1lcQB-AAR4cAAAAHJ0ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnNxAH4ACwAAAfd0AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRzcQB-AAsAAAAxdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5zcQB-AAsAAAA4dABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxsc3EAfgALAAAAZ3QAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AGnQAD2NhbGxXaXRoQ29udGV4dHNxAH4ACwAAADVxAH4AHXEAfgAadAAOcnVuV2l0aENvbnRleHRzcQB-AAsAAAAudAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-ABZxAH4AF3NxAH4ACwAAAcxxAH4AEXEAfgAScQB-ABdzcQB-AAsAAAHXdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-ABtzcQB-AAsAAAEGdAAfamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgAXc3EAfgALAAAEeXQAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci5qYXZhdAAJcnVuV29ya2Vyc3EAfgALAAACZ3QALmphdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4ALHEAfgAXc3EAfgALAAAC6XQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgAXc3IAJmphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZGlmaWFibGVMaXN0_A8lMbXsjhACAAFMAARsaXN0cQB-AAZ4cgAsamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWN0ABZMamF2YS91dGlsL0NvbGxlY3Rpb247eHBzcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h2Zx2GdAwABSQAEc2l6ZXhwAAAAAHcEAAAAAHhxAH4AOHg
2014-08-22 11:25:53,037 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940) Done executing
com.cloud.vm.VmWorkStart for job-156940
2014-08-22 11:25:53,068 WARN [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-95:ctx-63049a0d job-156937 ctx-54a932fe) Failed to implement
network Ntwk[00759e4d-4903-412f-bee3-0dcee9fc811c|Guest|8] elements and
resources as a part of network restart due to
java.lang.RuntimeException: Job failed due to exception Unable to start
instance 'r-6420-VM' (7ecbca93-f9c9-496a-9ad7-0c57cb63789f), see management
server log for details
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-08-22 11:25:53,070 WARN [c.c.n.NetworkServiceImpl]
(API-Job-Executor-95:ctx-63049a0d job-156937 ctx-54a932fe) Network id=1365
failed to restart.
2014-08-22 11:25:53,069 DEBUG [o.a.c.f.j.i.SyncQueueManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940) Sync queue (58718) is
currently empty
2014-08-22 11:25:53,071 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940) Remove job-156940
from job monitoring
2014-08-22 11:25:53,081 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-95:ctx-63049a0d job-156937) Complete async job-156937,
jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to restart network"}
2014-08-22 11:25:53,098 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-95:ctx-63049a0d job-156937) Done executing
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for job-156937
2014-08-22 11:25:53,104 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-95:ctx-63049a0d job-156937) Remove job-156937 from job
monitoring
In particular, this part:
2014-08-22 11:25:52,964 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Could
not find a potential host that has associated storage pools from the suitable
host/pool lists for this VM
2014-08-22 11:25:52,972 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-92:ctx-a176c26e job-156937/job-156940 ctx-2f39774a) Could
not find suitable Deployment Destination for this VM under any clusters,
returning.
All of our XenServer hypervisors use local storage, so each host has a local
storage pool, and can only access that pool. Cloudstack seems to be allocating
storage on one host for the vm, and then trying to deploy to a different host?
All hosts do have plenty of free space.
I tested deploying a normal(non virtual router/systemvm) and it is working fine
on the setup; so it might only be related to system vms.
Once the virtual router gets to this point, the VM will no longer start either
as the virtual router has to be running first.
Anyone have any advice on where to go from here? I can provide more
details/logs if needed; just not sure what is helpful at this point.
Thanks!
--
Justyn Shull
DevOps Engineer
