Hi Dan,
It is only happening with instances created from this specific template.
Also, sometimes creating an instance does not work at all, with various
errors. I just tried to recreate the template, and when I try to create
an instance from it, I get (after a minute or so):
Job failed due to exception Unable to create a deployment for
VM[User|i-2-60-VM]
In the management-server.log, I get:
com.cloud.exception.InsufficientServerCapacityException: Unable to
create a deployment for VM[User|i-2-60-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:946)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5190)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5335)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Earlier, I see
2014-08-05 12:53:57,646 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-36:ctx-ca0b66c4 job-633/job-635 ctx-0df40702)
Invocation exception, caused by:
com.cloud.exception.InsufficientServerCapacityException: Unable to
create a dep
loyment for VM[User|i-2-60-VM]Scope=interface com.cloud.dc.DataCenter; id=1
but there are lots of ressources available.
Scrolling further up, I see:
2014-08-05 12:53:57,095 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-36:ctx-ca0b66c4 job-633/job-635 ctx-0df40702
FirstFitRoutingAllocator) FirstFitAllocator has 2 hosts to check for
allocation: [Host[-5-Routing], Host[-6-Rou
ting]]
2014-08-05 12:53:57,098 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-36:ctx-ca0b66c4 job-633/job-635 ctx-0df40702
FirstFitRoutingAllocator) Found 2 hosts for allocation after
prioritization: [Host[-5-Routing], Host[-6-Routing
]]
2014-08-05 12:53:57,098 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-36:ctx-ca0b66c4 job-633/job-635 ctx-0df40702
FirstFitRoutingAllocator) Looking for speed=1000Mhz, Ram=1024
2014-08-05 12:53:57,098 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-36:ctx-ca0b66c4 job-633/job-635 ctx-0df40702
FirstFitRoutingAllocator) Host name: dev-xen01, hostId: 5 is in avoid
set, skipping this and trying other a
vailable hosts
2014-08-05 12:53:57,098 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-36:ctx-ca0b66c4 job-633/job-635 ctx-0df40702
FirstFitRoutingAllocator) Host name: dev-xen02, hostId: 6 is in avoid
set, skipping this and trying other a
vailable hosts
2014-08-05 12:53:57,098 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-36:ctx-ca0b66c4 job-633/job-635 ctx-0df40702
FirstFitRoutingAllocator) Host Allocator returning 0 suitable hosts
so both hosts are on some "avoid" list... At some other attempts, I get
less specific errors.
All in all, Cloudstack behaves "nondeterministic"... Sometimes an
operation fails, the next day, the same operation succeeds somewhat.
I also wonder why many of these error log messages are classified as
"DEBUG", which makes it hard to find the source of the problem. Did I
overlook something?
Thanks,
Martin
Am 05.08.2014 09:39, schrieb Daan Hoogland:
Martin, is the shutting down happening with the systemvms or guest vms or both?
Do you see this also when you boot a vm of the same template in xen directly?
regards,
On Tue, Aug 5, 2014 at 8:37 AM, Martin Emrich <[email protected]> wrote:
Hi!
Am 04.08.2014 21:01, schrieb Pierre-Luc Dion:
Hi Martin,
I might not be able to help you much on that, but did you look at
XenCenter
to see if you had something similar like this:
https://issues.apache.org/jira/browse/CLOUDSTACK-7225
Basically, SystemVM paused instead of remaining started or perform a
reboot.
Actually, I had this problem, too. After restarting (and possibly upgrading)
the SystemVMs with cloudstack-sysvmadm several times, I got them all
running.
All in all, this is quite "nondeterministic"...
Regards,
Martin
