We've got issues deploying successfully to either of our hosts in our 4.2.0
cluster. Instances either fail immediately or take a long time to get
started.
Most of the errors in our logs point to secondary storage, but I can mount
and unmount successfully. In both cases I have stopped libvirt,
cloudstack-agent and rebooted but to no avail.
Any help much appreciated.
Nick
>From the agent.log on the host that works some of the time, very slowly.
2013-11-22 08:56:47,559 ERROR [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) org.libvirt.LibvirtException: Requested
operation is not valid: Target
'/mnt/05648bab-b05a-377e-a3a7-52e616d5e390' is already mounted
2013-11-22 08:55:47,464 ERROR [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Target was already mounted, unmounted it
but failed to redefine storage:org.libvirt.LibvirtException: Requested
operation is not valid: Target
'/mnt/05648bab-b05a-377e-a3a7-52e616d5e390' is already mounted
2013-11-22 08:55:47,463 ERROR [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Succeeded in unmounting
/mnt/05648bab-b05a-377e-a3a7-52e616d5e390
2013-11-22 08:55:47,465 WARN [cloud.agent.Agent]
(agentRequest-Handler-4:null) Caught: java.lang.NullPointerException
at
com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:539)
at
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:185)
at
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:108)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2466)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1230)
at com.cloud.agent.Agent.processRequest(Agent.java:525)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
at com.cloud.utils.nio.Task.run(Task.java:83)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
And from the libvirt log on the host that we can't deploy to at all.
2013-11-22 15:12:55.316+0000: 6101: warning :
virStorageFileGetMetadataInternal:782 : Backing file
'/mnt/05648bab-b05a-377e-a3a7-52e616d5e390/f86a000d-aa46-41df-9bcb-4cba11d33b19'
of image
'/mnt/05648bab-b05a-377e-a3a7-52e616d5e390/cfd2c776-5505-4d1c-8f22-4e7824154ddb'
is missing.
2013-11-22 15:12:55.316+0000: 6101: error : virFindBackingFile:528 :
Can't canonicalize path
'/mnt/05648bab-b05a-377e-a3a7-52e616d5e390/f86a000d-aa46-41df-9bcb-4cba11d33b19':
No such file or directory
