Hi Daan and users,
the infrastructure is based on the Linux environment. The management
server, hosts and storage are all Ubuntu 16.04 except the new storage
server which is an Ubuntu 18.04. The hypervisor used is Qemu-kvm with NFS
to share the storage.
We tried to add another primary storage and creating a VM that would use it
we found no problems, the statistics update and no error messages appear.
Gere is an excerpt of the logs of the most complete agent:
2020-01-14 09:01:45,749 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:c3851d3a) Trying to fetch storage pool
171e90f4-511e-3b10-9310-b9eec0094be6 from libvirt
2020-01-14 09:01:45,752 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:c3851d3a) Asking libvirt to refresh
storage pool 171e90f4-511e-3b10-9310-b9eec0094be6
2020-01-14 09:01:46,641 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:c3851d3a) Trying to fetch storage pool
9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
2020-01-14 09:01:46,643 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:c3851d3a) Asking libvirt to refresh
storage pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
2020-01-14 09:05:51,529 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:2765ff88) Trying to fetch storage pool
9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
2020-01-14 09:05:51,532 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:2765ff88) Asking libvirt to refresh
storage pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
2020-01-14 09:10:47,286 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:null) (logid:6d27b740) Trying to fetch storage pool
9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
2020-01-14 09:10:47,419 WARN [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:6d27b740) Caught:
com.cloud.utils.exception.CloudRuntimeException: Can't find
volume:d93d3c0a-3859-4473-951d-9b5c5912c767
at
com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
at com.cloud.agent.Agent.processRequest(Agent.java:645)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-01-14 09:20:48,390 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ec72387b) Trying to fetch storage pool
9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
2020-01-14 09:20:48,536 WARN [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ec72387b) Caught:
com.cloud.utils.exception.CloudRuntimeException: Can't find
volume:d93d3c0a-3859-4473-951d-9b5c5912c767
at
com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
at com.cloud.agent.Agent.processRequest(Agent.java:645)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-01-14 09:25:15,259 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:1a7e082e) Trying to fetch storage pool
9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
2020-01-14 09:25:15,261 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:1a7e082e) Asking libvirt to refresh
storage pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
And here the management server log:
2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-2:null) (logid:) Seq 15-705657766613619075:
Processing: { Ans: , MgmtId: 220777304233416, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
Can't find volume:d93d3c0a-3859-4473-951d-9b5c5912c767\n\tat
com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)\n\tat
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)\n\tat
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)\n\tat
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)\n\tat
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)\n\tat
com.cloud.agent.Agent.processRequest(Agent.java:645)\n\tat
com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)\n\tat
com.cloud.utils.nio.Task.call(Task.java:83)\n\tat
com.cloud.utils.nio.Task.call(Task.java:29)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
java.lang.Thread.run(Thread.java:748)\n","wait":0}}] }
2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request]
(StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Seq 15-705657766613619075:
Received: { Ans: , MgmtId: 220777304233416, via: 15(csdell017), Ver: v1,
Flags: 10, { Answer } }
2020-01-14 09:21:27,105 DEBUG [c.c.a.m.AgentManagerImpl]
(StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Details from executing
class com.cloud.agent.api.GetVolumeStatsCommand:
com.cloud.utils.exception.CloudRuntimeException: Can't find
volume:d93d3c0a-3859-4473-951d-9b5c5912c767
at
com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
at com.cloud.agent.Agent.processRequest(Agent.java:645)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
on 09/01/20 12:58, Daan Hoogland wrote:
Charlie, I think you'll have to explain a bit more about your environment
to get an answer. what type of storage is it? Where did you migrate the VM
from and to? What types() of hypervisors are you using? Though saying **the**
agent logs suggests KVM, you are still leaving people guessing a lot.
