sounds good Charlie, I hope you get a chance to upgrade soon. On Mon, Feb 17, 2020 at 10:55 AM Charlie Holeowsky < [email protected]> wrote:
> Hi Daan and Users, > I opened the issue on Github and found that this bug (#3181) was fixed on > 4.13.0 release. > > Il giorno lun 3 feb 2020 alle ore 10:25 Daan Hoogland < > [email protected]> ha scritto: > >> thanks Charlie, can you take your issue to a github issue on the project >> please? thanks >> >> On Mon, Feb 3, 2020 at 9:57 AM Charlie Holeowsky < >> [email protected]> wrote: >> >>> Hi Daan and users, >>> Yes, I resolved the issue but it's only a work-around, I have found that >>> the problem is reproducible but I do not understand if I am the only one to >>> have noticed it and if there is another more elegant and definitive >>> solution. >>> >>> >>> >>> Il giorno ven 31 gen 2020 alle ore 13:31 Daan Hoogland < >>> [email protected]> ha scritto: >>> >>>> sorry to not have any focus on this Charlie, >>>> Do I read correctly that you resolved your issue? >>>> >>>> On Tue, Jan 28, 2020 at 2:46 PM Charlie Holeowsky < >>>> [email protected]> wrote: >>>> >>>>> In this period I have performed some testsand I found a workarount for >>>>> the metrics problem. >>>>> I created a test environment in the laboratory with the main >>>>> characteristics equal to that of production (acs 4.11.2.0, all Ubuntu >>>>> 16.04 >>>>> OS, KVM, NFS as shared storage and advanced network). Then I added a >>>>> ubuntu >>>>> 18.04 new primary storage. >>>>> >>>>> I create a new VM in the new storage server and and after a while the >>>>> metrics appeared as on the first storage, so the storage is working. >>>>> >>>>> I destroyed this VM, I create a new one on the first (old) storage and >>>>> then I migrate it on the new storage. >>>>> >>>>> After migrating the disk from the first primary storage to the second >>>>> one i have encountered the same problem (same error on agent.log >>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find >>>>> volume:5a521eb0...) and the volume metrics and metrics didn't appear or >>>>> update. >>>>> >>>>> In this test environment I created a symbolic link in the filesystem >>>>> of the storage server taking the name of the disk just migrated (the one >>>>> with the name path) and the uuid name (the one that appear in error >>>>> message). >>>>> >>>>> >>>>> Here is an example to better explain me. >>>>> >>>>> I took the list of volumes where path is different from uuid which >>>>> returns the data of the migrated volumes: >>>>> >>>>> mysql> select id,uuid,path from volumes where uuid!=path; >>>>> >>>>> +----+--------------------------------------+--------------------------------------+ >>>>> | id | uuid | path >>>>> | >>>>> >>>>> +----+--------------------------------------+--------------------------------------+ >>>>> | 10 | 5a521eb0-266f-4353-b4b2-1d63a483e5b5 | >>>>> 165b92ba-68f1-4172-be35-bbe1d032cb7c | >>>>> | 12 | acb3bb29-9bac-4a2a-aefa-3c6ac1c2846b | >>>>> 56aa3961-dbc2-4f98-9246-a7497eef3214 | >>>>> >>>>> +----+--------------------------------------+--------------------------------------+ >>>>> >>>>> In the storage server I make the symbolic links (ln -s <path> <uuid>): >>>>> >>>>> # ln >>>>> -s 5a521eb0-266f-4353-b4b2-1d63a483e5b5 >>>>> 165b92ba-68f1-4172-be35-bbe1d032cb7c >>>>> # ln -s acb3bb29-9bac-4a2a-aefa-3c6ac1c2846b >>>>> 56aa3961-dbc2-4f98-9246-a7497eef3214 >>>>> >>>>> After doing this, I waited some time and then I found that metrics >>>>> were updated and the message in agent.log no longer appeared. >>>>> >>>>> >>>>> Il giorno gio 23 gen 2020 alle ore 17:53 Charlie Holeowsky < >>>>> [email protected]> ha scritto: >>>>> >>>>>> I still don't understand >>>>>> why com.cloud.hypervisor.kvm.storage.LibvirtStoragePool don't find the >>>>>> volume d93d3c0a-3859-4473-951d-9b5c5912c76 that exists as file >>>>>> 39148fe1-842b-433a-8a7f-85e90f316e04... >>>>>> >>>>>> It's the only anomaly I have found. Where can I look again? >>>>>> >>>>>> Il giorno lun 20 gen 2020 alle ore 16:27 Daan Hoogland < >>>>>> [email protected]> ha scritto: >>>>>> >>>>>>> but the record you send earlier also says that is should be looking >>>>>>> for 39148fe1-842b-433a-8a7f-85e90f316e04, in the path field. the message >>>>>>> might be just that, a message. >>>>>>> >>>>>>> On Mon, Jan 20, 2020 at 3:35 PM Charlie Holeowsky < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> I think that's the problem because in the logs that I have >>>>>>>> forwarded it reads: >>>>>>>> Can't find volume: d93d3c0a-3859-4473-951d-9b5c5912c767 >>>>>>>> >>>>>>>> This is the volume ID of migrated file but it do not exist on >>>>>>>> primary storage (new or old one) but it exist >>>>>>>> as 39148fe1-842b-433a-8a7f-85e90f316e04. >>>>>>>> >>>>>>>> Il giorno lun 20 gen 2020 alle ore 12:35 Daan Hoogland < >>>>>>>> [email protected]> ha scritto: >>>>>>>> >>>>>>>>> also, can you see the primary storage being mounted? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Jan 20, 2020 at 12:33 PM Daan Hoogland < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Why do you think that Charlie? Is it in the logs like that >>>>>>>>>> somewhere? >>>>>>>>>> >>>>>>>>>> On Mon, Jan 20, 2020 at 9:52 AM Charlie Holeowsky < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Daan, >>>>>>>>>>> in fact I find the volume file >>>>>>>>>>> (39148fe1-842b-433a-8a7f-85e90f316e04) in the repositry id = 3 (the >>>>>>>>>>> new >>>>>>>>>>> one) but it seems to me that the cloudstack system goes looking for >>>>>>>>>>> the >>>>>>>>>>> volume with its "old" name (path) that doesn't exist... >>>>>>>>>>> >>>>>>>>>>> Il giorno sab 18 gen 2020 alle ore 21:41 Daan Hoogland < >>>>>>>>>>> [email protected]> ha scritto: >>>>>>>>>>> >>>>>>>>>>>> Charlie, >>>>>>>>>>>> forgive my not replying in a timely manner. This might happen >>>>>>>>>>>> if the disk was migrated. In this case probably from primary >>>>>>>>>>>> storage with >>>>>>>>>>>> id 1 to the one with id 3. the second record (pool_id 1) is >>>>>>>>>>>> removed, so you >>>>>>>>>>>> can ignore that one. The first seems legit. You should be able to >>>>>>>>>>>> find that >>>>>>>>>>>> disks on your primary storage with id 3. >>>>>>>>>>>> hope this helps. >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jan 16, 2020 at 2:07 PM Charlie Holeowsky < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Daan and users, >>>>>>>>>>>>> to better explain I show you the two records related to the >>>>>>>>>>>>> disk that generates the error message. >>>>>>>>>>>>> >>>>>>>>>>>>> In the first query there are the data of the disk currently in >>>>>>>>>>>>> use which has the "uuid" equal to the name searched by the >>>>>>>>>>>>> procedure >>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDiskand >>>>>>>>>>>>> a >>>>>>>>>>>>> has a different "path" field. >>>>>>>>>>>>> >>>>>>>>>>>>> In the second one we note that the "path" field is equal to >>>>>>>>>>>>> the "uuid" of the volume in use but has null "uuid" (and >>>>>>>>>>>>> state=Expunged). >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> mysql> select * from volumes where >>>>>>>>>>>>> uuid='d93d3c0a-3859-4473-951d-9b5c5912c767'; >>>>>>>>>>>>> >>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+ >>>>>>>>>>>>> | id | account_id | domain_id | pool_id | last_pool_id | >>>>>>>>>>>>> instance_id | device_id | name | uuid >>>>>>>>>>>>> | >>>>>>>>>>>>> size | folder | path >>>>>>>>>>>>> | >>>>>>>>>>>>> pod_id | data_center_id | iscsi_name | host_ip | volume_type | >>>>>>>>>>>>> pool_type | >>>>>>>>>>>>> disk_offering_id | template_id | first_snapshot_backup_uuid | >>>>>>>>>>>>> recreatable | >>>>>>>>>>>>> created | attached | updated | removed | >>>>>>>>>>>>> state | >>>>>>>>>>>>> chain_info | update_count | disk_type | vm_snapshot_chain_size | >>>>>>>>>>>>> iso_id | >>>>>>>>>>>>> display_volume | format | min_iops | max_iops | hv_ss_reserve | >>>>>>>>>>>>> provisioning_type | >>>>>>>>>>>>> >>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+ >>>>>>>>>>>>> | 213 | 2 | 1 | 3 | 1 | >>>>>>>>>>>>> 148 | 1 | DATA-148 | >>>>>>>>>>>>> d93d3c0a-3859-4473-951d-9b5c5912c767 | >>>>>>>>>>>>> 53687091200 | /srv/primary | 39148fe1-842b-433a-8a7f-85e90f316e04 >>>>>>>>>>>>> | NULL >>>>>>>>>>>>> | 1 | NULL | NULL | DATADISK | NULL >>>>>>>>>>>>> | >>>>>>>>>>>>> 34 | NULL | NULL | 0 >>>>>>>>>>>>> | >>>>>>>>>>>>> 2019-11-26 10:41:46 | NULL | 2019-11-26 10:41:50 | NULL | >>>>>>>>>>>>> Ready | >>>>>>>>>>>>> NULL | 2 | NULL | NULL | >>>>>>>>>>>>> NULL | >>>>>>>>>>>>> 1 | QCOW2 | NULL | NULL | NULL | thin >>>>>>>>>>>>> | >>>>>>>>>>>>> >>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+ >>>>>>>>>>>>> 1 row in set (0.00 sec) >>>>>>>>>>>>> >>>>>>>>>>>>> mysql> select * from volumes where >>>>>>>>>>>>> path='d93d3c0a-3859-4473-951d-9b5c5912c767'; >>>>>>>>>>>>> >>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+ >>>>>>>>>>>>> | id | account_id | domain_id | pool_id | last_pool_id | >>>>>>>>>>>>> instance_id | device_id | name | uuid | size | folder >>>>>>>>>>>>> | path >>>>>>>>>>>>> | pod_id | data_center_id | >>>>>>>>>>>>> iscsi_name | >>>>>>>>>>>>> host_ip | volume_type | pool_type | disk_offering_id | >>>>>>>>>>>>> template_id | >>>>>>>>>>>>> first_snapshot_backup_uuid | recreatable | created | >>>>>>>>>>>>> attached | >>>>>>>>>>>>> updated | removed | state | chain_info >>>>>>>>>>>>> | >>>>>>>>>>>>> update_count | disk_type | vm_snapshot_chain_size | iso_id | >>>>>>>>>>>>> display_volume >>>>>>>>>>>>> | format | min_iops | max_iops | hv_ss_reserve | >>>>>>>>>>>>> provisioning_type | >>>>>>>>>>>>> >>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+ >>>>>>>>>>>>> | 212 | 2 | 1 | 1 | NULL | >>>>>>>>>>>>> 148 | 1 | DATA-148 | NULL | 53687091200 | NULL | >>>>>>>>>>>>> d93d3c0a-3859-4473-951d-9b5c5912c767 | NULL | 1 | >>>>>>>>>>>>> NULL >>>>>>>>>>>>> | NULL | DATADISK | NULL | 34 | >>>>>>>>>>>>> NULL | NULL >>>>>>>>>>>>> | 0 | 2019-11-26 10:38:23 | NULL >>>>>>>>>>>>> | >>>>>>>>>>>>> 2019-11-26 10:41:50 | 2019-11-26 10:41:50 | Expunged | NULL >>>>>>>>>>>>> | >>>>>>>>>>>>> 8 | NULL | NULL | NULL | >>>>>>>>>>>>> 1 | QCOW2 >>>>>>>>>>>>> | NULL | NULL | NULL | thin | >>>>>>>>>>>>> >>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+ >>>>>>>>>>>>> 1 row in set (0.00 sec) >>>>>>>>>>>>> >>>>>>>>>>>>> Il giorno mar 14 gen 2020 alle ore 15:05 Daan Hoogland < >>>>>>>>>>>>> [email protected]> ha scritto: >>>>>>>>>>>>> >>>>>>>>>>>>>> So Charlie, >>>>>>>>>>>>>> d93d3c0a-3859-4473-951d-9b5c5912c767 is actually a valid >>>>>>>>>>>>>> disk? does it exist on the backend nfs? >>>>>>>>>>>>>> and the pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c does it >>>>>>>>>>>>>> exist both in cloudstack and on the backend? >>>>>>>>>>>>>> >>>>>>>>>>>>>> if both are answered with yes, you probably have a >>>>>>>>>>>>>> permissions issue, which might be in the network. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Jan 14, 2020 at 10:21 AM Charlie Holeowsky < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Daan and users, >>>>>>>>>>>>>>> the infrastructure is based on the Linux environment. The >>>>>>>>>>>>>>> management server, hosts and storage are all Ubuntu 16.04 >>>>>>>>>>>>>>> except the new >>>>>>>>>>>>>>> storage server which is an Ubuntu 18.04. The hypervisor used is >>>>>>>>>>>>>>> Qemu-kvm >>>>>>>>>>>>>>> with NFS to share the storage. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We tried to add another primary storage and creating a VM >>>>>>>>>>>>>>> that would use it we found no problems, the statistics update >>>>>>>>>>>>>>> and no error >>>>>>>>>>>>>>> messages appear. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Gere is an excerpt of the logs of the most complete agent: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2020-01-14 09:01:45,749 INFO >>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] >>>>>>>>>>>>>>> (agentRequest-Handler-2:null) >>>>>>>>>>>>>>> (logid:c3851d3a) Trying to fetch storage pool >>>>>>>>>>>>>>> 171e90f4-511e-3b10-9310-b9eec0094be6 from libvirt >>>>>>>>>>>>>>> 2020-01-14 09:01:45,752 INFO >>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] >>>>>>>>>>>>>>> (agentRequest-Handler-2:null) >>>>>>>>>>>>>>> (logid:c3851d3a) Asking libvirt to refresh storage pool >>>>>>>>>>>>>>> 171e90f4-511e-3b10-9310-b9eec0094be6 >>>>>>>>>>>>>>> 2020-01-14 09:01:46,641 INFO >>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] >>>>>>>>>>>>>>> (agentRequest-Handler-4:null) >>>>>>>>>>>>>>> (logid:c3851d3a) Trying to fetch storage pool >>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt >>>>>>>>>>>>>>> 2020-01-14 09:01:46,643 INFO >>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] >>>>>>>>>>>>>>> (agentRequest-Handler-4:null) >>>>>>>>>>>>>>> (logid:c3851d3a) Asking libvirt to refresh storage pool >>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c >>>>>>>>>>>>>>> 2020-01-14 09:05:51,529 INFO >>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] >>>>>>>>>>>>>>> (agentRequest-Handler-1:null) >>>>>>>>>>>>>>> (logid:2765ff88) Trying to fetch storage pool >>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt >>>>>>>>>>>>>>> 2020-01-14 09:05:51,532 INFO >>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] >>>>>>>>>>>>>>> (agentRequest-Handler-1:null) >>>>>>>>>>>>>>> (logid:2765ff88) Asking libvirt to refresh storage pool >>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c >>>>>>>>>>>>>>> 2020-01-14 09:10:47,286 INFO >>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] >>>>>>>>>>>>>>> (agentRequest-Handler-3:null) >>>>>>>>>>>>>>> (logid:6d27b740) Trying to fetch storage pool >>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt >>>>>>>>>>>>>>> 2020-01-14 09:10:47,419 WARN [cloud.agent.Agent] >>>>>>>>>>>>>>> (agentRequest-Handler-3:null) (logid:6d27b740) Caught: >>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find >>>>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767 >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450) >>>>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083) >>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83) >>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29) >>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748) >>>>>>>>>>>>>>> 2020-01-14 09:20:48,390 INFO >>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] >>>>>>>>>>>>>>> (agentRequest-Handler-4:null) >>>>>>>>>>>>>>> (logid:ec72387b) Trying to fetch storage pool >>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt >>>>>>>>>>>>>>> 2020-01-14 09:20:48,536 WARN [cloud.agent.Agent] >>>>>>>>>>>>>>> (agentRequest-Handler-4:null) (logid:ec72387b) Caught: >>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find >>>>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767 >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450) >>>>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083) >>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83) >>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29) >>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748) >>>>>>>>>>>>>>> 2020-01-14 09:25:15,259 INFO >>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] >>>>>>>>>>>>>>> (agentRequest-Handler-5:null) >>>>>>>>>>>>>>> (logid:1a7e082e) Trying to fetch storage pool >>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt >>>>>>>>>>>>>>> 2020-01-14 09:25:15,261 INFO >>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] >>>>>>>>>>>>>>> (agentRequest-Handler-5:null) >>>>>>>>>>>>>>> (logid:1a7e082e) Asking libvirt to refresh storage pool >>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> And here the management server log: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request] >>>>>>>>>>>>>>> (AgentManager-Handler-2:null) (logid:) Seq >>>>>>>>>>>>>>> 15-705657766613619075: >>>>>>>>>>>>>>> Processing: { Ans: , MgmtId: 220777304233416, via: 15, Ver: >>>>>>>>>>>>>>> v1, Flags: 10, >>>>>>>>>>>>>>> [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: >>>>>>>>>>>>>>> Can't find volume:d93d3c0a-3859-4473-951d-9b5c5912c767\n\tat >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)\n\tat >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)\n\tat >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)\n\tat >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)\n\tat >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)\n\tat >>>>>>>>>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:645)\n\tat >>>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)\n\tat >>>>>>>>>>>>>>> com.cloud.utils.nio.Task.call(Task.java:83)\n\tat >>>>>>>>>>>>>>> com.cloud.utils.nio.Task.call(Task.java:29)\n\tat >>>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat >>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat >>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat >>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)\n","wait":0}}] } >>>>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request] >>>>>>>>>>>>>>> (StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Seq >>>>>>>>>>>>>>> 15-705657766613619075: >>>>>>>>>>>>>>> Received: { Ans: , MgmtId: 220777304233416, via: >>>>>>>>>>>>>>> 15(csdell017), Ver: v1, >>>>>>>>>>>>>>> Flags: 10, { Answer } } >>>>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.m.AgentManagerImpl] >>>>>>>>>>>>>>> (StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Details from >>>>>>>>>>>>>>> executing >>>>>>>>>>>>>>> class com.cloud.agent.api.GetVolumeStatsCommand: >>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find >>>>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767 >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450) >>>>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083) >>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83) >>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29) >>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> on 09/01/20 12:58, Daan Hoogland wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Charlie, I think you'll have to explain a bit more about your >>>>>>>>>>>>>>> environment >>>>>>>>>>>>>>> to get an answer. what type of storage is it? Where did you >>>>>>>>>>>>>>> migrate the VM >>>>>>>>>>>>>>> from and to? What types() of hypervisors are you using? Though >>>>>>>>>>>>>>> saying **the** >>>>>>>>>>>>>>> agent logs suggests KVM, you are still leaving people guessing >>>>>>>>>>>>>>> a lot. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Daan >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Daan >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Daan >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Daan >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Daan >>>>>>> >>>>>> >>>> >>>> -- >>>> Daan >>>> >>> >> >> -- >> Daan >> > -- Daan
