sureshanaparti commented on PR #11200:
URL: https://github.com/apache/cloudstack/pull/11200#issuecomment-3127422030
Verified the fix with the below steps.
- Enabled local storage in the zone (restart cloudstack-management).
- Create a compute offering for local storage.
- Deploy an instance on the local storage (using the above offering).
- Restart the agent in KVM host, Stop and Start the instance deployed above.
Before FIX => Local pools removed after agent restart
MS:
```
2025-07-28 11:03:31,829 DEBUG [c.c.s.StorageManagerImpl]
(AgentConnectTaskPool-5:[ctx-8e591dc2]) (logid:b972c864) Removing pool
StoragePool
{"id":3,"name":"pr10149-t13934-kvm-ol8-kvm1-local-b4c53d3b","poolType":"Filesystem","uuid":"b4c53d3b-1006-4b7f-9e14-fcdb28e37cfc"}
from host Host
{"id":1,"name":"pr10149-t13934-kvm-ol8-kvm1","type":"Routing","uuid":"80d82041-269f-4838-b98f-1e4a7d4ec9a5"}
2025-07-28 11:03:31,832 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentConnectTaskPool-5:[ctx-8e591dc2]) (logid:b972c864) Wait time setting on
com.cloud.agent.api.DeleteStoragePoolCommand is 1800 seconds
...
2025-07-28 11:03:35,003 DEBUG [c.c.s.StorageManagerImpl]
(AgentConnectTaskPool-6:[ctx-3f61b71e]) (logid:161e6c57) Removing pool
StoragePool
{"id":4,"name":"pr10149-t13934-kvm-ol8-kvm2-local-cc67e53b","poolType":"Filesystem","uuid":"cc67e53b-d383-47c4-8a6a-e95992f477fb"}
from host Host
{"id":2,"name":"pr10149-t13934-kvm-ol8-kvm2","type":"Routing","uuid":"9a3d8e48-777a-4ba8-8b3c-4036ffe6e9ea"}
2025-07-28 11:03:35,006 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentConnectTaskPool-6:[ctx-3f61b71e]) (logid:161e6c57) Wait time setting on
com.cloud.agent.api.DeleteStoragePoolCommand is 1800 seconds
```
```
mysql> SELECT uuid, name, pool_type, scope, storage_provider_name,
host_address, path, status FROM cloud.storage_pool WHERE removed IS NULL;
+--------------------------------------+--------------------------------------------+-------------------+---------+-----------------------+--------------+---------------------------------------------------------------------+--------+
| uuid | name
| pool_type | scope | storage_provider_name | host_address |
path | status |
+--------------------------------------+--------------------------------------------+-------------------+---------+-----------------------+--------------+---------------------------------------------------------------------+--------+
| 89d82eb8-a6f3-39d5-8316-b0c96974fe09 | pr10149-t13934-kvm-ol8-kvm-pri1
| NetworkFilesystem | CLUSTER | DefaultPrimary | 10.0.32.4 |
/acs/primary/pr10149-t13934-kvm-ol8/pr10149-t13934-kvm-ol8-kvm-pri1 | Up |
| 5bd12560-426a-30cf-b5c8-dce10adf660f | pr10149-t13934-kvm-ol8-kvm-pri2
| NetworkFilesystem | CLUSTER | DefaultPrimary | 10.0.32.4 |
/acs/primary/pr10149-t13934-kvm-ol8/pr10149-t13934-kvm-ol8-kvm-pri2 | Up |
| b4c53d3b-1006-4b7f-9e14-fcdb28e37cfc |
pr10149-t13934-kvm-ol8-kvm1-local-b4c53d3b | Filesystem | HOST |
DefaultPrimary | 10.0.34.122 | /var/lib/libvirt/images
| Up |
| cc67e53b-d383-47c4-8a6a-e95992f477fb |
pr10149-t13934-kvm-ol8-kvm2-local-cc67e53b | Filesystem | HOST |
DefaultPrimary | 10.0.35.31 | /var/lib/libvirt/images
| Up |
+--------------------------------------+--------------------------------------------+-------------------+---------+-----------------------+--------------+---------------------------------------------------------------------+--------+
4 rows in set (0.00 sec)
```
KVM Host 1:
```
2025-07-28 11:03:31,881 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-2:[]) (logid:b972c864) Processing command:
com.cloud.agent.api.DeleteStoragePoolCommand
2025-07-28 11:03:31,881 INFO [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-2:[]) (logid:b972c864) Attempting to remove storage pool
b4c53d3b-1006-4b7f-9e14-fcdb28e37cfc from libvirt
```
```
Before Agent Restart:
[root@pr10149-t13934-kvm-ol8-kvm1 ~]# virsh pool-list
Name State Autostart
------------------------------------------------------------
5bd12560-426a-30cf-b5c8-dce10adf660f active no
89d82eb8-a6f3-39d5-8316-b0c96974fe09 active no
b4c53d3b-1006-4b7f-9e14-fcdb28e37cfc active no
After Agent Restart:
[root@pr10149-t13934-kvm-ol8-kvm1 ~]# virsh pool-list
Name State Autostart
------------------------------------------------------------
5bd12560-426a-30cf-b5c8-dce10adf660f active no
89d82eb8-a6f3-39d5-8316-b0c96974fe09 active no
```
KVM Host 2:
```
2025-07-28 11:03:35,056 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-2:[]) (logid:161e6c57) Processing command:
com.cloud.agent.api.DeleteStoragePoolCommand
2025-07-28 11:03:35,057 INFO [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-2:[]) (logid:161e6c57) Attempting to remove storage pool
cc67e53b-d383-47c4-8a6a-e95992f477fb from libvirt
...
2025-07-28 11:03:43,991 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-5:[]) (logid:4e7c780c) Processing command:
com.cloud.agent.api.GetStorageStatsCommand
2025-07-28 11:03:43,991 INFO [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-5:[]) (logid:4e7c780c) Trying to fetch storage pool
cc67e53b-d383-47c4-8a6a-e95992f477fb from libvirt
2025-07-28 11:03:43,991 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-5:[]) (logid:4e7c780c) Looking for libvirtd connection
at: qemu:///system
2025-07-28 11:03:43,993 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-5:[]) (logid:4e7c780c) Could not find storage pool
cc67e53b-d383-47c4-8a6a-e95992f477fb in libvirt
...
2025-07-28 11:07:52,327 INFO [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-1:[]) (logid:6ad357e8) Trying to fetch storage pool
cc67e53b-d383-47c4-8a6a-e95992f477fb from libvirt
2025-07-28 11:07:52,327 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-1:[]) (logid:6ad357e8) Looking for libvirtd connection
at: qemu:///system
2025-07-28 11:07:52,328 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-1:[]) (logid:6ad357e8) Could not find storage pool
cc67e53b-d383-47c4-8a6a-e95992f477fb in libvirt
2025-07-28 11:07:52,331 DEBUG [kvm.storage.KVMStoragePoolManager]
(AgentRequest-Handler-1:[]) (logid:6ad357e8) Failed to find
volume:8ae0dc00-3299-4e65-967a-f0e6fe97fa32 due to
com.cloud.utils.exception.CloudRuntimeException: Could not fetch storage pool
cc67e53b-d383-47c4-8a6a-e95992f477fb from libvirt due to
org.libvirt.LibvirtException: Storage pool not found: no storage pool with
matching uuid 'cc67e53b-d383-47c4-8a6a-e95992f477fb', retry:0
...
2025-07-28 11:12:16,505 INFO [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-1:[]) (logid:6ad357e8) Trying to fetch storage pool
cc67e53b-d383-47c4-8a6a-e95992f477fb from libvirt
2025-07-28 11:12:16,505 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-1:[]) (logid:6ad357e8) Looking for libvirtd connection
at: qemu:///system
2025-07-28 11:12:16,506 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-1:[]) (logid:6ad357e8) Could not find storage pool
cc67e53b-d383-47c4-8a6a-e95992f477fb in libvirt
2025-07-28 11:12:16,507 DEBUG [kvm.storage.KVMStoragePoolManager]
(AgentRequest-Handler-1:[]) (logid:6ad357e8) Failed to find
volume:8ae0dc00-3299-4e65-967a-f0e6fe97fa32 due to
com.cloud.utils.exception.CloudRuntimeException: Could not fetch storage pool
cc67e53b-d383-47c4-8a6a-e95992f477fb from libvirt due to
org.libvirt.LibvirtException: Storage pool not found: no storage pool with
matching uuid 'cc67e53b-d383-47c4-8a6a-e95992f477fb', retry:88
```
```
Before Agent Restart:
[root@pr10149-t13934-kvm-ol8-kvm2 ~]# virsh pool-list
Name State Autostart
------------------------------------------------------------
5bd12560-426a-30cf-b5c8-dce10adf660f active no
89d82eb8-a6f3-39d5-8316-b0c96974fe09 active no
cc67e53b-d383-47c4-8a6a-e95992f477fb active no
After Agent Restart:
[root@pr10149-t13934-kvm-ol8-kvm2 ~]# virsh pool-list
Name State Autostart
------------------------------------------------------------
5bd12560-426a-30cf-b5c8-dce10adf660f active no
89d82eb8-a6f3-39d5-8316-b0c96974fe09 active no
```
After FIX => Local storage pools are not removed during agent restart
MS:
```
mysql> SELECT uuid, name, pool_type, scope, storage_provider_name,
host_address, path, status FROM cloud.storage_pool WHERE removed IS NULL;
+--------------------------------------+--------------------------------------------+-------------------+---------+-----------------------+--------------+---------------------------------------------------------------------+--------+
| uuid | name
| pool_type | scope | storage_provider_name | host_address |
path | status |
+--------------------------------------+--------------------------------------------+-------------------+---------+-----------------------+--------------+---------------------------------------------------------------------+--------+
| 2aa7a529-b09e-305d-b345-c4bf3c561842 | pr11200-t13938-kvm-ol8-kvm-pri1
| NetworkFilesystem | CLUSTER | DefaultPrimary | 10.0.32.4 |
/acs/primary/pr11200-t13938-kvm-ol8/pr11200-t13938-kvm-ol8-kvm-pri1 | Up |
| 618a1bab-95a5-348b-9133-c355693e01fd | pr11200-t13938-kvm-ol8-kvm-pri2
| NetworkFilesystem | CLUSTER | DefaultPrimary | 10.0.32.4 |
/acs/primary/pr11200-t13938-kvm-ol8/pr11200-t13938-kvm-ol8-kvm-pri2 | Up |
| 37044676-7f87-4b35-b83f-8af70e8748c6 |
pr11200-t13938-kvm-ol8-kvm2-local-37044676 | Filesystem | HOST |
DefaultPrimary | 10.0.34.180 | /var/lib/libvirt/images
| Up |
| 11aea44c-8240-4067-9151-84861ce00d3d |
pr11200-t13938-kvm-ol8-kvm1-local-11aea44c | Filesystem | HOST |
DefaultPrimary | 10.0.32.223 | /var/lib/libvirt/images
| Up |
+--------------------------------------+--------------------------------------------+-------------------+---------+-----------------------+--------------+---------------------------------------------------------------------+--------+
4 rows in set (0.00 sec)
```
KVM Host 1:
```
Before Agent Restart:
[root@pr11200-t13938-kvm-ol8-kvm1 ~]# virsh pool-list
Name State Autostart
------------------------------------------------------------
11aea44c-8240-4067-9151-84861ce00d3d active no
2aa7a529-b09e-305d-b345-c4bf3c561842 active no
618a1bab-95a5-348b-9133-c355693e01fd active no
After Agent Restart:
[root@pr11200-t13938-kvm-ol8-kvm1 ~]# virsh pool-list
Name State Autostart
------------------------------------------------------------
11aea44c-8240-4067-9151-84861ce00d3d active no
2aa7a529-b09e-305d-b345-c4bf3c561842 active no
618a1bab-95a5-348b-9133-c355693e01fd active no
```
KVM Host 2:
```
Before Agent Restart:
[root@pr11200-t13938-kvm-ol8-kvm2 ~]# virsh pool-list
Name State Autostart
------------------------------------------------------------
2aa7a529-b09e-305d-b345-c4bf3c561842 active no
37044676-7f87-4b35-b83f-8af70e8748c6 active no
618a1bab-95a5-348b-9133-c355693e01fd active no
After Agent Restart:
[root@pr11200-t13938-kvm-ol8-kvm2 ~]# virsh pool-list
Name State Autostart
------------------------------------------------------------
2aa7a529-b09e-305d-b345-c4bf3c561842 active no
37044676-7f87-4b35-b83f-8af70e8748c6 active no
618a1bab-95a5-348b-9133-c355693e01fd active no
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
