Based on the logs, the error is due to some kind of rbd pool configuration. Can you try to add rbd pool on the kvm directly, see if it works? You can also try to check if from the KVM host you can reach your ceph nodes/mons?
Regards. Regards. ________________________________ From: Wei ZHOU <[email protected]> Sent: Wednesday, August 7, 2024 5:45:34 PM To: Muhammad Hanis Irfan Mohd Zaid <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: Unable to add Ceph RBD for primary storage (No such file or directory) Hi, I just tested adding ceph pool on alma9 (it has the same package/version installed), it worked What's the ceph version ? -Wei On Wed, Aug 7, 2024 at 12:11 PM Muhammad Hanis Irfan Mohd Zaid <[email protected]> wrote: > > I also noticed that comma in the end. I just took a blind eye and expected it > to be as designed. I don't enter any comma in the end: https://ibb.co/N3zMVvc > > Yep, the package is already installed. > > # dnf -y install libvirt-daemon-driver-storage-rbd > Last metadata expiration check: 1:29:57 ago on Wed 07 Aug 2024 04:37:02 PM > +08. > Package libvirt-daemon-driver-storage-rbd-10.0.0-6.6.el9_4.x86_64 is already > installed. > Dependencies resolved. > Nothing to do. > Complete! > > I'm running the KVM host in Rocky Linux 9.4 (Blue Onyx) with CloudStack > 4.19.1.1 > > > > On Wed, 7 Aug 2024 at 18:02, Wei ZHOU <[email protected]> wrote: >> >> Hi, >> >> There is a comma (,) after 10.0.32.75 , was it a mistake ? >> >> org.libvirt.LibvirtException: failed to connect to the RADOS monitor >> on: 10.0.32.71,10.0.32.72,10.0.32.73,10.0.32.74,10.0.32.75,: No such file >> or directory >> >> Have you installed the package "libvirt-daemon-driver-storage-rbd" on >> the kvm host ? >> >> -Wei >> >> On Wed, Aug 7, 2024 at 11:27 AM Muhammad Hanis Irfan Mohd Zaid >> <[email protected]> wrote: >> > >> > I'm trying to add a Ceph RBD pool for primary storage use. I've 5 Ceph MONs >> > in my POC lab. Ping and telnet to all the Ceph MONs with port 6789 works. >> > >> > I'm following the steps from this: >> > - https://docs.ceph.com/en/reef/rbd/rbd-cloudstack/ >> > - https://rohityadav.cloud/blog/ceph/ >> > >> > Agent log when specifying 5 monitors: >> > 2024-08-07 17:12:34,691 INFO [kvm.storage.LibvirtStorageAdaptor] >> > (agentRequest-Handler-3:null) (logid:db5277f2) Attempting to create storage >> > pool eb5ec036-c08a-3d3d-996d-40968077d391 (RBD) in libvirt >> > 2024-08-07 17:12:34,706 WARN [kvm.storage.LibvirtStorageAdaptor] >> > (agentRequest-Handler-3:null) (logid:db5277f2) Storage pool >> > eb5ec036-c08a-3d3d-996d-40968077d391 was not found running in libvirt. Need >> > to create it. >> > 2024-08-07 17:12:34,706 INFO [kvm.storage.LibvirtStorageAdaptor] >> > (agentRequest-Handler-3:null) (logid:db5277f2) Didn't find an existing >> > storage pool eb5ec036-c08a-3d3d-996d-40968077d391 by UUID, checking for >> > pools with duplicate paths >> > 2024-08-07 17:17:34,738 ERROR [kvm.storage.LibvirtStorageAdaptor] >> > (agentRequest-Handler-3:null) (logid:db5277f2) Failed to create RBD storage >> > pool: org.libvirt.LibvirtException: failed to connect to the RADOS monitor >> > on: 10.0.32.71,10.0.32.72,10.0.32.73,10.0.32.74,10.0.32.75,: No such file >> > or directory >> > 2024-08-07 17:17:34,739 ERROR [kvm.storage.LibvirtStorageAdaptor] >> > (agentRequest-Handler-3:null) (logid:db5277f2) Failed to create the RBD >> > storage pool, cleaning up the libvirt secret >> > 2024-08-07 17:17:34,739 WARN [cloud.agent.Agent] >> > (agentRequest-Handler-3:null) (logid:db5277f2) Caught: >> > com.cloud.utils.exception.CloudRuntimeException: Failed to create storage >> > pool: eb5ec036-c08a-3d3d-996d-40968077d391 >> > at >> > com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:743) >> > at >> > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:364) >> > at >> > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:358) >> > at >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) >> > at >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) >> > at >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) >> > at >> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929) >> > at com.cloud.agent.Agent.processRequest(Agent.java:683) >> > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106) >> > at com.cloud.utils.nio.Task.call(Task.java:83) >> > at com.cloud.utils.nio.Task.call(Task.java:29) >> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >> > at >> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >> > at >> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >> > at java.base/java.lang.Thread.run(Thread.java:829) >> > >> > Agent log when specifying 1 monitors: >> > 2024-08-07 17:06:09,791 INFO [kvm.storage.LibvirtStorageAdaptor] >> > (agentRequest-Handler-2:null) (logid:c790784b) Attempting to create storage >> > pool 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2 (RBD) in libvirt >> > 2024-08-07 17:06:09,806 WARN [kvm.storage.LibvirtStorageAdaptor] >> > (agentRequest-Handler-2:null) (logid:c790784b) Storage pool >> > 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2 was not found running in libvirt. Need >> > to create it. >> > 2024-08-07 17:06:09,806 INFO [kvm.storage.LibvirtStorageAdaptor] >> > (agentRequest-Handler-2:null) (logid:c790784b) Didn't find an existing >> > storage pool 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2 by UUID, checking for >> > pools with duplicate paths >> > 2024-08-07 17:11:09,840 ERROR [kvm.storage.LibvirtStorageAdaptor] >> > (agentRequest-Handler-2:null) (logid:c790784b) Failed to create RBD storage >> > pool: org.libvirt.LibvirtException: failed to connect to the RADOS monitor >> > on: 10.0.32.71,: No such file or directory >> > 2024-08-07 17:11:09,840 ERROR [kvm.storage.LibvirtStorageAdaptor] >> > (agentRequest-Handler-2:null) (logid:c790784b) Failed to create the RBD >> > storage pool, cleaning up the libvirt secret >> > 2024-08-07 17:11:09,841 WARN [cloud.agent.Agent] >> > (agentRequest-Handler-2:null) (logid:c790784b) Caught: >> > com.cloud.utils.exception.CloudRuntimeException: Failed to create storage >> > pool: 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2 >> > at >> > com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:743) >> > at >> > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:364) >> > at >> > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:358) >> > at >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) >> > at >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) >> > at >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) >> > at >> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929) >> > at com.cloud.agent.Agent.processRequest(Agent.java:683) >> > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106) >> > at com.cloud.utils.nio.Task.call(Task.java:83) >> > at com.cloud.utils.nio.Task.call(Task.java:29) >> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >> > at >> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >> > at >> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >> > at java.base/java.lang.Thread.run(Thread.java:829) >> > >> > Ran these commands on my Ceph node but still no luck. I refer this: >> > - https://github.com/apache/cloudstack/issues/5741 >> > >> > ceph config set mon auth_expose_insecure_global_id_reclaim false >> > ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false >> > ceph config set mon auth_allow_insecure_global_id_reclaim false >> > ceph orch restart mon >> > >> > Thanks for the help :)
