Hi Simon,
Thanks for the reply.
>> Also run ceph health and make sure your agent can talk to your ceph
monitors.
ceph health status shows fine and also we can connect our OSD's from
monitor server. Snippets pasted below.
===
[root@mon1 ~]# ceph -s
cluster ebac75fc-e631-4c9f-a310-880cbcdd1d25
health HEALTH_OK
monmap e1: 1 mons at {mon1=10.10.48.7:6789/0}
election epoch 3, quorum 0 mon1
osdmap e12: 2 osds: 2 up, 2 in
flags sortbitwise,require_jewel_osds
pgmap v3376: 192 pgs, 2 pools, 0 bytes data, 0 objects
73108 kB used, 1852 GB / 1852 GB avail
192 active+clean
==
[root@mon1 ~]# ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 1.80878 root default
-2 0.90439 host osd4
0 0.90439 osd.0 up 1.00000 1.00000
-3 0.90439 host osdceph3
1 0.90439 osd.1 up 1.00000 1.00000
====
>> Which OS are you running on your hosts?
Our cloudstack servers are on Centos 6 and ceph admin/mon/osd servers
are running on Centos 7.
After we have enabled the cloudstack agent log to DEBUG on hypervisor
server, we are seeing below errors while adding the ceph primary.
==
2017-02-22 21:01:00,444 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) <pool type='rbd'>
==
In a quick search, we found that the hypervisor KVM has no kernel module
loaded for RBD. Then we have upgraded the kernel to elrepo and loaded
the module with modprobe. When we tried to rebuild rbd module with
existing libvirtd configuration, that didn't worked. Apart from this, we
have custom compiled libvirtd with rbd support and we have no clues
regarding how to connect custom libvirtd with the qemu image utility.
=====
[root@hyperkvm3 ~]# libvirtd --version (default)
libvirtd (libvirt) 0.10.2
=
[root@hyperkvm3 ~]# lsmod | grep rbd
rbd 56743 0
libceph 148605 1 rbd
=
[root@hyperkvm3 ~]# qemu-img -h | grep "Supported formats"
Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2
qed vhdx parallels nbd blkdebug null host_cdrom host_floppy host_device
file gluster
== // no rbd support
[root@hyperkvm3 ~]# /usr/bin/sbin/libvirtd --version (custom)
/usr/bin/sbin/libvirtd (libvirt) 1.3.5
=====
Do you have any ideas/suggestions?
-Kurt
On 02/22/2017 08:20 PM, Simon Weller wrote:
I agree, agent logs would be good to look at.
You can enable kvm agent debugging by running this: sed -i 's/INFO/DEBUG/g'
/etc/cloudstack/agent/log4j-cloud.xml
Restart the agent and then tail -f /var/log/cloudstack/agent/agent.log
Also run ceph health and make sure your agent can talk to your ceph monitors.
Which OS are you running on your hosts?
- Si
________________________________
From: Abhinandan Prateek <[email protected]>
Sent: Wednesday, February 22, 2017 12:45 AM
To: [email protected]
Subject: Re: Getting errors while adding ceph storage to cloudstack
Take a look at the agent logs on kvm host there will be more clues.
On 22/02/17, 8:10 AM, "Kurt K" <[email protected]> wrote:
Hello,
I have created a ceph cluster with one admin server, one monitor and two
osd's. The setup is completed. But when trying to add the ceph as
primary storage of cloudstack, I am getting the below error in error logs.
Am I missing something ? Please help.
================
2017-02-20 21:03:02,842 DEBUG
[o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl]
(catalina-exec-6:ctx-f293a10c ctx-093b4faf) In createPool Adding the
pool to each of the hosts
2017-02-20 21:03:02,843 DEBUG [c.c.s.StorageManagerImpl]
(catalina-exec-6:ctx-f293a10c ctx-093b4faf) Adding pool null to host 1
2017-02-20 21:03:02,845 DEBUG [c.c.a.t.Request]
(catalina-exec-6:ctx-f293a10c ctx-093b4faf) Seq 1-653584895922143294:
Sending { Cmd , MgmtId: 207381009036, via: 1(hyperkvm.xxxxx.com), Ver:
v1, Flags: 100011,
[{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":14,"uuid":"9c51d737-3a6f-3bb3-8f28-109954fc2ef0","host":"mon1.xxxx.com","path":"cloudstack","userInfo":"cloudstack:AQDagqZYgSSpOBAATFvSt4tz3cOUWhNtR-NaoQ==","port":6789,"type":"RBD"},"localPath":"/mnt//ac5436a6-5889-30eb-b079-ac1e05a30526","wait":0}}]
}
2017-02-20 21:03:02,944 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-15:null) Seq 1-653584895922143294: Processing: {
Ans: , MgmtId: 207381009036, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
Failed to create storage pool:
9c51d737-3a6f-3bb3-8f28-109954fc2ef0\n\tat
com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:524)\n\tat
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:277)\n\tat
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:271)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2823)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1325)\n\tat
com.cloud.agent.Agent.processRequest(Agent.java:501)\n\tat
com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)\n\tat
com.cloud.utils.nio.Task.run(Task.java:84)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat
java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
2017-02-20 21:03:02,944 DEBUG [c.c.a.t.Request]
(catalina-exec-6:ctx-f293a10c ctx-093b4faf) Seq 1-653584895922143294:
Received: { Ans: , MgmtId: 207381009036, via: 1, Ver: v1, Flags: 10, {
Answer } }
2017-02-20 21:03:02,944 DEBUG [c.c.a.m.AgentManagerImpl]
(catalina-exec-6:ctx-f293a10c ctx-093b4faf) Details from executing class
com.cloud.agent.api.ModifyStoragePoolCommand:
com.cloud.utils.exception.CloudRuntimeException: Failed to create
storage pool: 9c51d737-3a6f-3bb3-8f28-109954fc2ef0
at
com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:524)
at
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:277)
at
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:271)
================
Any suggestions?
-Kurt
[email protected]
www.shapeblue.com<http://www.shapeblue.com>
ShapeBlue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Introduction Upgrading CloudStack can sometimes be a little daunting - but as
the 5P's proverb goes - Proper Planning Prevents Poor Performance.
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
--
*~Kurt *
