Hi Wido Thanks for you comment.
If we create vm on the NFS primary storage and mount additional disk on the RBD storage, it works fine. If we check vm from virt manager, there is no virtual disk. So we believe the problem should be vm configuration... We will check ceph version tomorrow. Thanks Kimi -----Original Message----- From: Wido den Hollander [mailto:w...@widodh.nl] Sent: Monday, July 22, 2013 11:43 PM To: dev@cloudstack.apache.org Subject: Re: Problem in adding Ceph RBD storage to CloudStack Hi, On 07/22/2013 02:25 PM, Kimihiko Kitase wrote: > Wido, Thank you very much. > > CloudStack: 4.1.0 > QEMU: 1.5.50 > Libvirt: 0.10.2 What version of Ceph on the nodes? $ ceph -v > > We will set "DEBUG" on the agent tomorrow. But the following is command > CloudStack issue. We got this command at KVM host. > > [root@rx200s7-07m ~]# ps -ef|grep 1517 > root 16099 1 27 19:36 ? 00:00:12 /usr/libexec/qemu-kvm -name > i-2-1517-VM -S -M pc-i440fx-1.6 -enable-kvm -m 256 -smp > 1,sockets=1,cores=1,threads=1 -uuid e67f1707-fe92-3426-978d-0441d5000d6a > -no-user-config -nodefaults -chardev > socket,id=charmonitor,path=/var/lib/libvirt/qemu/i-2-1517-VM.monitor,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown > -boot dc -drive > file=rbd:libvirt-pool/cd3688ab-e37b-4866-9ea7-4051b670a323:id=libvirt:key=AQC7OuZReMndFxAAY/qUwLbvfod6EMvgVWU21g==:auth_supported=cephx\;none:mon_host=192.168.10.20\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=none > -device > virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 > -drive > if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,cache=none > -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev > tap,fd=27,id=hostnet0,vhost=on,vhostfd=29 -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=02:00:0a:b9:00:16 ,bus=pci. 0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 0.0.0.0:3 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 > The argument to Qemu seems just fine, so I think the problem is not in CloudStack. Wido > Thanks > Kimi > > -----Original Message----- > From: Wido den Hollander [mailto:w...@widodh.nl] > Sent: Monday, July 22, 2013 7:47 PM > To: dev@cloudstack.apache.org > Subject: Re: Problem in adding Ceph RBD storage to CloudStack > > Hi, > > On 07/22/2013 12:43 PM, Kimihiko Kitase wrote: >> It seems secondary storage vm could copy template to primary storage >> successfully, but created VM doesn't point this vol.. >> If we create vm manually and add this vol as boot vol, it works fine.. >> > > Which version of CloudStack are you using? > > What is the Qemu version running on your hypervisor and what libvirt version? > > If you set the logging level on the Agent to "DEBUG", does it show deploying > the VM with the correct XML parameters? > > I haven't seen the things you are reporting. > > Wido > >> So it seems cloudstack cannot configure VM correctly in ceph rbd environment. >> >> Any idea? >> >> Thanks >> Kimi >> >> -----Original Message----- >> From: Kimihiko Kitase [mailto:kimihiko.kit...@citrix.co.jp] >> Sent: Monday, July 22, 2013 7:11 PM >> To: dev@cloudstack.apache.org >> Subject: RE: Problem in adding Ceph RBD storage to CloudStack >> >> Hello >> >> I am in the project with Nakajima san. >> >> We succeeded to add RBD storage to primary storage. >> But when we try to boot centos as user instance, it fail during system >> logger process. >> It works fine when we boot centos using NFS storage. >> It works fine when we boot centos using NFS storage and add additional disk >> from RBD storage. >> >> Do you have any idea to resolve this issue? >> >> Thanks >> Kimi >> >> -----Original Message----- >> From: Takuma Nakajima [mailto:penguin.trance.2...@gmail.com] >> Sent: Saturday, July 20, 2013 12:23 PM >> To: dev@cloudstack.apache.org >> Subject: Re: Problem in adding Ceph RBD storage to CloudStack >> >> I'm sorry but I forgot to tell you that the environment does not have the >> internet connection. >> It is not allowed to make a direct connection to the internet because of the >> security policy. >> >> Wido, >>> No, it works for me like a charm :) >>> >>> Could you set the Agent logging to DEBUG as well and show the output >>> of >> that log? Maybe paste the log on pastebin. >>> >>> I'm interested in the XMLs the Agent is feeding to libvirt when >>> adding >> the RBD pool. >> >> I thought the new libvirt overwrites the old one, but actually both libvirt >> (with RBD and without RBD) were installed to the system. qemu was installed >> from the package and so it might have the dependency to the libvirt >> installed from the package. After deleting the both libvirt installed from >> source and package, then installed it from rpm package with RBD support, RBD >> storage was registered to the CloudStack successfully. >> >> David, >>> Why not 6.4? >> >> Because of no internet connection, packages in the local mirror repository >> may be old. >> I checked /etc/redhat-release and it showed the version is 6.3. >> >> In current state, although the RBD storage was installed, system VMs >> won't start with "Unable to get vms >> org.libvirt.LibvirtException: Domain not found: no domain with >> matching uuid 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'" error like >> http://mail-archives.apache.org/mod_mbox/cloudstack-users/201303.mbox >> / >> %3CD2EE6B3265AD864EB3EA4F5C670D256F3546C1@EXMBX01L-CRP-03.webmdhealth. >> net%3E The uuid in the error message was not in the database of the >> management server nor in ceph storage node. >> >> I tried removing host from the CloudStack and cleaning up the computing >> node, but it cannot be added again to the CloudStack. >> agent log says it attempted to connect to localhost:8250 though the >> management server address is set to 10.40.1.190 in global settings. >> management server log is here: http://pastebin.com/muGz73c0 >> (10.40.1.24 is the address of the computing node) >> >> Now the computing node is under rebuilding. >> >> Takuma Nakajima >> >> 2013/7/19 David Nalley <da...@gnsa.us>: >>> On Thu, Jul 18, 2013 at 12:09 PM, Takuma Nakajima >>> <penguin.trance.2...@gmail.com> wrote: >>>> Hi, >>>> >>>> I'm building a CloudStack 4.1 with Ceph RBD storage using RHEL 6.3 >> recently >>>> but it fails when adding RBD storage to primary storage. >>>> Does anybody know about the problem? >>> >>> >>> Why not 6.4?
