Hi,
From the ones we managed to configure in our lab here. I noticed that using image format
"raw" instead of "qcow2" worked for us.
Regards,
Luke
-----Original Message-----
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Steven
Timm
Sent: Thursday, 18 September, 2014 5:01 AM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] ceph issue: rbd vs. qemu-kvm
I am trying to use Ceph as a data store with OpenNebula 4.6 and have followed
the instructions in OpenNebula's documentation at
http://docs.opennebula.org/4.8/administration/storage/ceph_ds.html
and compared them against the "using libvirt with ceph"
http://ceph.com/docs/master/rbd/libvirt/
We are using the ceph-recompiled qemu-kvm and qemu-img as found at
http://ceph.com/packages/qemu-kvm/
under Scientific Linux 6.5 which is a Redhat clone. Also a kernel-lt-3.10
kernel.
[root@fgtest15 qemu]# kvm -version
QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c)
2003-2008 Fabrice Bellard
From qemu-img
Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2
qed parallels nbd blkdebug host_cdrom host_floppy host_device file rbd
------------------
Libvirt is trying to execute the following KVM command:
2014-09-17 19:50:12.774+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -name one-60 -S -M rhel6.3.0 -enable-kvm -m 4096
-smp 2,sockets=2,cores=1,threads=1 -uuid
572499bf-07f3-3014-8d6a-dfa1ebb99aa4 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-60.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/var/lib/one//datastores/102/60/disk.1,if=none,id=drive-virtio-disk1,format=raw,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1
-drive
file=/var/lib/one//datastores/102/60/disk.2,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=54:52:00:02:0b:04,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:60 -k en-us -vga
cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
char device redirected to /dev/pts/3
qemu-kvm: -drive
file=rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none:
could not open disk image
rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789:
Invalid argument
2014-09-17 19:50:12.980+0000: shutting down
-----------
just to show that from the command line I can see the rbd pool fine
[root@fgtest15 qemu]# rbd list one
foo
one-19
one-19-58-0
one-19-60-0
[root@fgtest15 qemu]# rbd info one/one-19-60-0
rbd image 'one-19-60-0':
size 40960 MB in 10240 objects
order 22 (4096 kB objects)
block_name_prefix: rb.0.3c39.238e1f29
format: 1
and even mount stuff with rbd map, etc.
It's only inside libvirt that we had the problem.
At first we were getting "permission denied" but then I upped the
permissions allowed to the libvirt user (client.libvirt2) and then
we are just getting "invalid argument"
client.libvirt2
key: AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==
caps: [mon] allow r
caps: [osd] allow *, allow rwx pool=one
------
Any idea why kvm doesn't like the argument I am delivering in the file=
argument? Better--does anyone have a working kvm command out
of either opennebula or openstack against which I can compare?
Thanks
Steve Timm
------------------------------------------------------------------
Steven C. Timm, Ph.D (630) 840-8525
t...@fnal.gov http://home.fnal.gov/~timm/
Fermilab Scientific Computing Division, Scientific Computing Services Quad.
Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
________________________________
DISCLAIMER:
This e-mail (including any attachments) is for the addressee(s) only and may be
confidential, especially as regards personal data. If you are not the intended
recipient, please note that any dealing, review, distribution, printing,
copying or use of this e-mail is strictly prohibited. If you have received this
email in error, please notify the sender immediately and delete the original
message (including any attachments).
MIMOS Berhad is a research and development institution under the purview of the
Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions
and other information in this e-mail that do not relate to the official
business of MIMOS Berhad and/or its subsidiaries shall be understood as neither
given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS
Berhad nor its subsidiaries accepts responsibility for the same. All liability
arising from or in connection with computer viruses and/or corrupted e-mails is
excluded to the fullest extent permitted by law.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
