I am trying to use Ceph as a data store with OpenNebula 4.6 and
have followed the instructions in OpenNebula's documentation
at
http://docs.opennebula.org/4.8/administration/storage/ceph_ds.html
and compared them against the "using libvirt with ceph"
http://ceph.com/docs/master/rbd/libvirt/
We are using the ceph-recompiled qemu-kvm and qemu-img as found at
http://ceph.com/packages/qemu-kvm/
under Scientific Linux 6.5 which is a Redhat clone. Also a kernel-lt-3.10
kernel.
[root@fgtest15 qemu]# kvm -version
QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c)
2003-2008 Fabrice Bellard
From qemu-img
Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2
qed parallels nbd blkdebug host_cdrom host_floppy host_device file rbd
------------------
Libvirt is trying to execute the following KVM command:
2014-09-17 19:50:12.774+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -name one-60 -S -M rhel6.3.0 -enable-kvm -m 4096
-smp 2,sockets=2,cores=1,threads=1 -uuid
572499bf-07f3-3014-8d6a-dfa1ebb99aa4 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-60.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/var/lib/one//datastores/102/60/disk.1,if=none,id=drive-virtio-disk1,format=raw,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1
-drive
file=/var/lib/one//datastores/102/60/disk.2,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=54:52:00:02:0b:04,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:60 -k en-us -vga
cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
char device redirected to /dev/pts/3
qemu-kvm: -drive
file=rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none:
could not open disk image
rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789:
Invalid argument
2014-09-17 19:50:12.980+0000: shutting down
-----------
just to show that from the command line I can see the rbd pool fine
[root@fgtest15 qemu]# rbd list one
foo
one-19
one-19-58-0
one-19-60-0
[root@fgtest15 qemu]# rbd info one/one-19-60-0
rbd image 'one-19-60-0':
size 40960 MB in 10240 objects
order 22 (4096 kB objects)
block_name_prefix: rb.0.3c39.238e1f29
format: 1
and even mount stuff with rbd map, etc.
It's only inside libvirt that we had the problem.
At first we were getting "permission denied" but then I upped the
permissions allowed to the libvirt user (client.libvirt2) and then
we are just getting "invalid argument"
client.libvirt2
key: AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==
caps: [mon] allow r
caps: [osd] allow *, allow rwx pool=one
------
Any idea why kvm doesn't like the argument I am delivering in the file=
argument? Better--does anyone have a working kvm command out
of either opennebula or openstack against which I can compare?
Thanks
Steve Timm
------------------------------------------------------------------
Steven C. Timm, Ph.D (630) 840-8525
t...@fnal.gov http://home.fnal.gov/~timm/
Fermilab Scientific Computing Division, Scientific Computing Services Quad.
Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
