that helps. thx CC: pi...@pioto.org; ceph-users@lists.ceph.com From: j.michael.l...@gmail.com Subject: Re: [ceph-users] RBD vs RADOS benchmark performance Date: Sat, 11 May 2013 13:16:18 -0400 To: ws...@hotmail.com
Hmm try searching the libvirt git for josh as an author you should see the commit from Josh Durgan about white listing rbd migration. On May 11, 2013, at 10:53 AM, w sun <ws...@hotmail.com> wrote: The reference Mike provided is not valid to me. Anyone else has the same problem? --weiguo From: j.michael.l...@gmail.com Date: Sat, 11 May 2013 08:45:41 -0400 To: pi...@pioto.org CC: ceph-users@lists.ceph.com Subject: Re: [ceph-users] RBD vs RADOS benchmark performance I believe that this is fixed in the most recent versions of libvirt, sheepdog and rbd were marked erroneously as unsafe. http://libvirt.org/git/?p=libvirt.git;a=commit;h=78290b1641e95304c862062ee0aca95395c5926c Sent from my iPad On May 11, 2013, at 8:36 AM, Mike Kelly <pi...@pioto.org> wrote: (Sorry for sending this twice... Forgot to reply to the list) Is rbd caching safe to enable when you may need to do a live migration of the guest later on? It was my understanding that it wasn't, and that libvirt prevented you from doing the migration of it knew about the caching setting. If it isn't, is there anything else that could help performance? Like, some tuning of block size parameters for the rbd image or the qemu On May 10, 2013 8:57 PM, "Mark Nelson" <mark.nel...@inktank.com> wrote: On 05/10/2013 07:21 PM, Yun Mao wrote: Hi Mark, Given the same hardware, optimal configuration (I have no idea what that means exactly but feel free to specify), which is supposed to perform better, kernel rbd or qemu/kvm? Thanks, Yun Hi Yun, I'm in the process of actually running some tests right now. In previous testing, it looked like kernel rbd and qemu/kvm performed about the same with cache off. With cache on (in cuttlefish), small sequential write performance improved pretty dramatically vs without cache. Large write performance seemed to take more concurrency to reach peak performance, but ultimately aggregate throughput was about the same. Hopefully I should have some new results published in the near future. Mark On Fri, May 10, 2013 at 6:56 PM, Mark Nelson <mark.nel...@inktank.com <mailto:mark.nel...@inktank.com>> wrote: On 05/10/2013 12:16 PM, Greg wrote: Hello folks, I'm in the process of testing CEPH and RBD, I have set up a small cluster of hosts running each a MON and an OSD with both journal and data on the same SSD (ok this is stupid but this is simple to verify the disks are not the bottleneck for 1 client). All nodes are connected on a 1Gb network (no dedicated network for OSDs, shame on me :). Summary : the RBD performance is poor compared to benchmark A 5 seconds seq read benchmark shows something like this : sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 0 0 0 0 0 - 0 1 16 39 23 91.9586 92 0.966117 0.431249 2 16 64 48 95.9602 100 0.513435 0.53849 3 16 90 74 98.6317 104 0.25631 0.55494 4 11 95 84 83.9735 40 1.80038 0.58712 Total time run: 4.165747 Total reads made: 95 Read size: 4194304 Bandwidth (MB/sec): 91.220 Average Latency: 0.678901 Max latency: 1.80038 Min latency: 0.104719 91MB read performance, quite good ! Now the RBD performance : root@client:~# dd if=/dev/rbd1 of=/dev/null bs=4M count=100 100+0 records in 100+0 records out 419430400 bytes (419 MB) copied, 13.0568 s, 32.1 MB/s There is a 3x performance factor (same for write: ~60M benchmark, ~20M dd on block device) The network is ok, the CPU is also ok on all OSDs. CEPH is Bobtail 0.56.4, linux is 3.8.1 arm (vanilla release + some patches for the SoC being used) Can you show me the starting point for digging into this ? Hi Greg, First things first, are you doing kernel rbd or qemu/kvm? If you are doing qemu/kvm, make sure you are using virtio disks. This can have a pretty big performance impact. Next, are you using RBD cache? With 0.56.4 there are some performance issues with large sequential writes if cache is on, but it does provide benefit for small sequential writes. In general RBD cache behaviour has improved with Cuttlefish. Beyond that, are the pools being targeted by RBD and rados bench setup the same way? Same number of Pgs? Same replication? Thanks! _________________________________________________ ceph-users mailing list ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> _________________________________________________ ceph-users mailing list ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
