On 08/05/2014 02:48 AM, Mark Kirkwood wrote:
On 05/08/14 03:52, Tregaron Bayly wrote:
Does anyone have any insight on how we can tune librbd to perform closer
to the level of the rbd kernel module?
In our lab we have a four node cluster with 1GbE public network and
10GbE cluster network. A client node connects to the public network
with 10GbE.
When doing benchmarks on the client using the kernel module we get
decent performance and can cause the OSD nodes to max out their 1GbE
link at peak servicing the requests:
tx rx
max 833.66 Mbit/s | 639.44 Mbit/s
max 938.06 Mbit/s | 707.35 Mbit/s
max 846.78 Mbit/s | 702.04 Mbit/s
max 790.66 Mbit/s | 621.92 Mbit/s
However, using librbd we only get about 30% of performance and I can see
that it doesn't seem to generate requests fast enough to max out the
links on OSD nodes:
max 309.74 Mbit/s | 196.77 Mbit/s
max 300.15 Mbit/s | 154.38 Mbit/s
max 263.06 Mbit/s | 154.38 Mbit/s
max 368.91 Mbit/s | 234.38 Mbit/s
I know that I can play with cache settings to help give the client
better service on hits, but I'm wondering how I can soup up librbd so
that it can take advantage of more of the speed available in the
cluster. It seems like using librbd will leave a lot of the resources
idle.
Hi Tregaron,
I'm guessing that in the librbd case you are injecting the volume into a
VM before running your tests - might be interesting to see your libvirt
XML for the VM... in particular the 'cache' setting for the rbd volume.
If this are not set or is 'default' then changing to 'none' will
probably be significantly faster. In addition adding:
io='native'
may give a bit of a boost too!
Oh, that reminds me, also make sure to use the virtio bus instead of ide
or something else. That can make a very large performance difference.
Regards
Mark
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
