On Wed, Jul 18, 2018 at 10:55 AM Nikola Ciprich <nikola.cipr...@linuxbox.cz> wrote:
> Hi, > > historically I've found many discussions about this topic in > last few years, but it seems to me to be still a bit unresolved > so I'd like to open the question again.. > > In all flash deployments, under 12.2.5 luminous and qemu 12.2.0 > using lbirbd, I'm getting much worse results regarding IOPS then > with KRBD and direct block device access.. > > I'm testing on the same 100GB RBD volume, notable ceph settings: > > client rbd cache disabled > osd_enable_op_tracker = False > osd_op_num_shards = 64 > osd_op_num_threads_per_shard = 1 > > osds are running bluestore, 2 replicas (it's just for testing) > > when I run FIO using librbd directly, I'm getting ~160k reads/s > and ~60k writes/s which is not that bad. > > however when I run fio on block device under VM (qemu using librbd), > I'm getting only 60/40K op/s which is a huge loss.. > > when I use VM with block access to krbd mapped device, numbers > are much better, I'm getting something like 115/40K op/s which > is not ideal, but still much better.. tried many optimisations > and configuration variants (multiple queues, threads vs native aio > etc), but krbd still performs much much better.. > > My question is whether this is expected, or should both access methods > give more similar results? If possible, I'd like to stick to librbd > (especially because krbd still lacks layering support, but there are > more reasons) > Just to clarify: modern / rebased krbd block drivers definitely support layering. The only missing features right now are object-map/fast-diff, deep-flatten, and journaling (for RBD mirroring). > interesting is, that when I compare fio direct ceph access, librbd performs > better then KRBD, but this doesn't concern me that much.. > > another question, during the tests, I noticed that enabling exclusive lock > feature degrades write iops a lot as well, is this expected? (the > performance > falls to someting like 50%) > If you are running multiple fio jobs against the same image (or have the krbd device mapped to multiple hosts w/ active IO), then I would expect a huge performance hit since the lock needs to be transitioned between clients. > > I'm doing the tests on small 2 node cluster, VMS are running directly on > ceph nodes, > all is centos 7 with 4.14 kernel. (I know it's not recommended to run VMs > directly > on ceph nodes, but for small deployments it's necessary for us) > > if I could provide more details, I'll be happy to do so > > BR > > nik > > > -- > ------------------------------------- > Ing. Nikola CIPRICH > LinuxBox.cz, s.r.o. > 28.rijna 168, 709 00 Ostrava > > tel.: +420 591 166 214 > fax: +420 596 621 273 > mobil: +420 777 093 799 > www.linuxbox.cz > > mobil servis: +420 737 238 656 > email servis: ser...@linuxbox.cz > ------------------------------------- > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Jason
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
