Hi, >>To avoid any locks in qemu backend and not to introduce thread safety >>into qemu block-layer I open same backend device several times, one >>device per one MQ. e.g. the following is the stack for a virtio-blk >>with num-queues=2:
Could it be possible in the future to not open several times the same backend ? I'm thinking about ceph/librbd, which since last version allow only to open once a backend by default (exclusive-lock, which is a requirement for advanced features like rbd-mirroring, fast-diff,....) Regards, Alexandre Derumier ----- Mail original ----- De: "Stefan Hajnoczi" <stefa...@gmail.com> À: "Roman Pen" <roman.peny...@profitbricks.com> Cc: "qemu-devel" <qemu-devel@nongnu.org>, "stefanha" <stefa...@redhat.com> Envoyé: Samedi 28 Mai 2016 00:27:10 Objet: Re: [Qemu-devel] [RFC] virtio-blk: simple multithreaded MQ implementation for bdrv_raw On Fri, May 27, 2016 at 01:55:04PM +0200, Roman Pen wrote: > Hello, all. > > This is RFC because mostly this patch is a quick attempt to get true > multithreaded multiqueue support for a block device with native AIO. > The goal is to squeeze everything possible on lockless IO path from > MQ block on a guest to MQ block on a host. > > To avoid any locks in qemu backend and not to introduce thread safety > into qemu block-layer I open same backend device several times, one > device per one MQ. e.g. the following is the stack for a virtio-blk > with num-queues=2: > > VirtIOBlock > / \ > VirtQueue#0 VirtQueue#1 > IOThread#0 IOThread#1 > BH#0 BH#1 > Backend#0 Backend#1 > \ / > /dev/null0 > > To group all objects related to one vq new structure is introduced: > > typedef struct VirtQueueCtx { > BlockBackend *blk; > struct VirtIOBlock *s; > VirtQueue *vq; > void *rq; > QEMUBH *bh; > QEMUBH *batch_notify_bh; > IOThread *iothread; > Notifier insert_notifier; > Notifier remove_notifier; > /* Operation blocker on BDS */ > Error *blocker; > } VirtQueueCtx; > > And VirtIOBlock includes an array of these contexts: > > typedef struct VirtIOBlock { > VirtIODevice parent_obj; > + VirtQueueCtx mq[VIRTIO_QUEUE_MAX]; > ... > > This patch is based on Stefan's series: "virtio-blk: multiqueue support", > with minor difference: I reverted "virtio-blk: multiqueue batch notify", > which does not make a lot sense when each VQ is handled by it's own > iothread. > > The qemu configuration stays the same, i.e. put num-queues=N and N > iothreads will be started on demand and N drives will be opened: > > qemu -device virtio-blk-pci,num-queues=8 > > My configuration is the following: > > host: > Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz, > 8 CPUs, > /dev/nullb0 as backend with the following parameters: > $ cat /sys/module/null_blk/parameters/submit_queues > 8 > $ cat /sys/module/null_blk/parameters/irqmode > 1 > > guest: > 8 VCPUs > > qemu: > -object iothread,id=t0 \ > -drive > if=none,id=d0,file=/dev/nullb0,format=raw,snapshot=off,cache=none,aio=native > \ > -device > virtio-blk-pci,num-queues=$N,iothread=t0,drive=d0,disable-modern=off,disable-legacy=on > > > where $N varies during the tests. > > fio: > [global] > description=Emulation of Storage Server Access Pattern > bssplit=512/20:1k/16:2k/9:4k/12:8k/19:16k/10:32k/8:64k/4 > fadvise_hint=0 > rw=randrw:2 > direct=1 > > ioengine=libaio > iodepth=64 > iodepth_batch_submit=64 > iodepth_batch_complete=64 > numjobs=8 > gtod_reduce=1 > group_reporting=1 > > time_based=1 > runtime=30 > > [job] > filename=/dev/vda > > Results: > num-queues RD bw WR bw > ---------- ----- ----- > > * with 1 iothread * > > 1 thr 1 mq 1225MB/s 1221MB/s > 1 thr 2 mq 1559MB/s 1553MB/s > 1 thr 4 mq 1729MB/s 1725MB/s > 1 thr 8 mq 1660MB/s 1655MB/s > > * with N iothreads * > > 2 thr 2 mq 1845MB/s 1842MB/s > 4 thr 4 mq 2187MB/s 2183MB/s > 8 thr 8 mq 1383MB/s 1378MB/s > > Obviously, 8 iothreads + 8 vcpu threads is too much for my machine > with 8 CPUs, but 4 iothreads show quite good result. Cool, thanks for trying this experiment and posting results. It's encouraging to see the improvement. Did you use any CPU affinity settings to co-locate vcpu and iothreads onto host CPUs? Stefan