28.04.2019 13:01, Liang Li wrote: > If the backup target is a slow device like ceph rbd, the backup > process will affect guest BLK write IO performance seriously, > it's cause by the drawback of COW mechanism, if guest overwrite the > backup BLK area, the IO can only be processed after the data has > been written to backup target. > The impact can be relieved by buffering data read from backup > source and writing to backup target later, so the guest BLK write > IO can be processed in time. > Data area with no overwrite will be process like before without > buffering, in most case, we don't need a very large buffer. > > An fio test was done when the backup was going on, the test resut > show a obvious performance improvement by buffering.
Hi Liang! Good thing. Something like this I've briefly mentioned in my KVM Forum 2018 report as "RAM Cache", and I'd really prefer this functionality to be a separate filter, instead of complication of backup code. Further more, write notifiers will go away from backup code, after my backup-top series merged. v5: https://lists.gnu.org/archive/html/qemu-devel/2018-12/msg06211.html and separated preparing refactoring v7: https://lists.gnu.org/archive/html/qemu-devel/2019-04/msg04813.html RAM Cache should be a filter driver, with an in-memory buffer(s) for data written to it and with ability to flush data to underlying backing file. Also, here is another approach for the problem, which helps if guest writing activity is really high and long and buffer will be filled and performance will decrease anyway: 1. Create local temporary image, and COWs will go to it. (previously considered on list, that we should call these backup operations issued by guest writes CBW = copy-before-write, as copy-on-write is generally another thing, and using this term in backup is confusing). 2. We also set original disk as a backing for temporary image, and start another backup from temporary to real target. This scheme is almost possible now, you need to start backup(sync=none) from source to temp, to do [1]. Some patches are still needed to allow such scheme. I didn't send them, as I want my other backup patches go first anyway. But I can. On the other hand if approach with in-memory buffer works for you it may be better. Also, I'm not sure for now, should we really do this thing through two backup jobs, or we just need one separate backup-top filter and one backup job without filter, or we need an additional parameter for backup job to set cache-block-node. > > Test result(1GB buffer): > ======================== > fio setting: > [random-writers] > ioengine=libaio > iodepth=8 > rw=randwrite > bs=32k > direct=1 > size=1G > numjobs=1 > > result: > IOPS AVG latency > no backup: 19389 410 us > backup: 1402 5702 us > backup w/ buffer: 8684 918 us > ============================================== > > Cc: John Snow <js...@redhat.com> > Cc: Kevin Wolf <kw...@redhat.com> > Cc: Max Reitz <mre...@redhat.com> > Cc: Wen Congyang <wencongya...@huawei.com> > Cc: Xie Changlong <xiechanglon...@gmail.com> > Cc: Markus Armbruster <arm...@redhat.com> > Cc: Eric Blake <ebl...@redhat.com> > Cc: Fam Zheng <f...@euphon.net> > > Liang Li (2): > backup: buffer COW request and delay the write operation > qapi: add interface for setting backup cow buffer size > > block/backup.c | 118 > +++++++++++++++++++++++++++++++++++++++++----- > block/replication.c | 2 +- > blockdev.c | 5 ++ > include/block/block_int.h | 2 + > qapi/block-core.json | 5 ++ > 5 files changed, 118 insertions(+), 14 deletions(-) > -- Best regards, Vladimir
