>>I think qmrestore isn't issuing any flush request (until maybe the end), Need to be checked! (but if I think we open restore block storage with writeback, so I hope we send flush)
>>so for ceph storage backend we should set >>rbd_cache_writethrough_until_flush=false for better performance. I think it's possible to pass theses flag in qemu block driver option, when opening the rbd storage http://docs.ceph.com/docs/hammer/rbd/qemu-rbd/ qemu-img {command} [options] rbd:{pool-name}/{image-name}[@snapshot-name][:option1=value1][:option2=value2...] for qemu-img or with qemu drive option, I think it's possible to send as option, ":rbd_cache_writethrough_until_flush=false" But if missing flush if really the problem, it should be added to restore command directly. (maybe 1 flush each 4MB for example) ----- Mail original ----- De: "Eneko Lacunza" <elacu...@binovo.es> À: "dietmar" <diet...@proxmox.com>, "pve-devel" <pve-devel@pve.proxmox.com> Envoyé: Jeudi 21 Juillet 2016 13:19:10 Objet: Re: [pve-devel] Speed up PVE Backup Hi, El 21/07/16 a las 09:34, Dietmar Maurer escribió: > >>> But you can try to assemble larger blocks, and write them once you get >>> an out of order block... >> Yes, this is the plan. >>> I always thought the ceph libraries does (or should do) that anyways? >>> (write combining) >> Reading the docs: >> http://docs.ceph.com/docs/hammer/rbd/rbd-config-ref/ >> >> It should be true when write-back rbd cache is activated. This seems to >> be the default, but maybe we're using disk cache setting on restore too? >> >> I'll try to change the disk cache setting and will report the results. > thanks! > Looking at more docs: http://docs.ceph.com/docs/hammer/rbd/qemu-rbd/ This says: " QEMU’s cache settings override Ceph’s default settings (i.e., settings that are not explicitly set in the Ceph configuration file). If you explicitly set RBD Cache <http://docs.ceph.com/docs/hammer/rbd/rbd-config-ref/> settings in your Ceph configuration file, your Ceph settings override the QEMU cache settings. If you set cache settings on the QEMU command line, the QEMU command line settings override the Ceph configuration file settings. " I have been doing tests all morning with a different backup (only one 10GB disk) so that I could perform tests faster. I thought maybe we were restoring without writeback cache (rbd cache), but have tried the following ceph.conf tweaks and conclude that rbd cache is enabled: 1. If I set rbd cache = true I get the same performance. 2. If I set rbd cache writethrough until flush = true (rbd cache = true not necessary), I get x2-x3 the restore performance. This setting is a security measure for non-flushing virtio drivers, but it is safe for a restore. No writeback until a flush is detected I think qmrestore isn't issuing any flush request (until maybe the end), so for ceph storage backend we should set rbd_cache_writethrough_until_flush=false for better performance. Restore is happening at about 30-45MB/s vs 15MB/s before, but all this may be affected by a slow OSD, so I don't think my absolute figures are good, only the fact that there is a noticeable improvement. (we'll have this fixed next week). If someone can test and confirm this, it should be quite easy to patch qmrestore... Thanks Eneko -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943493611 943324914 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa) www.binovo.es _______________________________________________ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel _______________________________________________ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
