> From: Qemu-devel <qemu-devel- > bounces+yishaih=nvidia....@nongnu.org> On Behalf Of Jason Gunthorpe > Sent: Tuesday, 18 October 2022 15:23 > To: Joao Martins <joao.m.mart...@oracle.com> > Cc: quint...@redhat.com; Alex Williamson <alex.william...@redhat.com>; > Eric Blake <ebl...@redhat.com>; Stefan Hajnoczi <stefa...@redhat.com>; > Fam Zheng <f...@euphon.net>; qemu-s3...@nongnu.org; Cornelia Huck > <coh...@redhat.com>; Thomas Huth <th...@redhat.com>; Vladimir > Sementsov-Ogievskiy <vsement...@yandex-team.ru>; Laurent Vivier > <lviv...@redhat.com>; John Snow <js...@redhat.com>; Dr. David Alan > Gilbert <dgilb...@redhat.com>; Christian Borntraeger > <borntrae...@linux.ibm.com>; Halil Pasic <pa...@linux.ibm.com>; Paolo > Bonzini <pbonz...@redhat.com>; qemu-block@nongnu.org; Eric Farman > <far...@linux.ibm.com>; Richard Henderson > <richard.hender...@linaro.org>; David Hildenbrand <da...@redhat.com>; > Avihai Horon <avih...@nvidia.com>; qemu-de...@nongnu.org > Subject: Re: [RFC 7/7] migration: call qemu_savevm_state_pending_exact() > with the guest stopped > > On Fri, Oct 14, 2022 at 01:29:51PM +0100, Joao Martins wrote: > > On 14/10/2022 12:28, Juan Quintela wrote: > > > Joao Martins <joao.m.mart...@oracle.com> wrote: > > >> On 13/10/2022 17:08, Juan Quintela wrote: > > >>> Oops. My understanding was that once the guest is stopped you can > > >>> say how big is it. > > > > > > Hi > > > > > >> It's worth keeping in mind that conceptually a VF won't stop (e.g. > > >> DMA) until really told through VFIO. So, stopping CPUs (guest) just > > >> means that guest code does not arm the VF for more I/O but still is > > >> a weak guarantee as VF still has outstanding I/O to deal with until > > >> VFIO tells it to quiesce DMA (for devices that support it). > > > > > > How can we make sure about that? > > > > > > i.e. I know I have a vfio device. I need two things: > > > - in the iterative stage, I eed to check the size, but a estimate is ok. > > > for example with RAM, we use whatever is the size of the dirty bitmap > > > at that moment. > > > If the estimated size is smaller than the theshold, we > > > * stop the guest > > > * sync dirty bitmap > > > * now we test the (real) dirty bitmap size > > > > > > How can we do something like that with a vfio device. > > > > > You would have an extra intermediate step that stops the VF prior to > > asking the device state length. What I am not sure is whether stopping > > vCPUs can be skipped as an optimization. > > It cannot, if you want to stop the VFIO device you must also stop the vCPUs > because the device is not required to respond properly to MMIO operations > when stopped. > > > > My understanding from NVidia folks was that newer firmware have an > > > ioctl to return than information. > > > > Maybe there's something new. I was thinking from this here: > > Juan is talking about the ioctl we had in the pre-copy series. > > I expect it to come into some different form to support this RFC. >
Do we really need to STOP the VM to get the exact data length that will be required to complete stop copy ? Can't we simply go with some close estimation when the device is running and drop all the complexity in QEMU/Kernel to STOP and then RE-START the VM if the threshold didn't meet, etc.? Yishai
