On Fri, Sep 15, 2023 at 10:57:33AM +0800, Zhu, Lingshan wrote:
>
>
> On 9/14/2023 7:34 PM, Michael S. Tsirkin wrote:
> > On Wed, Sep 06, 2023 at 04:16:34PM +0800, Zhu Lingshan wrote:
> > > This patch introduces a new status bit in the device status: SUSPEND.
> > >
> > > This SUSPEND bit can be used by the driver to suspend a device,
> > > in order to stabilize the device states and virtqueue states.
> > >
> > > Its main use case is live migration.
> > >
> > > Signed-off-by: Zhu Lingshan <[email protected]>
> > > Signed-off-by: Jason Wang <[email protected]>
> > > Signed-off-by: Eugenio Pérez <[email protected]>
> > > ---
> > > content.tex | 31 +++++++++++++++++++++++++++++++
> > > 1 file changed, 31 insertions(+)
> > >
> > > diff --git a/content.tex b/content.tex
> > > index 0e492cd..0fab537 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic
> > > Facilities of a Virtio Dev
> > > \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
> > > drive the device.
> > > +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that
> > > the
> > > + device has been suspended by the driver.
> > > +
> > > \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced
> > > an error from which it can't recover.
> > > \end{description}
> > > @@ -73,6 +76,10 @@ \section{\field{Device Status} Field}\label{sec:Basic
> > > Facilities of a Virtio Dev
> > > recover by issuing a reset.
> > > \end{note}
> > > +The driver SHOULD NOT set SUSPEND if FEATURES_OK is not set.
> > > +
> > > +When setting SUSPEND, the driver MUST re-read \field{device status} to
> > > ensure the SUSPEND bit is set.
> > > +
> > > \devicenormative{\subsection}{Device Status Field}{Basic Facilities of
> > > a Virtio Device / Device Status Field}
> > > The device MUST NOT consume buffers or send any used buffer
> > > @@ -82,6 +89,26 @@ \section{\field{Device Status} Field}\label{sec:Basic
> > > Facilities of a Virtio Dev
> > > that a reset is needed. If DRIVER_OK is set, after it sets
> > > DEVICE_NEEDS_RESET, the device
> > > MUST send a device configuration change notification to the driver.
> > > +The device MUST ignore SUSPEND if FEATURES_OK is not set.
> > > +
> > > +The device MUST ignore SUSPEND if VIRTIO_F_SUSPEND is not negotiated.
> > why? let's just forbid driver from setting it.
> OK
> >
> > > +
> > > +The device SHOULD allow settings to \field{device status} even when
> > > SUSPEND is set.
> > > +
> > > +If VIRTIO_F_SUSPEND is negotiated and SUSPEND is set, the device SHOULD
> > > clear SUSPEND
> > > +and resumes operation upon DRIVER_OK.
> > > +
> > sorry what?
> In case of a failed or cancelled Live Migration, the device needs to resume
> operation.
> However the spec forbids the driver to clear a device status bit, so
> re-writing
> DRIVER_OK is expected to clear SUSPEND and the device resume operation.
No, DRIVER_OK is already set. Setting a bit that is already set should
not have side effects. In fact auto-clearing suspend is problematic too.
> >
> > > +If VIRTIO_F_SUSPEND is negotiated, when the driver sets SUSPEND,
> > > +the device SHOULD perform the following actions before presenting
> > > SUSPEND bit in the \field{device status}:
> > > +
> > > +\begin{itemize}
> > > +\item Stop consuming buffers of any virtqueues and mark all finished
> > > descritors as used.
> > > +\item Wait until all descriptors that being processed to finish and mark
> > > them as used.
> > > +\item Flush all used buffer and send used buffer notifications to the
> > > driver.
> > flush how?
> This is device-type-specific, and we will include tracking inflight
> descriptors(buffers) in V2.
> >
> > > +\item Record Virtqueue State of each enabled virtqueue, see section
> > > \ref{sec:Virtqueues / Virtqueue State}
> >
> > record where?
> This is transport specific, for PCI, patch 5 introduces two new fields for
> avail and used state
they clearly can't store state for all vqs, these are just two 16 bit fields.
> >
> > > +\item Pause its operation except \field{device status} and preserve
> > > configurations in its Device Configuration Space, see \ref{sec:Basic
> > > Facilities of a Virtio Device / Device Configuration Space}
> > pause in what sense? completely? this does not seem realistic.
> > e.g. pci express link has to stay active or device will die.
> only pause virtio, I will rephrase the sentence as "pause its virtio
> operation".
that is vague too. for example what happens to link state of
a networking device?
> Others like PCI link in the example is out of the spec and we don't need
> to migrate them.
> >
> >
> > also, presumably here it is except a bunch of other fields.
> > e.g. what about queue select and all related queue fields?
> For now they are forbidden.
>
> As SiWei suggested, we will introduce a new feature bit to control whether
> allowing resetting a VQ after SUSPEND. We can use more feature bits if
> there are requirements to perform anything after SUSPEND. But for now
> they are forbidden.
I don't know how this means, but whatever. you need to make
all this explicit though.
> >
> > > +\end{itemize}
> > > +
> > > \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device /
> > > Feature Bits}
> > > Each virtio device offers all the features it understands. During
> > > @@ -937,6 +964,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
> > > Feature Bits}
> > > \ref{devicenormative:Basic Facilities of a Virtio Device /
> > > Feature Bits} for
> > > handling features reserved for future use.
> > > + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can
> > > + SUSPEND the device.
> > > + See \ref{sec:Basic Facilities of a Virtio Device / Device Status
> > > Field}.
> > > +
> > > \end{description}
> > > \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> > > --
> > > 2.35.3
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
