> -----Original Message----- > From: Qemu-devel <qemu-devel- > bounces+eddie.dong=intel....@nongnu.org> On Behalf Of Alex Williamson > Sent: Thursday, May 26, 2022 11:44 AM > To: Rao, Lei <lei....@intel.com> > Cc: Tian, Kevin <kevin.t...@intel.com>; Dong, Eddie > <eddie.d...@intel.com>; Zeng, Jason <jason.z...@intel.com>; > quint...@redhat.com; dgilb...@redhat.com; Li, Yadong > <yadong...@intel.com>; Liu, Yi L <yi.l....@intel.com>; qemu- > de...@nongnu.org > Subject: Re: [RFC PATCH 00/13] Add a plugin to support out-of-band live > migration for VFIO pass-through device > > On Tue, 24 May 2022 14:18:35 +0800 > Lei Rao <lei....@intel.com> wrote: > > > Migration of a VFIO passthrough device can be supported by using a > > device specific kernel driver to save/restore the device state thru > > device specific interfaces. But this approach doesn't work for devices > > that lack a state migration interface, e.g. NVMe. > > > > On the other hand, Infrastructure Process Unit (IPU) or Data > > Processing Unit > > (DPU) vendors may choose to implement an out-of-band interface from > > the SoC to help manage the state of such non-migratable devices e.g. > > via gRPC or JSON-RPC protocols. > > > > This RFC attempts to support such out-of-band migration interface by > > introducing the concept of migration backends in vfio. The existing > > logic around vfio migration uAPI is now called the 'local' backend while a > new 'out-of-band' > > backend is further introduced allowing vfio to redirect VMState ops to > > an external plugin. > > > > Currently, the backend migration Ops is defined close to > > SaveVMHandlers. We also considered whether there is value of > > abstracting it in a lower level e.g. close to vfio migration uAPI but > > no clear conclusion. Hence this is one part which we'd like to hear > suggestions. > > > > This proposal adopts a plugin mechanism (an example can be found in > > [1]) given that IPU/DPU vendors usually implement proprietary > > migration interfaces without a standard. But we are also open if an > > alternative option makes better sense, e.g. via loadable modules (with > > Qemu supporting gRPC or JSON-RPC support) or an IPC mechanism similar > to vhost-user. > > AFAIU, QEMU is not interested in supporting plugin modules, especially > proprietary ones. I don't see that a case has really been made that this > cannot be done in-band, through a vfio-pci variant driver, possibly making > use of proprietary interfaces to a userspace agent if necessary (though > please don't ask such to be accepted in-tree for the kernel either). A vfio- > user device server might also host such proprietary, device specific agents > while supporting the standard, in-band migration interface. Thanks, >
Thanks Alex. Removing plug-in module is not a problem. Do you mean to implement the migration and protocol handling inside Qemu-client (probably vfio-client after the VFIO-user is merged)? Or to build as part of libvfio-user? We can also build it as a separate process of Qemu-server as part of Multi-Process Qemu. In here, the protocol between host CPU and SoC is based on gRPC, which support Rust code in client (Host CPU side here) more than C/C++. Do you have any suggestion to better support Rust code with Qemu C/C++ code? Thx Eddie > Alex > > > > > The following graph describes the overall component relationship: > > > > +----------------------------------------------------+ > > | QEMU | > > | +------------------------------------------------+ | > > | | VFIO Live Migration Framework | | > > | | +--------------------------------------+ | | > > | | | VFIOMigrationOps | | | > > | | +-------^---------------------^--------+ | | > > | | | | | | > > | | +-------v-------+ +-------v--------+ | | > > | | | LM Backend Via| | LM Backend Via | | | > > | | | Device Fd | | Plugins | | | > > | | +-------^-------+ | +----------+ | | > > | | | | |Plugin Ops+----+-+------------+ > > | | | +-----+----------+ | | | > > | | | | | > > +---------v----------+ > > | +------------+-----------------------------------+ | | Vendor Specific > > | > > | | | | Plugins(.so) > > | > > +--------------+-------------------------------------+ > > +----------+---------+ > > UserSpace | | > > ----------------+--------------------------------------------- | > > Kernel | | > > | | > > +----------v----------------------+ | > > | Kernel VFIO Driver | | > > | +-------------------------+ | | > > | | | | | > > Network > > | | Vendor-Specific Driver | | | > > | | | | | > > | +----------^--------------+ | | > > | | | | > > +---------------+-----------------+ | > > | | > > | | > > ---------------------+----------------------------------------- | > > Hardware | | > > | +-----+-----+-----+----+-----+ | > > +----------v------+ | VF0 | VF1 | VF2 | ...| VFn | | > > | Traditional | +-----+-----+-----+----+-----+ | > > | PCIe Devices | | | | > > +-----------------+ | +--------+------------+ | | > > | | | Agent |<-+----+ > > | | +------------+ | > > | | | | > > | | SOC | | > > | +---------------------+ | > > | IPU | > > +----------------------------+ > > > > Two command-line parameters (x-plugin-path and x-plugin-arg) are > > introduced to enable the out-of-band backend. If specified, vfio will > > attempt to use the out-of-band backend. > > > > The following is an example of VFIO command-line parameters for OOB- > Approach: > > > > -device > > vfio-pci,id=$ID,host=$bdf,x-enable-migration,x-plugin-path=$plugin_pat > > h,x-plugin-arg=$plugin_arg > > > > [1] https://github.com/raolei-intel/vfio-lm-plugin-example.git > > > > Lei Rao (13): > > vfio/migration: put together checks of migration initialization > > conditions > > vfio/migration: move migration struct allocation out of > > vfio_migration_init > > vfio/migration: move vfio_get_dev_region_info out of > > vfio_migration_probe > > vfio/migration: Separated functions that relate to the In-Band > > approach > > vfio/migration: rename functions that relate to the In-Band approach > > vfio/migration: introduce VFIOMigrationOps layer in VFIO live > > migration framework > > vfio/migration: move the statistics of bytes_transferred to generic > > VFIO migration layer > > vfio/migration: split migration handler registering from > > vfio_migration_init > > vfio/migration: move the functions of In-Band approach to a new file > > vfio/pci: introduce command-line parameters to specify migration > > method > > vfio/migration: add a plugin layer to support out-of-band live > > migration > > vfio/migration: add some trace-events for vfio migration plugin > > vfio/migration: make the region and plugin member of struct > > VFIOMigration to be a union > > > > docs/devel/vfio-migration-plugin.rst | 165 +++++++ > > hw/vfio/meson.build | 2 + > > hw/vfio/migration-local.c | 456 +++++++++++++++++++ > > hw/vfio/migration-plugin.c | 266 +++++++++++ > > hw/vfio/migration.c | 577 ++++++------------------ > > hw/vfio/pci.c | 2 + > > hw/vfio/trace-events | 9 +- > > include/hw/vfio/vfio-common.h | 37 +- > > include/hw/vfio/vfio-migration-plugin.h | 21 + > > 9 files changed, 1096 insertions(+), 439 deletions(-) create mode > > 100644 docs/devel/vfio-migration-plugin.rst > > create mode 100644 hw/vfio/migration-local.c create mode 100644 > > hw/vfio/migration-plugin.c create mode 100644 > > include/hw/vfio/vfio-migration-plugin.h > > >
