> -----Original Message----- > From: Cao, Yahui <yahui....@intel.com> > Sent: Friday, July 7, 2023 2:22 PM > To: Liu, Lingyu <lingyu....@intel.com>; dev@dpdk.org; Zhang, Qi Z > <qi.z.zh...@intel.com>; Xing, Beilei <beilei.x...@intel.com> > Subject: Re: [PATCH v4] doc: add iavf live migration guide > > > On 7/7/2023 2:08 PM, Lingyu Liu wrote: > > Add iavf live migration steps based on KVM VFIO migration. > > > > Signed-off-by: Lingyu Liu <lingyu....@intel.com> > > --- > > v2: Fixed CI. > > Added brief introduction about live migration. > > Clarified this is iavf feature. > > > > v3: Added intro and link about vfio live migration. > > Added description about kernel boot parameters. > > Changed to use sysfs to bind device to driver. > > Noted for running dpdk-testpmd. > > Highlighted KVM vfio migration. > > > > v4: Removed ice kernel driver version. > > --- > > doc/guides/nics/intel_vf.rst | 113 +++++++++++++++++++++++++ > > doc/guides/rel_notes/release_23_07.rst | 3 + > > 2 files changed, 116 insertions(+) > > > > diff --git a/doc/guides/nics/intel_vf.rst > > b/doc/guides/nics/intel_vf.rst index d365dbc185..85b3b4fc24 100644 > > --- a/doc/guides/nics/intel_vf.rst > > +++ b/doc/guides/nics/intel_vf.rst > > @@ -622,6 +622,119 @@ which belongs to the destination VF on the VM. > > Inter-VM Communication > > > > > > +Live Migrating a VM running DPDK > > +-------------------------------- > > + > > +Live migration refers to the process of moving a running virtual > > +machine (VM) or application between different physical machines > > +without disconnecting the client or application (see > https://en.wikipedia.org/wiki/Live_migration for more information). > > + > > +VFIO device migration refers to migrating a VM which have VFIO device > > +pass-through (see https://qemu.readthedocs.io/en/latest/devel/vfio- > migration.html for more information). > > + > > +This part describes stpes to migrate a VM which has a iavf device pass > through. > > + > > +The following describes a target environment: > > + > > +* Host Operating System: Ubuntu 20.04.5 > > + > > +* Guest Operating System: Ubuntu 20.04.5 > > + > > +* Linux Kernel Version: 5.15.0-72-generic > > + > > +* Target Applications: dpdk-testpmd > > + > > +* Ice Kernel Driver: Download from > `<https://www.intel.com/content/www/us/en/download/19630>`_ > > + > > +* Qemu Version: 7.2 > > + > > +The setup procedure is as follows: > > + > > +#. Before booting the Host OS, open **BIOS setup** and enable **Intel® > VT features**. > > + > > +#. While booting the Host OS kernel, pass the intel_iommu=on kernel > command line argument using GRUB. > > + > > +#. In the Host OS > > + > > + Install the ice driver and migration driver: > > + > > + .. code-block:: console > > + > > + insmod ice.ko > > + insmod ice-vfio-pci.ko > > + > > + Create 2 VFs and bind them to vfio pci driver: > > + > > + .. code-block:: console > > + > > + echo 2 > /sys/bus/pci/devices/0000:ca:00.1/sriov_numvfs > > + echo "8086 1889" > /sys/bus/pci/drivers/ice-vfio-pci/new_id > > + echo 0000:ca:11.0 > /sys/bus/pci/devices/0000:ca:11.0/driver/unbind > > + echo 0000:ca:11.0 > /sys/bus/pci/drivers/ice-vfio-pci/bind > > + echo 0000:ca:11.1 > /sys/bus/pci/devices/0000:ca:11.1/driver/unbind > > + echo 0000:ca:11.1 > /sys/bus/pci/drivers/ice-vfio-pci/bind > > + > > + .. note:: > > + > > + The command above creates two vfs for device 0000:ca:00.1: > > + > > + .. code-block:: console > > + > > + 0000:ca:11.0 'Ethernet Adaptive Virtual Function 1889' if= > > drv=ice-vfio- > pci unused=iavf > > + 0000:ca:11.1 'Ethernet Adaptive Virtual Function 1889' if= > > + drv=ice-vfio-pci unused=iavf > > + > > +#. Now, start the migration source Virtual Machine by running the > following command: > > + > > + .. code-block:: console > > + > > + qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu > > + host -m 4G -smp 1 -device > > + vfio-pci,host=0000:ca:11.0,x-enable-migration=true,x-pre-copy-dirty- > > + page-tracking=off -drive file=ubuntu-2004.qcow2 -nic > > + user,hostfwd=tcp::5555-:22 -monitor stdio > > + > > + .. note:: > > + The vfio-pci,host=0000:ca:11.0 value indicates that you want to > > attach > a vfio PCI device > > + to a Virtual Machine and the respective (Bus:Device.Function) > > numbers > should be passed, > > + x-enable-migration=true indicates that this VF supports migration. > > Dirty > page tracking > > + is not supported, so set x-pre-copy-dirty-page-tracking=off. > > + > > +#. In VM, install iavf driver and vfio-pci driver > > + > > + .. code-block:: console > > + > > + insmod iavf.ko > > + modprobe vfio enable_unsafe_noiommu_mode=1 > > + moodprobe vfio-pci > > + > > +#. Bind net device to vfio-pci driver and launch dpdk-testpmd > > + > > + .. code-block:: console > > + > > + dpdk-testpmd -l 0-1 -- -i > > + testpmd> set txpkts 64 > > + testpmd> start tx_first > > + > > + .. note:: > > + Please ensure dpdk-testpmd to run independently of ssh console. > > + Suggest to put it in a background process like tmux/screen so that > > + migration will not causing ssh console exit and dpdk-testpmd > > killed. > > + > > +#. Start the migration destination Virtual Machine > > + > > + .. code-block:: console > > + > > + qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu > > + host -m 4G -smp 1 -device > > + vfio-pci,host=0000:ca:11.1,x-enable-migration=true,x-pre-copy-dirty- > > + page-tracking=off -drive file=ubuntu-2004.qcow2 -nic > > + user,hostfwd=tcp::5556-:22 -monitor stdio -incoming > > + tcp:127.0.0.1:4444 > > + > > + #. Start migration by issuing the command in qemu console > > + > > + .. code-block:: console > > + > > + migrate -d tcp:127.0.0.1:4444 > > + > > +#. Log in the destination VM, and dpdk-testpmd is not interrupt > > + > > +.. note:: > > + > > + Host kernel version above 5.17 is currently not supported due to > > kAPI > change. > > + > > + > > Windows Support > > --------------- > > > > diff --git a/doc/guides/rel_notes/release_23_07.rst > > b/doc/guides/rel_notes/release_23_07.rst > > index 4459144140..794a86eba3 100644 > > --- a/doc/guides/rel_notes/release_23_07.rst > > +++ b/doc/guides/rel_notes/release_23_07.rst > > @@ -200,6 +200,9 @@ New Features > > > > Enhanced the GRO library to support TCP packets over IPv6 network. > > > > +* **Updated Intel iavf driver.** > > + > > + Added the document for iavf driver live migration based on KVM vfio > migration. > > > > Removed Items > > ------------- > > > Reviewed-by: Yahui Cao <yahui....@intel.com>‘
Applied to dpdk-next-net-intel. Thanks Qi
