> -----Original Message----- > From: Shameer Kolothum <shameerali.kolothum.th...@huawei.com> > Sent: Wednesday, May 29, 2024 5:44 PM > To: pet...@redhat.com; faro...@suse.de; Liu, Yuan1 <yuan1....@intel.com> > Cc: qemu-devel@nongnu.org; linux...@huawei.com; linwenk...@hisilicon.com; > zhangfei....@linaro.org; huangchengh...@huawei.com > Subject: [PATCH 1/7] docs/migration: add uadk compression feature > > Document UADK(User Space Accelerator Development Kit) library details > and how to use that for migration. > > Signed-off-by: Shameer Kolothum <shameerali.kolothum.th...@huawei.com> > --- > docs/devel/migration/uadk-compression.rst | 144 ++++++++++++++++++++++ > 1 file changed, 144 insertions(+) > create mode 100644 docs/devel/migration/uadk-compression.rst > > diff --git a/docs/devel/migration/uadk-compression.rst > b/docs/devel/migration/uadk-compression.rst > new file mode 100644 > index 0000000000..988b92631e > --- /dev/null > +++ b/docs/devel/migration/uadk-compression.rst > @@ -0,0 +1,144 @@ > +========================================================= > +User Space Accelerator Development Kit (UADK) Compression > +========================================================= > +UADK is a general-purpose user space accelerator framework that uses > shared > +virtual addressing (SVA) to provide a unified programming interface for > +hardware acceleration of cryptographic and compression algorithms. > + > +UADK includes Unified/User-space-access-intended Accelerator Framework > (UACCE), > +which enables hardware accelerators from different vendors that support > SVA to > +adapt to UADK. > + > +Currently, HiSilicon Kunpeng hardware accelerators have been registered > with > +UACCE. Through the UADK framework, users can run cryptographic and > compression > +algorithms using hardware accelerators instead of CPUs, freeing up CPU > +computing power and improving computing performance. > + > +https://github.com/Linaro/uadk/tree/master/docs > + > +UADK Framework > +============== > +UADK consists of UACCE, vendors' drivers, and an algorithm layer. UADK > requires > +the hardware accelerator to support SVA, and the operating system to > support > +IOMMU and SVA. Hardware accelerators from different vendors are > registered as > +different character devices with UACCE by using kernel-mode drivers of > the > +vendors. A user can access the hardware accelerators by performing user- > mode > +operations on the character devices. > + > +:: > + > + +----------------------------------+ > + | apps | > + +----+------------------------+----+ > + | | > + | | > + +-------+--------+ +-------+-------+ > + | scheduler | | alg libraries | > + +-------+--------+ +-------+-------+ > + | | > + | | > + | | > + | +--------+------+ > + | | vendor drivers| > + | +-+-------------+ > + | | > + | | > + +--+------------------+--+ > + | libwd | > + User +----+-------------+-----+ > + -------------------------------------------------- > + Kernel +--+-----+ +------+ > + | uacce | | smmu | > + +---+----+ +------+ > + | > + +---+------------------+ > + | vendor kernel driver | > + +----------------------+ > + -------------------------------------------------- > + +----------------------+ > + | HW Accelerators | > + +----------------------+ > + > +UADK Installation > +----------------- > +Build UADK > +^^^^^^^^^^ > + > +.. code-block:: shell > + > + git clone https://github.com/Linaro/uadk.git > + cd uadk > + mkdir build > + ./autogen.sh > + ./configure --prefix=$PWD/build > + make > + make install > + > +Without --prefix, UADK will be installed to /usr/local/lib by default. > +If get error:"cannot find -lnuma", please install the libnuma-dev > + > +Run pkg-config libwd to ensure env is setup correctly > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > + > +* export PKG_CONFIG_PATH=$PWD/build/lib/pkgconfig > +* pkg-config libwd --cflags --libs > + -I/usr/local/include -L/usr/local/lib -lwd > + > +* export PKG_CONFIG_PATH is required on demand. > + Not required if UADK is installed to /usr/local/lib > + > +UADK Host Kernel Requirements > +----------------------------- > +User needs to make sure that ``UACCE`` is already supported in Linux > kernel. > +The kernel version should be at least v5.9 with SVA (Shared Virtual > +Addressing) enabled. > + > +Kernel Configuration > +^^^^^^^^^^^^^^^^^^^^ > + > +``UACCE`` could be built as module or built-in. > + > +Here's an example to enable UACCE with hardware accelerator in HiSilicon > +Kunpeng platform. > + > +* CONFIG_IOMMU_SVA_LIB=y > +* CONFIG_ARM_SMMU=y > +* CONFIG_ARM_SMMU_V3=y > +* CONFIG_ARM_SMMU_V3_SVA=y > +* CONFIG_PCI_PASID=y > +* CONFIG_UACCE=y > +* CONFIG_CRYPTO_DEV_HISI_QM=y > +* CONFIG_CRYPTO_DEV_HISI_ZIP=y > + > +Make sure all these above kernel configurations are selected. > + > +Accelerator dev node permissions > +-------------------------------- > +Harware accelerators(eg: HiSilicon Kunpeng Zip accelerator) gets > registered to > +UADK and char devices are created in dev directory. In order to access > resources > +on hardware accelerator devices, write permission should be provided to > user. > + > +.. code-block:: shell > + > + $ sudo chmod 777 /dev/hisi_zip-* > + > +How To Use UADK Compression In Qemu Migration > +--------------------------------------------- > +* Make sure UADK is installed as above > +* Build ``Qemu`` with ``--enable-uadk`` parameter > + > + E.g. configure --target-list=aarch64-softmmu --enable-kvm ``--enable- > uadk`` > + > +* Enable ``UADK`` compression during migration > + > + Set ``migrate_set_parameter multifd-compression uadk`` > + > +Since UADK uses Shared Virtual Addressing(SVA) and device access virtual > memory > +directly it is possible that SMMUv3 may enounter page faults while > walking the > +IO page tables. This may impact the performance. In order to mitigate > this, > +please make sure to specify ``-mem-prealloc`` parameter to the > destination VM > +boot parameters.
Thank you so much for putting the IAA solution at the top and cc me. I think migration performance will be better with '-mem-prealloc' option, but I am considering whether '-mem-prealloc' is a mandatory option, from my experience, SVA performance drops mainly caused by IOTLB flush and IO page fault, I had some discussions with Peter Xu about the IOTLB flush issue, and it has been improved. https://patchew.org/QEMU/ph7pr11mb5941f04fbfb964cb2c968866a3...@ph7pr11mb5941.namprd11.prod.outlook.com/ For IO page fault, the QPL(IAA userspace library) can process page fault request instead of IOMMU, it means we can disable the I/O page fault feature on the IAA device, and let the device still use SVA technology to avoid memory copy. I will provide the test results in my next version, do you have any ideas or suggestions about this, thanks. > +Though both UADK and ZLIB are based on the deflate compression algorithm, > UADK > +is not fully compatible with ZLIB. Hence, please make sure to use > ``uadk`` on > +both source and destination during migration. > -- > 2.17.1
