add QPL compression method introduction
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
docs/devel/migration/features.rst| 1 +
docs/devel/migration/qpl-compression.rst | 231 +++
2 files changed, 232 insertions(+)
create mode 100644 docs/devel/migration/qpl
add zlib and zstd compression levels in multifd parameter
testing and application and add compression level tests
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
Reported-by: Xiaohui Li
---
migration/options.c | 12
tests/qtest/migration-test.c | 16
2
algorithm
and use Intel In-Memory Analytics Accelerator(IAA) hardware for
compression and decompression acceleration.
Please refer to the following for more information about QPL
https://intel.github.io/qpl/documentation/introduction_docs/introduction.html
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai
method for migration.
How to enable qpl compression during migration:
migrate_set_parameter multifd-compression qpl
The qpl only supports one compression level, there is no qpl
compression level parameter added, users do not need to specify
the qpl compression level.
Signed-off-by: Yuan Liu
add qpl to compression method test for multifd migration
the migration with qpl compression needs to access IAA hardware
resource, please run "check-qtest" with sudo or root permission,
otherwise migration test will fail
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
t
this data
block. For no compression, each IOV is used to send a page, so the
number of IOVs required is the same as the number of pages.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration/multifd-zlib.c | 18 +-
migration/multifd-zstd.c | 18 +-
migration
each qpl job is used to (de)compress a normal page and it can
be processed independently by the IAA hardware. All qpl jobs
are submitted to the hardware at once, and wait for all jobs
completion.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration/multifd-qpl.c | 219
compression initialization will fail.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration/multifd-qpl.c | 128 ++--
1 file changed, 122 insertions(+), 6 deletions(-)
diff --git a/migration/multifd-qpl.c b/migration/multifd-qpl.c
index 6b94e732ac
tps://lore.kernel.org/all/CAAYibXib+TWnJpV22E=adncdBmwXJRqgRjJXK7X71J=bdfa...@mail.gmail.com/
2.
https://lore.kernel.org/all/ph7pr11mb594123f7eefebfce219af100a3...@ph7pr11mb5941.namprd11.prod.outlook.com/
Yuan Liu (1):
migration/multifd: solve zero page causing multiple page faults
migratio
unlikely that the zero page will be migrated multiple times.
Signed-off-by: Yuan Liu
---
migration/multifd-zero-page.c | 4 +++-
migration/multifd-zlib.c | 1 +
migration/multifd-zstd.c | 1 +
migration/multifd.c | 1 +
migration/ram.c | 4
migration/ram.h
add Intel Query Processing Library (QPL) compression method
introduction
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
docs/devel/migration/features.rst| 1 +
docs/devel/migration/qpl-compression.rst | 231 +++
2 files changed, 232 insertions(+)
create mode
add qpl to compression method test for multifd migration
the migration with qpl compression needs to access IAA hardware
resource, please run "check-qtest" with sudo or root permission,
otherwise migration test will fail
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
t
compression initialization will fail.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration/multifd-qpl.c | 243 +++-
1 file changed, 242 insertions(+), 1 deletion(-)
diff --git a/migration/multifd-qpl.c b/migration/multifd-qpl.c
index 056a68a060
method for migration.
How to enable qpl compression during migration:
migrate_set_parameter multifd-compression qpl
The qpl only supports one compression level, there is no qpl
compression level parameter added, users do not need to specify
the qpl compression level.
Signed-off-by: Yuan Liu
Different compression methods may require different numbers of IOVs.
Based on streaming compression of zlib and zstd, all pages will be
compressed to a data block, so two IOVs are needed for packet header
and compressed data block.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration
algorithm
and use Intel In-Memory Analytics Accelerator(IAA) hardware for
compression and decompression acceleration.
Please refer to the following for more information about QPL
https://intel.github.io/qpl/documentation/introduction_docs/introduction.html
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai
each qpl job is used to (de)compress a normal page and it can
be processed independently by the IAA hardware. All qpl jobs
are submitted to the hardware at once, and wait for all jobs
completion.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration/multifd-qpl.c | 229
Different compression methods may require different numbers of IOVs.
Based on streaming compression of zlib and zstd, all pages will be
compressed to a data block, so two IOVs are needed for packet header
and compressed data block.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration
In-Memory Analytics Accelerator(IAA) hardware for compression
and decompression acceleration.
For more live migration with IAA, please refer to the document
docs/devel/migration/qpl-compression.rst
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
meson.build | 8
.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
hw/core/qdev-properties-system.c | 2 +-
migration/meson.build| 1 +
migration/multifd-qpl.c | 20
migration/multifd.h | 1 +
qapi/migration.json | 7 ++-
5 files changed
: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration/multifd-qpl.c | 284 +++-
1 file changed, 280 insertions(+), 4 deletions(-)
diff --git a/migration/multifd-qpl.c b/migration/multifd-qpl.c
index 89fa51091a..9a1fddbdd0 100644
--- a/migration/multifd-qpl.c
+++ b
: Yuan Liu
Reviewed-by: Nanhai Zou
---
tests/qtest/migration-test.c | 24
1 file changed, 24 insertions(+)
diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
index 5d6d8cd634..0f75ed7c49 100644
--- a/tests/qtest/migration-test.c
+++ b/tests/qtest
not
available, the qpl compression will fallback to the software path.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration/multifd-qpl.c | 272 +++-
1 file changed, 271 insertions(+), 1 deletion(-)
diff --git a/migration/multifd-qpl.c b/migration
add Intel Query Processing Library (QPL) compression method
introduction
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
docs/devel/migration/features.rst| 1 +
docs/devel/migration/qpl-compression.rst | 262 +++
2 files changed, 263 insertions(+)
create mode
In-Memory Analytics Accelerator(IAA) hardware for compression
and decompression acceleration.
For more live migration with IAA, please refer to the document
docs/devel/migration/qpl-compression.rst
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
meson.build | 8
compression and decompression CPU overhead.
How to enable qpl compression during migration:
migrate_set_parameter multifd-compression qpl
There is no qpl compression level parameter added since it only supports
level one, users do not need to specify the qpl compression level.
Signed-off-by: Yuan Liu
add Intel Query Processing Library (QPL) compression method
introduction
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
docs/devel/migration/features.rst| 1 +
docs/devel/migration/qpl-compression.rst | 262 +++
2 files changed, 263 insertions(+)
create mode
during initialization, a software job is allocated to each channel
for software path fallabck when the IAA hardware is unavailable or
the hardware job submission fails. If the IAA hardware is available,
multiple hardware jobs are allocated for batch processing.
Signed-off-by: Yuan Liu
Reviewed
: Yuan Liu
Reviewed-by: Nanhai Zou
Reviewed-by: Peter Xu
---
tests/qtest/migration-test.c | 24
1 file changed, 24 insertions(+)
diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
index b7e3406471..ef0c3f5e28 100644
--- a/tests/qtest/migration-test.c
Different compression methods may require different numbers of IOVs.
Based on streaming compression of zlib and zstd, all pages will be
compressed to a data block, so two IOVs are needed for packet header
and compressed data block.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
Reviewed-by
QPL compression and decompression will use IAA hardware first.
If IAA hardware is not available, it will automatically fall
back to QPL software path, if the software job also fails,
the uncompressed page is sent directly.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration/multifd
compression and decompression CPU overhead.
How to enable qpl compression during migration:
migrate_set_parameter multifd-compression qpl
There is no qpl compression level parameter added since it only supports
level one, users do not need to specify the qpl compression level.
Signed-off-by: Yuan Liu
Different compression methods may require different numbers of IOVs.
Based on streaming compression of zlib and zstd, all pages will be
compressed to a data block, so two IOVs are needed for packet header
and compressed data block.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
Reviewed-by
during initialization, a software job is allocated to each channel
for software path fallabck when the IAA hardware is unavailable or
the hardware job submission fails. If the IAA hardware is available,
multiple hardware jobs are allocated for batch processing.
Signed-off-by: Yuan Liu
Reviewed
In-Memory Analytics Accelerator(IAA) hardware for compression
and decompression acceleration.
For more live migration with IAA, please refer to the document
docs/devel/migration/qpl-compression.rst
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
Reviewed-by: Fabiano Rosas
---
meson.build
: Yuan Liu
Reviewed-by: Nanhai Zou
Reviewed-by: Peter Xu
Reviewed-by: Fabiano Rosas
---
tests/qtest/migration-test.c | 24
1 file changed, 24 insertions(+)
diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
index b7e3406471..ef0c3f5e28 100644
--- a
add Intel Query Processing Library (QPL) compression method
introduction
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
Reviewed-by: Fabiano Rosas
Acked-by: Peter Xu
---
docs/devel/migration/features.rst| 1 +
docs/devel/migration/qpl-compression.rst | 260
fallback, always wait for work queues to become available
total time: 18381 ms
downtime: 25 ms
throughput: 13698.65 mbps
pages-per-second: 859607
If both the hardware and software paths fail, the uncompressed page is
sent directly.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
Refactor legacy RAM compression functions to support both IAA
compression and CPU compression.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration/migration.c| 6 +--
migration/ram-compress.c | 81
migration/ram-compress.h | 10
compress on. If the migration compression
capability is enabled and the IAA compression parameter is set, IAA will
be used instead of CPU for data (de)compression.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration/migration-hmp-cmds.c | 8
migration/options.c| 20
This patch defines the structure for IAA jobs related to data
compression and decompression, as well as the initialization and
deinitialization processes for IAA.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration/iaa-ram-compress.c | 152
data (de)compression
using IAA during the live migration process.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
meson.build | 9 -
meson_options.txt | 2 ++
scripts/meson-buildoptions.sh | 3 +++
3 files changed, 13 insertions(+), 1 deletion(-)
diff
, resulting in performance gains.
The implementation of the IAA (de)compression code is based on Intel Query
Processing Library (QPL), an open-source software project designed for
IAA high-level software programming.
Best regards,
Yuan Liu
Yuan Liu (5):
configure: add qpl meson option
qapi
Implement the functions of IAA for data compression and decompression.
The implementation uses non-blocking job submission and polling to check
the job completion status to reduce IAA's overhead in the live migration
process.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migratio
Zlib compression algorithm. QPL can automatically choose
software or hardware acceleration based on the platform.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration/meson.build | 1 +
migration/multifd-qpl.c | 323
2 files changed, 324
during the live migration.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
meson.build | 18 ++
meson_options.txt | 2 ++
scripts/meson-buildoptions.sh | 3 +++
3 files changed, 23 insertions(+)
diff --git a/meson.build b/meson.build
index
ction in multifd.c file
Yuan Liu (4):
migration: Introduce multifd-compression-accel parameter
multifd: Implement multifd compression accelerator
configure: add qpl option
multifd: Introduce QPL compression accelerator
hw/core/qdev-properties-system.c| 11 +
include/hw/qdev-prope
acceleration function can be disabled.
Similarly, users can explicitly specify a specific accelerator name, such
as multifd-compression-accel=qpl.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
hw/core/qdev-properties-system.c| 11 ++
include/hw/qdev-properties-system.h | 4
when starting multifd live migration, if the compression method is
enabled, compression method can be accelerated using accelerators.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
Reviewed-by: Fabiano Rosas
---
migration/multifd.c | 40 ++--
migration
when starting multifd live migration, if the compression method is
enabled, compression method can be accelerated using accelerators.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration/multifd.c | 38 --
migration/multifd.h | 8
2 files
during the live migration.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
meson.build | 7 +++
meson_options.txt | 2 ++
scripts/meson-buildoptions.sh | 3 +++
3 files changed, 12 insertions(+)
diff --git a/meson.build b/meson.build
index 259dc5f308
Zlib compression algorithm. QPL can automatically choose
software or hardware acceleration based on the platform.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
migration/meson.build | 1 +
migration/multifd-qpl.c | 326
2 files changed, 327
acceleration function can be disabled.
Similarly, users can explicitly specify a specific accelerator name, such
as multifd-compression-accel=qpl.
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
hw/core/qdev-properties-system.c| 11 +++
include/hw/qdev-properties-system.h | 4
rator in the multifd
compression accelerator
- fixed the issue that QPL was compiled into the migration
module by default
Yuan Liu (4):
migration: Introduce multifd-compression-accel parameter
multifd: Implement multifd compression accelerator
configure: add qpl option
multifd: Intr
add Intel QATzip compression method introduction
Signed-off-by: Yuan Liu
Reviewed-by: Nanhai Zou
---
docs/devel/migration/features.rst | 1 +
docs/devel/migration/qatzip-compression.rst | 112
2 files changed, 113 insertions(+)
create mode 100644 docs/devel
The page_size member has been removed from the MultiFDSendParams
and MultiFDRecvParams. The function multifd_ram_page_size is used to
provide the page size in the multifd compressor.
Signed-off-by: Yuan Liu
---
migration/multifd-qpl.c | 10 +-
1 file changed, 5 insertions(+), 5
: Yuan Liu
Reviewed-by: Jason Zeng
---
migration/multifd-nocomp.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/migration/multifd-nocomp.c b/migration/multifd-nocomp.c
index 55191152f9..2e4aaac285 100644
--- a/migration/multifd-nocomp.c
+++ b/migration/multifd-nocomp.c
and
to export the memory data of the source and target VMs respectively.
4. Use "cmp -l source_memory target_memory" to verify memory data.
Yuan Liu (3):
multifd: bugfix for migration using compression methods
multifd: bugfix for incorrect migration data with QPL compression
.
The root cause is that the target side does not record the normal pages
to the receivedmap.
The solution is to add ramblock_recv_bitmap_set_offset in target side
to record the normal pages.
Signed-off-by: Yuan Liu
Reviewed-by: Jason Zeng
---
migration/multifd-qatzip.c | 1 +
1 file changed, 1
.
The root cause is that the target side does not record the normal pages
to the receivedmap.
The solution is to add ramblock_recv_bitmap_set_offset in target side
to record the normal pages.
Signed-off-by: Yuan Liu
Reviewed-by: Jason Zeng
---
migration/multifd-qpl.c | 1 +
1 file changed, 1
60 matches
Mail list logo
