Yes, the HW-GZIP is able to work on multiple threads too, but the test program lzbench https://github.com/inikep/lzbench seems work on single thread, so I can't run it with multiple threads.
Thanks, XieQi -----Original Message----- From: Antoine Pitrou <anto...@python.org> Sent: Thursday, October 22, 2020 4:30 PM To: Xie, Qi <qi....@intel.com>; dev <dev@arrow.apache.org> Cc: Xu, Cheng A <cheng.a...@intel.com>; Dong, Xin <xin.d...@intel.com>; Zhang, Jie1 <jie1.zh...@intel.com> Subject: Re: [Discuss] Provide pluggable APIs to support user customized compression codec Le 22/10/2020 à 05:38, Xie, Qi a écrit : > Hi, > > I just tested with the Intel QuickAssist Technology, which provide hardware > accelerate to GZIP, you can see detail here > https://www.intel.com/content/www/us/en/architecture-and-technology/intel-quick-assist-technology-overview.html > > > Here is the benchmark result run on Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz > with single thread > > lzbench 1.7.2 (64-bit Linux) Assembled by P.Skibinski > | Compressor name | Compression| Decompress.| Compr. size | Ratio | > Filename | > | memcpy | 4942 MB/s | 5688 MB/s | 3263523 | 1.00 | > calgary/calgary.tar | > | qat 1.0.0 | 2312 MB/s | 3538 MB/s | 1274379 | 2.56 | > calgary/calgary.tar | > | snappy 1.1.4 | 283 MB/s | 1144 MB/s | 1686240 | 1.94 | > calgary/calgary.tar | > | lz4 1.7.5 | 453 MB/s | 2514 MB/s | 1685795 | 1.94 > | calgary/calgary.tar | > | zstd 1.3.1 -1 | 279 MB/s | 723 MB/s | 1187211 | 2.75 | > calgary/calgary.tar | > | zlib 1.2.11 -1 | 79 MB/s | 261 MB/s | 1240838 | 2.63 > | calgary/calgary.tar | Very nice, thank you. Is it able to work on multiple threads too? Regards Antoine.
