Surya Hebbar has uploaded a new patch set (#7). (
http://gerrit.cloudera.org:8080/22718 )
Change subject: IMPALA-13923: Support more compression levels for ZSTD and ZLIB
......................................................................
IMPALA-13923: Support more compression levels for ZSTD and ZLIB
This patch adds support for more compression levels for ZLIB and ZSTD,
and expands the negative range of ZSTD compression levels.
The following additional compression levels are now supported.
For ZSTD,
ZSTD_minCLevel(-ZSTD_TARGETLENGTH_MAX) to ZSTD_maxCLevel(20)
For ZLIB i.e. ZLIB, GZIP and DEFLATE,
Z_DEFAULT_COMPRESSION(-1) to Z_BEST_COMPRESSION(9)
These are now supported with the "compression_codec" query option.
This has been implemented by refactoring compression levels as an
optional parameter in CodecInfo. This also solves IMPALA-10630.
Usage of compression level has been refactored with std::optional in
- exec/parquet/hdfs-parquet-table-writer
- runtime/tmp-file-mgr
- service/query-options
To validate compression levels externally, the following method has
been added
- Status Codec::ValidateCompressionLevel
Added tests for
* Additional compression levels for GZIP
- from Z_BEST_SPEED, Z_BEST_COMPRESSION
* Negative compression levels for ZSTD
- Lowered the range to ZSTD_minCLevel()
- Seperated tests for negative compression levels
* Query option - "compression_codec" for the newly added formats
and compression levels
Change-Id: I5b98c735246f08e04598a4e752c8cca04e31a88a
---
M be/src/exec/parquet/hdfs-parquet-table-writer.cc
M be/src/runtime/tmp-file-mgr.h
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/util/codec.cc
M be/src/util/codec.h
M be/src/util/compress.cc
M be/src/util/compress.h
M be/src/util/decompress-test.cc
M be/src/util/parse-util.cc
M be/src/util/parse-util.h
11 files changed, 163 insertions(+), 76 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/22718/7
--
To view, visit http://gerrit.cloudera.org:8080/22718
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5b98c735246f08e04598a4e752c8cca04e31a88a
Gerrit-Change-Number: 22718
Gerrit-PatchSet: 7
Gerrit-Owner: Surya Hebbar <sheb...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ara...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <ale...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sheb...@cloudera.com>
