This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new d33a3eb1c5e [cherry-pick](branch-2.1) Pick "[Fix](LZ4 compression) Fix
wrong LZ4 compression max input size limit (#41239)" (#41505)
d33a3eb1c5e is described below
commit d33a3eb1c5ea417fc059d29c139cbc6f1721dd34
Author: abmdocrt <[email protected]>
AuthorDate: Tue Oct 1 22:43:11 2024 +0800
[cherry-pick](branch-2.1) Pick "[Fix](LZ4 compression) Fix wrong LZ4
compression max input size limit (#41239)" (#41505)
## Proposed changes
LZ4 compression max supported value is LZ4_MAX_INPUT_SIZE, which is
0x7E000000(2,113,929,216 bytes). Doris use wrong max size INT_MAX, which
is 2,147,483,647, to check. If input data size is between this two size,
then it can pass the check but LZ4 compression will fail.
This PR fix it.
<!--Describe your changes.-->
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
---
be/src/util/block_compression.cpp | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/be/src/util/block_compression.cpp
b/be/src/util/block_compression.cpp
index cf8b8a62062..8959e9e6b2c 100644
--- a/be/src/util/block_compression.cpp
+++ b/be/src/util/block_compression.cpp
@@ -127,11 +127,13 @@ public:
}
Status compress(const Slice& input, faststring* output) override {
- if (input.size > INT_MAX) {
+ if (input.size > LZ4_MAX_INPUT_SIZE) {
return Status::InvalidArgument(
- "LZ4 not support those case(input.size>INT_MAX), maybe you
should change "
- "fragment_transmission_compression_codec to snappy,
size={}",
- input.size);
+ "LZ4 not support those
case(input.size>LZ4_MAX_INPUT_SIZE), maybe you should "
+ "change "
+ "fragment_transmission_compression_codec to snappy,
input.size={}, "
+ "LZ4_MAX_INPUT_SIZE={}",
+ input.size, LZ4_MAX_INPUT_SIZE);
}
std::unique_ptr<Context> context;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
