This is an automated email from the ASF dual-hosted git repository.
airborne12 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 428c24187ce [improvement](be) Let SNII skip zstd for .prx windows
where it saves little over PFOR (#69649)
428c24187ce is described below
commit 428c24187ce94c5dd6168d7b2b822e572a8381fc
Author: Jack <[email protected]>
AuthorDate: Wed Oct 28 07:41:34 2026 +0800
[improvement](be) Let SNII skip zstd for .prx windows where it saves little
over PFOR (#69649)
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The auto .prx codec stores each position window in the
smallest of PFOR, zstd and
raw. Every read of a zstd window pays a decompression that PFOR does
not, and the choice ignores
that cost. On 99M AgentLogs rows 74% of the position windows read by an
unfiltered phrase query
were zstd, and decoding them took 58% of the index execution time of the
query.
PrxWindowLimits gets zstd_min_saving_percent, set from the new mutable
config
snii_prx_zstd_min_saving_percent. A readable PFOR frame is replaced by
the zstd one only if zstd is
smaller by at least that percentage of the PFOR frame. The default is
30; 0 picks the smaller frame
as before and writes a byte-identical index (99M rows: 61,886,854,661
bytes with and without this
change). Readers handle every codec, so the setting affects only
segments written afterwards, and
compaction rewrites with whatever is set then. The compression is still
computed, so the write cost
does not change.
Effect of the setting, 10M rows (index size, DS7 = unfiltered phrase on
the output column, position
decode time from the query profile): 10 -> +0.02% size, no change for
DS7; 20 -> +0.5%, no change;
30 -> +2.1%, decode 252 ms -> 85 ms; 50 -> +6.3%, no further gain; 100
(never zstd) -> +8.5%.
30 is the smallest value that removes the zstd windows of frequent
terms, which zstd beats PFOR by
20-30% on and which phrase queries read most. On 99M rows with 30: index
+1.9% (63.07 GB against
61.89 GB, still below the 65.70 GB of V3), load time within 2% of a
control load, hot queries in the
same process 0.63x (unfiltered phrase), 0.67x (phrase in input or
output), 0.75x (phrase prefix),
0.77x, 0.87x, the others within 1.02x (one query measured 1.06x once and
1.00-1.05x in three
repeats); cold runs 10-18% faster for the phrase queries. The best value
depends on the data, since
it is how much zstd beats PFOR on the window contents; it was calibrated
on one corpus.
---
be/src/common/config.cpp | 6 +++
be/src/common/config.h | 3 ++
be/src/storage/index/index_file_writer.cpp | 2 +
.../storage/index/inverted/snii/format/prx_pod.cpp | 31 +++++++++----
.../storage/index/inverted/snii/format/prx_pod.h | 11 ++++-
.../inverted/snii/writer/posting_prx_encoder.cpp | 11 +++--
.../index/inverted/snii/format/prx_pod_test.cpp | 35 ++++++++++++++
.../snii/writer/snii_writer_golden_bytes_test.cpp | 54 ++++++++++++++++++++++
8 files changed, 137 insertions(+), 16 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 2666d88faf5..b8cf6e3273e 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -1426,6 +1426,12 @@ DEFINE_mInt32(inverted_index_max_buffered_docs, "-1");
// Zstd levels for SNII DICT blocks and .prx windows. These affect writes;
each segment records its compression mode.
DEFINE_mInt32(snii_dict_block_zstd_level, "3");
DEFINE_mInt32(snii_prx_zstd_level, "3");
+// A .prx window is stored zstd-compressed only if that frame is smaller than
the PFOR one by at
+// least this percentage; 0 picks the smaller frame. Every read of a zstd
window pays a
+// decompression that PFOR does not, and on text positions zstd beats PFOR by
20-30% only on the
+// large windows of frequent terms, which are the ones phrase queries read
most: 30 costs about 2%
+// of the index size. Readers handle every codec, so changing this never
affects existing segments.
+DEFINE_mInt32(snii_prx_zstd_min_saving_percent, "30");
// Zstd level for direct-load .prx regions. Compaction rewrites them using
snii_prx_zstd_level.
DEFINE_mInt32(snii_prx_zstd_level_direct_load, "3");
// G16-d: target SNII dict block size in bytes; 0 uses the format default
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 6afc6e34eee..632cd8fd404 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -1466,6 +1466,9 @@ DECLARE_mInt32(inverted_index_max_buffered_docs);
// import CPU; see the DEFINEs in config.cpp).
DECLARE_mInt32(snii_dict_block_zstd_level);
DECLARE_mInt32(snii_prx_zstd_level);
+// A .prx window is stored zstd-compressed only if that frame is smaller than
the PFOR one by at
+// least this percentage; 0 picks the smaller frame. Affects segments written
from now on.
+DECLARE_mInt32(snii_prx_zstd_min_saving_percent);
// Patch C: prx zstd level for DIRECT-LOAD segments only (default 3, cheaper
// import); compaction rewrites at snii_prx_zstd_level so settled segments are
// unaffected. Full contract at the DEFINE in config.cpp.
diff --git a/be/src/storage/index/index_file_writer.cpp
b/be/src/storage/index/index_file_writer.cpp
index 30b81302f62..dd3f86fe1b2 100644
--- a/be/src/storage/index/index_file_writer.cpp
+++ b/be/src/storage/index/index_file_writer.cpp
@@ -146,6 +146,8 @@ void snii_resolve_index_write_params(bool is_direct_load,
bool has_norms,
input->prx_zstd_level = std::clamp(
is_direct_load ? config::snii_prx_zstd_level_direct_load :
config::snii_prx_zstd_level,
3, 19);
+ input->prx_window_limits.zstd_min_saving_percent =
+
static_cast<uint32_t>(std::clamp(config::snii_prx_zstd_min_saving_percent, 0,
100));
// G16-d: dict block size experiment knob; <= 0 keeps the format default.
if (config::snii_target_dict_block_bytes > 0) {
input->target_dict_block_bytes =
diff --git a/be/src/storage/index/inverted/snii/format/prx_pod.cpp
b/be/src/storage/index/inverted/snii/format/prx_pod.cpp
index 5a8328d6b1c..b7c22771b9b 100644
--- a/be/src/storage/index/inverted/snii/format/prx_pod.cpp
+++ b/be/src/storage/index/inverted/snii/format/prx_pod.cpp
@@ -60,6 +60,10 @@ Status validate_prx_window_limits(const PrxWindowLimits&
limits) {
return Status::Error<ErrorCode::INVALID_ARGUMENT, false>(
"prx: writer window limits exceed reader limits");
}
+ if (limits.zstd_min_saving_percent > 100) {
+ return Status::Error<ErrorCode::INVALID_ARGUMENT, false>(
+ "prx: zstd minimum saving is a percentage");
+ }
return Status::OK();
}
@@ -451,7 +455,8 @@ size_t zstd_frame_size(size_t plain_size, size_t
compressed_size) {
// before PFOR materialization in build_prx_window_auto_from_flat.
AutoPrxCodecChoice select_auto_prx_codec(size_t pfor_payload_size, size_t
plain_payload_size,
size_t compressed_payload_size, bool
has_compressed,
- uint32_t max_uncomp_bytes) {
+ uint32_t max_uncomp_bytes,
+ uint32_t zstd_min_saving_percent) {
const bool pfor_readable = pfor_payload_size <= max_uncomp_bytes;
const bool plain_readable = plain_payload_size <= max_uncomp_bytes;
AutoPrxCodecChoice choice;
@@ -462,7 +467,14 @@ AutoPrxCodecChoice select_auto_prx_codec(size_t
pfor_payload_size, size_t plain_
}
if (has_compressed && plain_readable) {
const size_t frame_size = zstd_frame_size(plain_payload_size,
compressed_payload_size);
- if (!choice.readable || frame_size < selected_frame_size) {
+ // Against a PFOR frame ZSTD must save the configured share; without
one it only has to win.
+ const bool saves_enough =
+ !choice.readable ||
+ (frame_size < selected_frame_size &&
+ static_cast<uint64_t>(frame_size) * 100 <=
+ static_cast<uint64_t>(selected_frame_size) *
+ (100 -
std::min<uint32_t>(zstd_min_saving_percent, 100)));
+ if (saves_enough) {
choice = {.codec = PrxCodec::kZstd, .readable = true};
selected_frame_size = frame_size;
}
@@ -543,9 +555,9 @@ Status build_prx_window_auto_from_flat(std::span<const
uint32_t> positions_flat,
testing::note_prx_raw_build();
RETURN_IF_ERROR(zstd_compress(plain.view(), zstd_level,
&compressed));
}
- const AutoPrxCodecChoice choice =
- select_auto_prx_codec(payload.size(), plain.size(),
compressed.size(),
- has_compressed, limits.max_uncomp_bytes);
+ const AutoPrxCodecChoice choice = select_auto_prx_codec(
+ payload.size(), plain.size(), compressed.size(),
has_compressed,
+ limits.max_uncomp_bytes, limits.zstd_min_saving_percent);
DCHECK(choice.readable);
if (choice.codec == PrxCodec::kZstd) {
write_zstd_compressed(plain.view(), Slice(compressed), sink);
@@ -1405,10 +1417,11 @@ void note_prx_delta_materialization() {
}
uint8_t select_auto_prx_codec_for_test(size_t pfor_payload_size, size_t
plain_payload_size,
- size_t compressed_payload_size,
uint32_t max_uncomp_bytes) {
- const AutoPrxCodecChoice choice =
- select_auto_prx_codec(pfor_payload_size, plain_payload_size,
compressed_payload_size,
- plain_payload_size >= kPrxAutoZstdMinBytes,
max_uncomp_bytes);
+ size_t compressed_payload_size,
uint32_t max_uncomp_bytes,
+ uint32_t zstd_min_saving_percent) {
+ const AutoPrxCodecChoice choice = select_auto_prx_codec(
+ pfor_payload_size, plain_payload_size, compressed_payload_size,
+ plain_payload_size >= kPrxAutoZstdMinBytes, max_uncomp_bytes,
zstd_min_saving_percent);
DCHECK(choice.readable);
return static_cast<uint8_t>(choice.codec);
}
diff --git a/be/src/storage/index/inverted/snii/format/prx_pod.h
b/be/src/storage/index/inverted/snii/format/prx_pod.h
index 6000ce14487..22ca2f8287f 100644
--- a/be/src/storage/index/inverted/snii/format/prx_pod.h
+++ b/be/src/storage/index/inverted/snii/format/prx_pod.h
@@ -65,14 +65,20 @@ struct AutoPrxCodecChoice {
// Common policy for resident and replayable encoders: compare complete frames,
// preserve PFOR/ZSTD/RAW priority on ties, and honor the reader byte limit.
+// A readable PFOR frame is replaced by the ZSTD one only if ZSTD is smaller
by at least
+// `zstd_min_saving_percent` of the PFOR frame: every read of a ZSTD window
pays a decompression,
+// which PFOR does not. 0 picks the smallest frame.
AutoPrxCodecChoice select_auto_prx_codec(size_t pfor_payload_size, size_t
plain_payload_size,
size_t compressed_payload_size, bool
has_compressed,
- uint32_t max_uncomp_bytes);
+ uint32_t max_uncomp_bytes,
+ uint32_t zstd_min_saving_percent = 0);
struct PrxWindowLimits {
uint32_t max_docs;
uint32_t max_positions;
uint32_t max_uncomp_bytes;
+ // Writer-only choice, ignored by readers: see select_auto_prx_codec.
+ uint32_t zstd_min_saving_percent = 0;
};
enum class PrxWindowBuildOutcome : uint8_t {
@@ -197,7 +203,8 @@ uint64_t prx_delta_materialization_count();
void reset_prx_delta_materialization_count();
void note_prx_delta_materialization();
uint8_t select_auto_prx_codec_for_test(size_t pfor_payload_size, size_t
plain_payload_size,
- size_t compressed_payload_size,
uint32_t max_uncomp_bytes);
+ size_t compressed_payload_size,
uint32_t max_uncomp_bytes,
+ uint32_t zstd_min_saving_percent = 0);
#endif
} // namespace doris::snii::format::testing
diff --git a/be/src/storage/index/inverted/snii/writer/posting_prx_encoder.cpp
b/be/src/storage/index/inverted/snii/writer/posting_prx_encoder.cpp
index 649fa3f984b..36ebed0338f 100644
--- a/be/src/storage/index/inverted/snii/writer/posting_prx_encoder.cpp
+++ b/be/src/storage/index/inverted/snii/writer/posting_prx_encoder.cpp
@@ -176,7 +176,8 @@ Status encode_position_payloads(const PostingPositionView&
positions,
Status choose_position_codec(PostingByteBuffer* plain, PostingByteBuffer* pfor,
PostingByteBuffer* compressed, int level, bool
singleton,
- uint32_t byte_limit, format::PrxCodec* codec) {
+ uint32_t byte_limit, uint32_t
zstd_min_saving_percent,
+ format::PrxCodec* codec) {
if (level >= 0) {
*codec = format::PrxCodec::kRaw;
if (level > 0) {
@@ -194,7 +195,7 @@ Status choose_position_codec(PostingByteBuffer* plain,
PostingByteBuffer* pfor,
RETURN_IF_ERROR(compress_posting_bytes(plain, -level, compressed));
}
*codec = format::select_auto_prx_codec(pfor->size(), plain->size(),
compressed->size(),
- use_zstd, byte_limit)
+ use_zstd, byte_limit,
zstd_min_saving_percent)
.codec;
return Status::OK();
}
@@ -450,9 +451,9 @@ Status PostingPrxEncoder::build_replayable(const
PostingPositionView& positions,
return Status::OK();
}
format::PrxCodec codec = format::PrxCodec::kRaw;
- RETURN_IF_ERROR(choose_position_codec(&plain, &pfor, &compressed, level,
- freqs.size() == 1 && freqs.front()
== 1,
- limits.max_uncomp_bytes, &codec));
+ RETURN_IF_ERROR(choose_position_codec(
+ &plain, &pfor, &compressed, level, freqs.size() == 1 &&
freqs.front() == 1,
+ limits.max_uncomp_bytes, limits.zstd_min_saving_percent, &codec));
ByteSink header;
header.put_u8(static_cast<uint8_t>(codec));
header.put_varint32(
diff --git a/be/test/storage/index/inverted/snii/format/prx_pod_test.cpp
b/be/test/storage/index/inverted/snii/format/prx_pod_test.cpp
index d926f9b7f90..dded71d4455 100644
--- a/be/test/storage/index/inverted/snii/format/prx_pod_test.cpp
+++ b/be/test/storage/index/inverted/snii/format/prx_pod_test.cpp
@@ -1453,6 +1453,41 @@ TEST(SniiPrxPodTest,
MaterializedRawCandidateMustStrictlyReduceFrame) {
static_cast<uint8_t>(PrxCodec::kPfor));
}
+// ZSTD replaces a readable PFOR frame only if it saves the configured share
of the PFOR frame.
+// PFOR frame 1007 bytes, ZSTD frame 859 bytes: it saves 14.7%.
+TEST(SniiPrxPodTest, ZstdMustSaveTheConfiguredShareOverPfor) {
+ constexpr uint32_t kLimit = 4096;
+ auto select = [&](uint32_t min_saving_percent) {
+ return doris::snii::format::testing::select_auto_prx_codec_for_test(
+ /*pfor_payload_size=*/1000, /*plain_payload_size=*/2000,
+ /*compressed_payload_size=*/850, kLimit, min_saving_percent);
+ };
+ EXPECT_EQ(select(0), static_cast<uint8_t>(PrxCodec::kZstd));
+ EXPECT_EQ(select(10), static_cast<uint8_t>(PrxCodec::kZstd));
+ EXPECT_EQ(select(14), static_cast<uint8_t>(PrxCodec::kZstd));
+ EXPECT_EQ(select(15), static_cast<uint8_t>(PrxCodec::kPfor));
+ EXPECT_EQ(select(20), static_cast<uint8_t>(PrxCodec::kPfor));
+ EXPECT_EQ(select(100), static_cast<uint8_t>(PrxCodec::kPfor));
+}
+
+// The share only protects PFOR: when the PFOR frame exceeds the reader byte
limit, ZSTD is the
+// only readable compressed form and is chosen whatever the share.
+TEST(SniiPrxPodTest, ZstdShareDoesNotApplyWhenPforIsUnreadable) {
+ constexpr uint32_t kLimit = 1500;
+ EXPECT_EQ(doris::snii::format::testing::select_auto_prx_codec_for_test(
+ /*pfor_payload_size=*/1600, /*plain_payload_size=*/1400,
+ /*compressed_payload_size=*/500, kLimit,
/*zstd_min_saving_percent=*/100),
+ static_cast<uint8_t>(PrxCodec::kZstd));
+}
+
+TEST(SniiPrxPodTest, ZstdMinSavingAboveOneHundredPercentIsRejected) {
+ doris::snii::format::PrxWindowLimits limits =
doris::snii::format::kReaderPrxWindowLimits;
+ limits.zstd_min_saving_percent = 100;
+ EXPECT_TRUE(doris::snii::format::validate_prx_window_limits(limits).ok());
+ limits.zstd_min_saving_percent = 101;
+ EXPECT_FALSE(doris::snii::format::validate_prx_window_limits(limits).ok());
+}
+
// [functional] F-RT-small: a small flat window round-trips through the CSR
reader
// (codec=pfor path) back to the original per-doc positions.
TEST(SniiPrxPodTest, SmallFlatWindowRoundTripsViaCsr) {
diff --git
a/be/test/storage/index/inverted/snii/writer/snii_writer_golden_bytes_test.cpp
b/be/test/storage/index/inverted/snii/writer/snii_writer_golden_bytes_test.cpp
index 3f221d648b0..e9a74d04f3a 100644
---
a/be/test/storage/index/inverted/snii/writer/snii_writer_golden_bytes_test.cpp
+++
b/be/test/storage/index/inverted/snii/writer/snii_writer_golden_bytes_test.cpp
@@ -253,6 +253,25 @@ uint64_t golden_digest(const std::string& name, const
TabletIndex& meta) {
return fnv1a64(bytes);
}
+// Writes `rows` through the production stack and returns the size of the
segment file.
+size_t written_size(const std::string& name, const TabletIndex& meta,
+ const std::vector<std::string>& rows) {
+ const std::string path = std::string(kTestDir) + "/" + name + ".idx";
+ io::FileWriterPtr file_writer;
+ EXPECT_TRUE(io::global_local_filesystem()->create_file(path,
&file_writer).ok());
+ IndexFileWriter index_file_writer(io::global_local_filesystem(), path,
"size_rowset",
+ /*seg_id=*/0,
InvertedIndexStorageFormatPB::SNII,
+ std::move(file_writer),
/*can_use_ram_dir=*/true,
+ /*tablet_id=*/901);
+ SniiIndexColumnWriter writer(&index_file_writer, &meta,
FieldType::OLAP_FIELD_TYPE_VARCHAR);
+ EXPECT_TRUE(writer.init().ok());
+ add_batch(&writer, rows);
+ EXPECT_TRUE(writer.finish().ok());
+ EXPECT_TRUE(index_file_writer.begin_close().ok());
+ EXPECT_TRUE(index_file_writer.finish_close().ok());
+ return read_file_bytes(path).size();
+}
+
class SniiWriterGoldenBytes : public testing::Test {
protected:
void SetUp() override {
@@ -261,17 +280,20 @@ protected:
_saved_dict_lvl = config::snii_dict_block_zstd_level;
_saved_prx_lvl = config::snii_prx_zstd_level;
_saved_prx_load_lvl = config::snii_prx_zstd_level_direct_load;
+ _saved_prx_saving = config::snii_prx_zstd_min_saving_percent;
assert_ok(io::global_local_filesystem()->delete_directory(kTestDir));
assert_ok(io::global_local_filesystem()->create_directory(kTestDir));
config::snii_dict_block_zstd_level = 3;
config::snii_prx_zstd_level = 3;
config::snii_prx_zstd_level_direct_load = 3;
+ config::snii_prx_zstd_min_saving_percent = 0;
}
void TearDown() override {
config::snii_dict_block_zstd_level = _saved_dict_lvl;
config::snii_prx_zstd_level = _saved_prx_lvl;
config::snii_prx_zstd_level_direct_load = _saved_prx_load_lvl;
+ config::snii_prx_zstd_min_saving_percent = _saved_prx_saving;
EXPECT_TRUE(io::global_local_filesystem()->delete_directory(kTestDir).ok());
}
@@ -279,6 +301,7 @@ private:
int32_t _saved_dict_lvl = 3;
int32_t _saved_prx_lvl = 3;
int32_t _saved_prx_load_lvl = 3;
+ int32_t _saved_prx_saving = 0;
};
// Whole-image digests re-harvested after core metadata stopped writing an
empty
@@ -295,6 +318,37 @@ constexpr uint64_t kGoldenEnglishPhrase =
0x8157f9dd81c5397cULL;
constexpr uint64_t kGoldenUnicodePhrase = 0x9e2a4ba4abedaef9ULL;
constexpr uint64_t kGoldenKeywordDocsOnly = 0x27cec04c70573201ULL;
+// The default of the knob is part of the written format's behavior: the
fixture above pins it to 0
+// for the digests, so this reads the value a fresh process starts with.
+TEST(SniiWriterPrxZstdSaving, DefaultIsThirtyPercent) {
+ EXPECT_EQ(config::snii_prx_zstd_min_saving_percent, 30);
+}
+
+// Every document repeats the same positions, so a window is far smaller as
zstd than as PFOR.
+// 0 and 30 keep zstd; 100 never picks it, so the segment is larger.
+TEST_F(SniiWriterGoldenBytes, PrxZstdMinSavingConfigReachesTheWriter) {
+ std::vector<std::string> rows;
+ for (int doc = 0; doc < 2000; ++doc) {
+ std::string row;
+ for (int token = 0; token < 31; ++token) {
+ row += (token == 1 || token == 5 || token == 6 || token == 17 ||
token == 30)
+ ? "x "
+ : "t" + std::to_string(token) + " ";
+ }
+ rows.push_back(row);
+ }
+ const TabletIndex meta =
+ make_meta({{"parser", "unicode"}, {"lower_case", "true"},
{"support_phrase", "true"}});
+ config::snii_prx_zstd_min_saving_percent = 0;
+ const size_t smallest = written_size("saving_0", meta, rows);
+ config::snii_prx_zstd_min_saving_percent = 30;
+ const size_t thirty = written_size("saving_30", meta, rows);
+ config::snii_prx_zstd_min_saving_percent = 100;
+ const size_t never_zstd = written_size("saving_100", meta, rows);
+ EXPECT_EQ(thirty, smallest);
+ EXPECT_GT(never_zstd, smallest + 1000) << "smallest=" << smallest;
+}
+
TEST_F(SniiWriterGoldenBytes, EnglishPhrase) {
const TabletIndex meta =
make_meta({{"parser", "english"}, {"lower_case", "true"},
{"support_phrase", "true"}});
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
