airborne12 opened a new pull request, #69795: URL: https://github.com/apache/doris/pull/69795
### What problem does this PR solve? Issue Number: None Related PR: #69649, #69769 Problem Summary: Backport of #69649 to branch-4.2; the backport of #69769, stacked on this one, builds on it. The auto .prx codec stores each position window in the smallest of PFOR, zstd and raw. Every read of a zstd window pays a decompression that PFOR does not, and the choice ignores that cost. On 99M AgentLogs rows 74% of the position windows read by an unfiltered phrase query were zstd, and decoding them took 58% of the index execution time of the query. PrxWindowLimits gets zstd_min_saving_percent, set from the new mutable config snii_prx_zstd_min_saving_percent. A readable PFOR frame is replaced by the zstd one only if zstd is smaller by at least that percentage of the PFOR frame. The default is 30; 0 picks the smaller frame as before and writes a byte-identical index (99M rows: 61,886,854,661 bytes with and without this change). Readers handle every codec, so the setting affects only segments written afterwards, and compaction rewrites with whatever is set then. The compression is still computed, so the write cost does not change. Effect of the setting, 10M rows (index size, DS7 = unfiltered phrase on the output column, position decode time from the query profile): 10 -> +0.02% size, no change for DS7; 20 -> +0.5%, no change; 30 -> +2.1%, decode 252 ms -> 85 ms; 50 -> +6.3%, no further gain; 100 (never zstd) -> +8.5%. 30 is the smallest value that removes the zstd windows of frequent terms, which zstd beats PFOR by 20-30% on and which phrase queries read most. On 99M rows with 30: index +1.9% (63.07 GB against 61.89 GB, still below the 65.70 GB of V3), load time within 2% of a control load, hot queries in the same process 0.63x (unfiltered phrase), 0.67x (phrase in input or output), 0.75x (phrase prefix), 0.77x, 0.87x, the others within 1.02x (one query measured 1.06x once and 1.00-1.05x in three repeats); cold runs 10-18% faster for the phrase queries. The best value depends on the data, since it is how much zstd beats PFOR on the window contents; it was calibrated on one corpus. ### Branch adaptations None. `git cherry-pick 428c24187c` applied cleanly to the current branch-4.2 (9f63be85ce) and touches the same eight files as the original. This PR on its own sets the default to 30, as #69649 did on master; the stacked backport of #69769 changes it to the per-index budget, as on master now. ### Validation - Built and run with both backports (this one and the stacked #69769 backport) applied to branch-4.2 at 711db43b00, the commit before #69625: its lance-c update is not in the local third-party build, and the 16 commits after it touch no SNII file. `PrxPod*`, `Snii*`, `DorisSniiFileReader*`, `*InvertedIndex*`, `*IndexFileWriter*` and `*IndexFileReader*`: 1710 tests passed, including the new budget, codec choice, writer and configuration tests. - On master both changes passed CI. - Not built locally on the branch-4.2 tip; this PR's BE UT run covers that. ### Release note SNII position windows are stored with zstd only where it saves at least 30% over PFOR (snii_prx_zstd_min_saving_percent): phrase queries are faster and the index is about 2% larger. Set it to 0 for the previous behavior. ### Check List (For Author) - Test: Unit Test on master, 1723 tests pass with the inline-read change stacked (SniiPrxPodTest: the saving boundary, PFOR unreadable, limit validation; SniiWriterGoldenBytes: the config reaches the writer; the default is 30). Manual test: index size and load time at 10M and 99M rows, interleaved hot A/B and repeated cold runs. - Behavior changed: Yes (newly written segments are about 2% larger; set snii_prx_zstd_min_saving_percent to 0 to restore) - Does this need documentation: No 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
