Rich-T-kid opened a new pull request, #27500:
URL: https://github.com/apache/datafusion/pull/27500

   ## Which issue does this PR close?
   
   - Closes #24113.
   - Part of #24111.
   
   ## Rationale for this change
   
   `enable_rle_to_dictionary` is an unbounded opt-in: if a top-level string or 
binary Parquet column has a dictionary page, DataFusion can infer and scan it 
as a dictionary.
   
   NDV alone is not a safe promotion signal because a column chunk can start 
with dictionary-encoded data pages and later fall back to plain encoding. That 
transition can force the Arrow reader to expand and rebuild dictionaries. 
Parquet page-encoding statistics let us exclude those mixed chunks, after which 
an NDV cap can provide the bounded user control requested in #24113.
   
   ## What changes are included in this PR?
   
   This adds 
`datafusion.execution.parquet.dictionary_read_max_distinct_values`. Setting the 
cap enables guarded dictionary reads without also setting 
`enable_rle_to_dictionary`.
   
   A top-level string or binary column is eligible only when every row group 
has NDV statistics, every data page is dictionary encoded, and the sum of 
row-group NDVs (a conservative per-file upper bound) is at most the configured 
cap. Missing statistics and mixed page encodings fail closed.
   
   Schema inference keeps a column as a dictionary only when every file 
containing it is eligible. The file opener rechecks each actual footer, so 
files added after schema inference and files under additional listing paths 
decode as plain values when they are ineligible, then use normal schema 
adaptation if the table already published a dictionary type.
   
   The option is wired through protobuf serialization, generated configuration 
documentation, and the 57.0 upgrade guide. `None` preserves the existing 
behavior.
   
   ## What is the testing strategy for this PR?
   
   Focused unit and SQL logic tests cover cap-only activation, missing NDV, 
exact and above-cap values, zero, multiple row groups, mixed files, 
dictionary-to-plain fallback, embedded Arrow dictionary schemas, per-file 
opener rechecks, legacy behavior, and protobuf round trips.
   
   Local benchmarks used SF1 Parquet data, 12 partitions, an 8,192-row batch 
size, a 32 GiB memory limit, and two-pair ABBA ordering. They ran on the 
pre-rebase candidate (`c23c7106c`), whose patch is identical to this PR head. 
Cap testing compared the same candidate binary with the cap disabled and set to 
5. A cap-only type preflight on the exact benchmark files inferred TPC-H 
`region.r_name` and TPC-DS `customer_demographics.cd_gender` as 
`Dictionary(Int32, Utf8)`, confirming that the treatment exercised dictionary 
reads. Default-off testing compared the candidate with its exact build base 
(`375bf976`) while the cap remained unset and the legacy boolean remained 
false. All 5,604 measured iterations succeeded with row-count parity.
   
   | Comparison | Coverage | Aggregate mean delta (pair 1 / pair 2; combined) | 
Confirmed regression above 5% |
   | --- | --- | --- | --- |
   | cap=5 vs disabled | TPC-H 1-22, 7 iterations | -2.85% / -2.67%; -2.76% | 
none |
   | cap=5 vs disabled | TPC-DS 1-99, 3 iterations plus focused 25-50 iteration 
reruns | -2.39% / +3.22%; +0.39% | none after focused confirmation |
   | candidate default-off vs exact base | 11 previously flagged queries, 
50-100 iterations | +0.61% / +7.17%; +3.90% | none in both independent pairs |
   
   The initial low-iteration TPC-DS Q1, Q22, and Q34 signals were rerun. Q22 
and Q34 cleared. Q1's first focused median signal did not reproduce in a fresh 
reverse-order 50-iteration matrix: its first cap leg was an isolated slow run, 
while the closing pair was +2.3% median and +1.2% mean. No query's min, median, 
or mean regression above 5% survived both independent confirmation pairs.
   
   Validation also includes:
   
   - `cargo test -p datafusion-datasource-parquet test_rle_`
   - `cargo test -p datafusion-datasource-parquet 
dictionary_promotion_requires_dictionary_only_data_pages`
   - `cargo test -p datafusion-proto 
dictionary_read_options_round_trip_through_codec`
   - `cargo test --profile=ci --test sqllogictests -- 
parquet_rle_to_dictionary.slt`
   - `cargo test --profile=ci --test sqllogictests -- information_schema.slt`
   - `cargo clippy --all-targets --all-features -- -D warnings`
   
   ## Are there any user-facing changes?
   
   Yes. `ParquetOptions` gains the public field 
`dictionary_read_max_distinct_values: Option<usize>`, and the generated 
protobuf `ParquetOptions` gains the optional 
`dictionary_read_max_distinct_values_opt` field.
   
   Downstream exhaustive struct literals must add the new field or use 
`..Default::default()`. The protobuf wire format remains backward compatible, 
and the default runtime behavior is unchanged.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to