adriangb opened a new pull request, #23795: URL: https://github.com/apache/datafusion/pull/23795
# Which issue does this PR close? None yet — **draft companion** to the arrow-rs bounded streaming RFC: apache/arrow-rs#10410. This PR will not compile in CI until that lands and is released (it was developed and benchmarked against a `[patch.crates-io]` of that branch, not included here). # Rationale for this change The parquet push-decoder fetch loop materializes every projected byte of a row group before decoding. For wide-value columns (e.g. 100 MB single-row-group files whose projected blob column is ~97 MB), peak reader memory ≈ row group size × concurrent partitions, and decode start is gated on the slowest range completing. arrow-rs#10410 adds an opt-in bounded streaming mode; this PR wires it through DataFusion. # What changes are included in this PR? - New config option `datafusion.execution.parquet.bounded_streaming` (default `false`). - `DecoderBuilderConfig` applies `with_bounded_streaming` to the push decoder builder when enabled. - `PushDecoderStreamState` gains a `FetchState`: the plain `get_byte_ranges` loop as today, or the parquet `AdaptiveFetcher` (small coalesced ranges materialized exactly as before; large column chunks consumed incrementally window by window). Falls back to the plain path when no tokio runtime is available. - `ParquetFileReader` / `CachedParquetFileReader` forward `get_bytes_stream` to `ParquetObjectReader` (without this the fallback silently materializes). # Benchmark summary (harness changes in the companion benchmarks PR) On the motivating shape (8 × 94 MB single-row-group files, 92 MB blob column, full-scan checksum): identical results and request counts (8 GETs both), wall neutral (29.9 → 29.7 ms median SSD), peak RSS 1261 → 702 MB (−44%). TPC-H SF1 / TPC-DS SF1 / stock ClickBench: neutral within noise, zero request inflation (streaming does not engage below the 16 MiB threshold or without an offset index). Known issue inherited from the arrow-rs draft: ~+14% wall on ClickBench-with-page-indexes string queries from per-window dictionary re-decode (fix identified upstream: cache decoded dictionaries across windows). # Are these changes tested? `datafusion-datasource-parquet` suite green; correctness validated off-vs-on (identical results incl. checksums) on the benchmark datasets. sqllogictest `information_schema` and config docs updated. # Are there any user-facing changes? New experimental config option, default off. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MGnT9oETcFMydc9cp95HLG -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
