peterxcli opened a new pull request, #5772: URL: https://github.com/apache/datafusion-comet/pull/5772
## Which issue does this PR close? Related to #5739 and #5553. This draft demonstrates nested-field statistics pruning, but does not close either issue: checked `TIMESTAMP_MILLIS` conversion and Spark filtering parity still need separate work. ## Rationale for this change Comet already serializes nested predicates, but its custom `GetStructField` expression does not expose DataFusion's field-access capability. A selective `s.inner.k` filter therefore reads all row groups even when leaf statistics can exclude most of them. This draft connects Comet to [peterxcli/datafusion#2](https://github.com/peterxcli/datafusion/pull/2) to demonstrate the performance and correctness of the integration. It pins that DataFusion fork and needs the upstream dependency work before it is ready to merge. On a synthetic local Parquet file with 4,194,304 rows and 64 row groups, the selective nested query improves from **642.3 ms to 182.6 ms (3.52x)**. It prunes **63/64 row groups** and reduces reader bytes from **182,481,422 to 2,851,301 (98.4% less)**, with identical aggregate checksums. | Query | Baseline median | Candidate median | Baseline / candidate | | --- | ---: | ---: | ---: | | Nested filter, sorted keys | 642.3 ms | 182.6 ms | 3.52x | | Top-level filter, sorted keys | 138.8 ms | 138.3 ms | 1.00x | | Projection only | 530.2 ms | 537.4 ms | 0.99x | | Nested filter, shuffled keys | 585.4 ms | 577.1 ms | 1.01x | Disabling pruning in the candidate restores all 64 groups and baseline reader bytes, with a 638.0 ms median over five control measurements. The other matched comparisons differ by less than 2%, with overlapping interquartile ranges. These are warm-cache measurements on an AMD EPYC 7282 host with Spark 4.1.3, `local[1]`, matching release build settings, and 20 samples per revision in alternating run order. Timings include SQL planning, execution, and collecting the result. Every measured plan uses `CometNativeScan`. This does not establish production or object-storage speedups, or improvements for arrays, maps, Variant, or unfiltered scans. ## What changes are included in this PR? - Pin DataFusion packages to fork revision `0b0506a9acab9d5892ecf7e89243c3b34664bcc6`. - Expose struct field access through a scalar UDF while preserving Comet's parent-null handling. Resolve fields by name after schema adaptation and retain ordinal access for duplicate names. - Route native `GetStructField` planning through the adapter and add a real scan regression for nested statistics pruning. - Include the benchmark harness, raw timings, plans, metrics, checksums, hashes, and reproduction instructions in [`benchmarks/nested-pruning`](https://github.com/peterxcli/datafusion-comet/tree/perf/nested-struct-statistics-pruning/benchmarks/nested-pruning). - Correct comments about the remaining nested timestamp overflow limitation; conversion behavior is unchanged. ## How are these changes tested? - Four Rust expression tests passed, including parent nulls, scalar structs, reordered fields, and duplicate field names. - Five Spark tests passed, covering nested pruning enabled/disabled, nested schema evolution, field-ID mapping, and nullable struct access; one pre-existing test was ignored. - Native release builds and Maven packaging passed. Rust formatting and Spark style/format checks passed. - The benchmark checks aggregate results and pruning counters, including candidate pruning-disabled and decoder-filter-enabled controls. The latter are correctness controls, not a matched decoder-filter performance comparison. Build, test, and benchmark results were collected against Comet `bb9e74020adc228e486f6f4d0fa68292b30bff31` and the patch on that base. The baseline uses its checked-in DataFusion 55.0.0 dependencies, so the comparison includes both the DataFusion fork and the Comet adapter. Before publication, the patch was rebased unchanged onto `5af898b2e` (the intervening Iceberg unknown-transform fix); `git range-diff` confirms an identical patch. Formatting and diff checks were repeated after the rebase; builds and benchmark timings were not rerun. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
