Seanium opened a new pull request, #25987: URL: https://github.com/apache/datafusion/pull/25987
## Which issue does this PR close? Part of #25484. Depends on the Arrow `StatisticsConverter` INT96 support in https://github.com/apache/arrow-rs/pull/11348 and a compatible parquet release; this draft is not ready to merge until that dependency is available. ## Rationale for this change Files carrying explicit INT96 timestamp ordering can safely support timestamp pruning, but DataFusion currently rejects their bounds. ## What changes are included in this PR? - Trust INT96 bounds only when the footer explicitly contains `INT96_TIMESTAMP_ORDER`; missing, legacy, and unknown orders remain untrusted. - Use the reader's configured INT96 unit and timezone for file statistics. A table-schema cast alone does not authorize direct decoding at that resolution. - Preserve conservative fallback and null counts when bounds are unavailable or incompatible. ## What is the testing strategy for this PR? `statistics_order_tests` writes and reads real INT96 files, then checks file, row-group, page, and runtime pruning across all timestamp units and timezones. It also verifies legacy footer behavior and overflowing bounds. `int96_format_statistics_use_configured_resolution` covers both production statistics-inference paths and distinguishes configured decoding from a table-schema cast. The integration has been validated locally with the Arrow prerequisite applied to parquet 60.0.0, using released Arrow 60 crates. The dependency override is not included in this PR. ## Are there any user-facing changes? Newly written files with explicit INT96 timestamp ordering can be pruned. Existing Spark/Hive files without this metadata do not gain pruning retroactively. Public APIs remain compatible. ## AI assistance Implementation, tests, and review were AI-assisted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
