Pastoray opened a new pull request, #27514: URL: https://github.com/apache/datafusion/pull/27514
## Which issue does this PR close? - Closes #27513. ## Rationale for this change A TopK query (`ORDER BY v LIMIT n`) normally skips Parquet row groups that the current TopK threshold rules out (`row_groups_pruned_dynamic_filter`). That stops working when the file stores the sort column as `REQUIRED` but the table schema declares it nullable, which is common when a table is given an explicit all-nullable schema so files with and without a column read as one table. Results stay correct, but every row group in the file gets decoded. With the repro from the issue (1M rows, 16 row groups, `LIMIT 100`): | | row groups pruned | bytes scanned | | ------------------------ | ----------------- | ------------- | | table schema = file schema | 15 | 97 KB | | nullable table schema, before | 0 | 1.5 MB | | nullable table schema, after | 15 | 97 KB | ## What changes are included in this PR? When the table field is nullable and the file field is not, the schema adapter wraps the column in a same-type `CAST` to the nullable field. The TopK filter then reaches the statistics pruner as `CAST(v) IS NULL OR CAST(v) < t`. `build_is_null_column_expr` only accepted a bare column, so the `IS NULL` branch was treated as unhandled and became `true`, which makes the whole `OR` unprunable. With matching schemas, or an `OPTIONAL` file column, there is no cast, which is why those cases already worked. `build_is_null_column_expr` in `datafusion/pruning/src/pruning_predicate.rs` now looks through a cast that is lossless and preserves nulls, using the existing `CastExpr::is_bigger_cast`, and builds the null-count expression for the column underneath. A cast that can introduce nulls (for example `Utf8` to `Int64`) is still left unhandled, so pruning stays conservative for it. ## What is the testing strategy for this PR? Three unit tests are added next to the existing `IS NULL` pruning tests in `pruning_predicate.rs`: - `prune_is_null_through_nullability_only_cast`: `IS NULL` and `IS NOT NULL` over the nullability-only cast. - `prune_topk_filter_through_nullability_only_cast`: the TopK shape `v IS NULL OR v < t` from the issue. - `prune_is_null_through_null_introducing_cast_is_not_pruned`: a cast that can introduce nulls must not be pruned. The first two fail without the change (no containers are pruned) and pass with it. I also ran the repro from the issue against the change, and the nullable-schema case now prunes 15 row groups. Existing tests in `datafusion-pruning`, `datafusion-datasource-parquet` and the `parquet`/`physical_optimizer` tests in `core_integration` pass. ## Are there any user-facing changes? No API changes. Queries that were already correct now skip row groups they previously decoded. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
