hareshkh opened a new pull request, #27308: URL: https://github.com/apache/datafusion/pull/27308
## Which issue does this PR close? - Part of #23805. - Backport of #26310 to `branch-54`, as discussed in https://github.com/apache/datafusion/issues/23805#issuecomment-6739184093. - Related to #18083. ## Rationale for this change Hive-written tables containing a `__HIVE_DEFAULT_PARTITION__` directory cannot be read when the partition column is a date or another non-string type. String partition columns return the marker literally, producing incorrect results for `IS NULL` and other predicates. This backports the merged fix from #26310 so the correction is available in the 54 release line. ## What changes are included in this PR? Two commits separate the #26310 cherry-pick (with conflict resolutions) from the branch-54 adaptations and required safeguards. - Backport `407bef62ca965815af27624564937aae50c3578a` by Xander: interpret the configurable Hive marker as a typed NULL, make partition fields nullable, fix NULL predicate handling and per-file partition statistics, and reject writes of the reserved marker as a real partition value. - Include the NULL-aware file-ordering bounds and regression tests from #26280 (`04c02c58085e7928393711721d4c7d0569740f8e`). This prerequisite is needed to preserve ordering for the newly supported NULL partitions. - Include #24951 (`99a6af80e395dd77b33d17338155a3f96a802fae`), the upstream guard against silently writing NULL partition keys as non-NULL values. It covers plain and dictionary arrays and prevents corrupting the newly readable Hive NULL values when copying them. - Adapt the changes to branch-54's partition parser and `TableSchema` constructor. The ordering tests retain this branch's conservative fallback for empty files with absent bounds. - Document the behavior and migration options in the 54.2.0 upgrade guide and configuration reference. ## Are these changes tested? Validation of the combined two-commit series with Rust 1.95.0: - `cargo fmt --all` and `cargo clippy --all-targets --all-features -- -D warnings` (no warnings). - `cargo test --profile ci -p datafusion-datasource -p datafusion-catalog-listing --lib`: 180 tests. - `core_integration` with the `catalog_listing` and `sql::path_partition` filters: 16 tests. - `sqllogictests` with `listing_table_partitions`, `insert_to_external`, `information_schema`, and `copy`: seven files, including a regression that rejects copying a Hive NULL partition with a NULL key. - `./ci/scripts/doc_prettier_check.sh --write --allow-dirty`; regenerated the configuration table with `print_config_docs`. - Ablation checks: the typed-NULL parsing test fails without the marker mapping, and the file-ordering regression fails without the NULL-aware bounds. Restoring the fixes passes the full 180-test unit suite. - `split_groups_by_statistics` benchmark smoke checks (`--profile ci`, `--test`): all 60 cases pass on the backport and local `main` (`941146372a`). These runs check benchmark correctness only. ## Are there any user-facing changes? Yes: - Hive NULL marker directories produce NULL partition values; all `ListingTable` partition fields become nullable. - Adds `datafusion.execution.listing_table_hive_default_partition_name` (default `__HIVE_DEFAULT_PARTITION__`) and the public `HIVE_DEFAULT_PARTITION_NAME` constant. The new `ExecutionOptions` field affects callers using exhaustive struct literals. - Partitioned writes reject the configured marker as a literal value. Setting the option to `''` restores literal reads and writes; it does not restore non-nullable partition schemas. - Partitioned writes containing actual NULL keys now fail instead of silently changing their values. Checks happen per batch, so files from earlier batches may remain after an error. - NULL-aware bounds prevent scans from advertising an invalid order across files. AI assistance: Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
