snmvaughan opened a new pull request, #7264:
URL: https://github.com/apache/datafusion-comet/pull/7264
## Which issue does this PR close?
Backport of #7159 (squash commit 01fc37a80) to `branch-1.1`, for #7158.
## Rationale for this change
`branch-1.1` has the bug #7159 fixes on `main`. With
`spark.comet.parquet.rowFilterPushdown.enabled=true` on Spark 3.x, a
pushed-down filter that contains a cast Comet's planner inserted, such as the
Int64 positions of `substring(s, 1, 2)`, fails with `ParquetSchemaConvert {
physical_type: "INT32", spark_type: "bigint" }`. The native scan's expression
adapter treats every cast in the predicate as a Parquet schema conversion. See
#7158 for the details.
## What changes are included in this PR?
A cherry-pick of #7159 with `-x`, adapted in one test line: the substring
scan test builds its proto `ScalarFunc` without `null_short_circuit`, which
`branch-1.1`'s proto doesn't have. The fix itself is identical to the source
commit. `replace_with_spark_cast` now rewrites only the casts DataFusion's
schema adapter builds, which set the explicit-target flag that
`CastExpr::has_explicit_metadata()` returns, and leaves the planner's casts
alone.
Backports to other branches: `branch-1.2`, in a separate PR.
## How are these changes tested?
The tests come with the cherry-pick, in
`native/core/src/parquet/schema_adapter.rs`. On this branch all 96
`schema_adapter` tests pass, and `cargo fmt --check` and `cargo clippy
--all-targets --workspace -- -D warnings` are clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
