peterxcli opened a new pull request, #6594: URL: https://github.com/apache/datafusion-comet/pull/6594
## Which issue does this PR close? Part of #5429. ## Rationale for this change Evaluate Variant null and validity predicates natively over the whole-value scan output. This can proceed independently of #6443 with `spark.sql.variant.pushVariantIntoScan=false`. ## What changes are included in this PR? Add `is_variant_null` for Spark 4.x and `is_valid_variant` for Spark 4.2, preserving Spark's distinct SQL NULL and malformed-value rules. The `implement-comet-expression` workflow guided the implementation and compatibility tests. Malformed Parquet values still encounter scan validation before the predicate. A captured regression keeps that remaining work under #5429. ## How are these changes tested? All 18 selected native tests and 21 Spark 4.2 SQL/Variant tests passed. The known malformed-scan regression is ignored with #5429. Full workspace Clippy, formatting, and Scalastyle passed. Broader CI is pending. Please apply `run-all-spark-profiles` and `run-spark-4.2-tests`; I do not have label permissions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
