sunchao opened a new pull request, #5851: URL: https://github.com/apache/datafusion-comet/pull/5851
## Which issue does this PR close? Follow-up to #5558 and #3178. ## Rationale for this change `array_join` still differs from Spark in cases that cannot be evaluated correctly in isolation. Before Spark 4.2, a nullable replacement can depend on nullability inferred by the surrounding project or filter (SPARK-57200). With `NO_CODEGEN`, Spark evaluates the array before the replacement, so using a native null guard or codegen dispatch can suppress an array-argument error. A compound nullable replacement can also be evaluated more than once by the native guard or an enclosing native expression. Keep the enclosing Spark operator responsible for these cases, including when a parent expression would otherwise hide the join inside JVM codegen dispatch. ## What changes are included in this PR? - Check independent expression roots once, using an iterative traversal before decimal promotion and recursive serialization. - Preserve Spark execution for interpreted joins, compound nullable replacements, and default pre-4.2 nullable non-literal replacements. A native compatibility opt-in still permits simple nullable column replacements. - Keep existing native support for ordinary literal/non-nullable replacements and nullable column replacements on Spark 4.2+. - Add project/filter, parent-dispatch, stateful-replacement, interpreted error-order, and deep-tree regression coverage; update compatibility documentation and affected SQL expectations. ## How are these changes tested? - Spark 4.1.3: three focused serializer tests passed after compiling the full reactor. On the unchanged upstream production code, the two new routing regressions failed and the existing null-guard test passed; restoring the fix made all three pass. - Spark 4.1.3 JVM packaging and documentation generation passed, along with Scalastyle and Spotless checks. Markdown formatting and `git diff --check` passed. - Spark 4.2.0: the same three serializer tests passed with a clean full-reactor build, Scalastyle, and Spotless. - Native-backed execution tests are included and compiled but have not been run locally: the native build is blocked because the configured Cargo mirror does not contain the locked `aws-smithy-runtime-api` 1.16.0 dependency. No dependency or lockfile changes are included. This draft needs the native-backed CI runs before it is ready to merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
