grorge123 commented on PR #5526: URL: https://github.com/apache/datafusion-comet/pull/5526#issuecomment-5979243545
Rebased onto `569eaa59d` to resolve the conflicts with main. The only change in behavior comes from #5750, which makes floating-point `array_union` `Incompatible` (falling back to Spark) on every Spark version except 4.2.0. This PR mixes `CodegenDispatchFallback` into `CometArrayUnion` for its NullType-element case, and that mixin would also have sent the floating-point case through the dispatcher instead of falling back. So `CodegenDispatchFallback` now has `dispatchesIncompatible` (default `true`). `CometArrayUnion` sets it to `false`: a NullType-element side still runs through the dispatcher, and a floating-point element type falls back as #5750 intends. `QueryPlanSerde` skips the dispatcher for an `Incompatible` result when the flag is false, and `GenerateDocs` documents those reasons as falling back. #5750's own fallback test in `CometArrayExpressionSuite` fails when the flag is set back to `true`. The other conflicts were in SQL fixtures and `expressions.md`, where I kept ma in's content and re-added this PR's queries and notes. #4715 now builds a nested list literal whose inner lists are all empty at its declared depth, the same fix this PR had made, so the planner change here is down to its unit test. Tested locally on the rebased head, with the native library built in release mode: - `cargo fmt`, `cargo clippy`, `spotless:check`, and the `datafusion-comet` native unit tests (592). - Spark 4.1: `CometSqlFileTestSuite` (620), `CometNullTypeCompositionSuite` (28), `CometCodegenSuite` (111), `CometNativeCastSuite` (190), `CometCodegenSourceSuite` (67), `CometExpressionSuite` (176), `CometArrayExpressionSuite` (75), `CometMapExpressionSuite` (32), `CometTemporalExpressionSuite` (37), `CometJsonExpressionSuite` (8), `CometAggregateSuite` (128), `CometExecSuite` (154), `CometJoinSuite` (65), `CometNativeShuffleSuite` (65), `CometShuffleSuite` (48), `DisableAQECometShuffleSuite` (48), `CometNativePositionalRoundRobinSuite` (11), `UtilsSuite` (11), `GenerateDocsSuite` (4), `CometFloatSemanticsSuite` (412), `SqlFileTestParserSuite` (13), `CometRangeExecSuite` (14), and `CometLiteralSuite` (4). - Spark 3.5: `CometSqlFileTestSuite` (620), `CometNullTypeCompositionSuite` (28), and `CometCodegenSuite` (110). - Spark 3.4: `CometCodegenSuite` (110). All passed. Spark's own SQL suites have not been run locally. Assisted-by: Claude Code (claude-opus-5-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
