andygrove opened a new issue, #4949: URL: https://github.com/apache/datafusion-comet/issues/4949
https://github.com/apache/datafusion-comet/issues/4142 ## Describe the bug On Spark 4.2.0, TPC-DS `q77a` loses Comet acceleration for a large part of its plan. This showed up while regenerating the TPC-DS plan-stability golden files for the released Spark 4.2.0 (previously the `spark-4.2` profile targeted `4.2.0-preview4`). Comparing the approved plan for `q77a` against the fallback baseline (`approved-plans-v2_7/q77a`): | | `approved-plans-v2_7` (3.4 - 4.1) | `approved-plans-v2_7-spark4_2` | | --- | --- | --- | | `CometUnion` | 4 | **0** | | bare `Union` | 0 | **4** | | non-Comet `Exchange` / `HashAggregate` | 0 | **12** | | `Scan OneRowRelation` | 0 | **3** | ## Root cause Spark 4.2 plans a `OneRowRelation` into the `Union` branches of this query (it is absent from the plan on 3.4 - 4.1). Comet does not support it: ``` +- Scan OneRowRelation [COMET: Scan OneRowRelation is not supported] ``` That single unsupported leaf cascades: 1. The `Union` can no longer be converted, so all 4 `CometUnion` become bare `Union`. 2. Its Comet children get capped with `CometNativeColumnarToRow`. 3. The `Final` / `PartialMerge` aggregates above the `Union` then have no Comet partial aggregate producing their intermediate buffers, so the guard in `CometBaseAggregate.doConvert` (`spark/src/main/scala/org/apache/spark/sql/comet/operators.scala:1639`) rejects them: ``` HashAggregate [COMET: Comet aggregate that merges intermediate buffers requires a Comet child aggregate when the intermediate buffer formats are incompatible with Spark. Incompatible aggregate function(s): sum] ``` ## Expected behavior `q77a` should keep the same degree of Comet acceleration on Spark 4.2 that it has on 3.4 - 4.1 (fully native `CometUnion` + `CometHashAggregate`). ## Additional context `OneRowRelation` is already listed in the default value of `spark.comet.sparkToColumnar.supportedOperatorList` (`spark/src/main/scala/org/apache/comet/CometConf.scala:801`), so routing it through the Spark-to-Arrow path may be most of the fix. The `adding_a_new_spark_version.md` contributor guide also calls out "a new `OneRowRelation` planning path" as a known Spark 4.x behavioral difference. The regenerated `approved-plans-v2_7-spark4_2/q77a` golden bakes in the degraded plan so that CI is green for the Spark 4.2.0 bring-up. That golden should be removed (letting `q77a` fall back to the shared baseline plan) as part of fixing this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
