dwsmith1983 opened a new issue, #7284:
URL: https://github.com/apache/datafusion-comet/issues/7284
## Describe the bug
Since #7261, a `make_date` under an expression that Comet runs through the
codegen dispatcher falls back to Spark, and the whole operator holding it
leaves Comet. Before #7261 the same expressions stayed in Comet.
`CometMakeDate.unsupportedParentReason` walks the parents of every
`make_date`, and any multi-child parent it does not list ends in the default
arm ("Other native multi-child expressions may evaluate operands that Spark
skips"):
https://github.com/apache/datafusion-comet/blob/967d09b52694d64c423012ed7dcc8f7ec6e20413/spark/src/main/scala/org/apache/comet/serde/datetime.scala#L470-L524
That covers dispatched parents such as `AddMonths` and `MonthsBetween`. A
dispatched parent never runs the native `make_date`, though.
`CometCodegenDispatch` binds the whole subtree and runs Spark's own generated
code for it, so the `make_date` under it is evaluated by Spark with Spark's
short-circuiting:
https://github.com/apache/datafusion-comet/blob/967d09b52694d64c423012ed7dcc8f7ec6e20413/spark/src/main/scala/org/apache/comet/serde/CometScalaUDF.scala#L176-L179
Results are the same either way; only the operator moves back to Spark.
## Steps to reproduce
On `main` (967d09b52), Spark 4.1 and 3.5, default configuration:
```sql
CREATE TABLE md_parent(y INT, m INT, d INT, dt DATE) USING parquet;
INSERT INTO md_parent VALUES
(2024, 1, 31, DATE '2024-03-15'), (NULL, 2, 1, NULL), (2024, NULL, 1, DATE
'2023-12-01');
SELECT add_months(dt, 1) FROM md_parent; -- stays in
Comet (dispatched)
SELECT add_months(make_date(y, m, d), 1) FROM md_parent; -- falls back
SELECT months_between(make_date(y, m, d), dt) FROM md_parent; -- falls back
```
The second query plans as:
```
Project [COMET: make_date requires Spark evaluation under AddMonths]
+- CometColumnarToRow
+- CometNativeScan parquet spark_catalog.default.md_parent
```
and the third falls back with `make_date requires Spark evaluation under
MonthsBetween`. On f1121ca14, the commit before #7261, all three queries pass
`expect_dispatch` in `CometSqlFileTestSuite`.
## Expected behavior
A `make_date` under a parent that the codegen dispatcher will take stays in
Comet, as it did before #7261. One way: have the walk stop at a parent whose
serde is a `CometCodegenDispatch` while
`spark.comet.exec.scalaUDF.codegen.enabled` is on, since that subtree runs as
Spark code. Natively evaluated parents keep their current checks.
## Additional context
Found while merging `main` into #5864, which adds more dispatched datetime
parents (`DateAddInterval`, `DateAddYMInterval`, `SubtractDates`,
`SubtractTimestamps`, `TimestampAddYMInterval`); a `make_date` under any of
them falls back the same way.
@pingzh, this follows from #7261. Would you like to take it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
