andygrove opened a new pull request, #6554:
URL: https://github.com/apache/datafusion-comet/pull/6554
## Which issue does this PR close?
Closes #6425 on `branch-1.1`, for 1.1.0-rc2.
## Rationale for this change
This is the `branch-1.1` backport of #6542. Since #5692, `CometStaticInvoke`
and `CometInvoke` send every call they don't otherwise handle to the codegen
dispatcher. That includes calls to DataSource V2 catalog functions. Such a
function can return a `Decimal` at a different scale from the type it declares,
or one that doesn't fit it, and Spark corrects that only when it writes a row.
The dispatcher writes an Arrow vector of the declared type, so 1.1.0-rc1
returns `0.03` where Spark returns `3.00`, and `1000000.00` where Spark returns
null. 1.0.0 ran these calls in Spark. #6542 has the details.
`branch-1.0` doesn't need it, because it doesn't have #5692.
## What changes are included in this PR?
A cherry-pick (`-x`) of #6542's commit. The dispatcher now runs a
`StaticInvoke` or `Invoke` only when it calls into Spark's own classes, or when
it is the predicate of a typed `Dataset.filter`. It checks the whole dispatched
tree, so any other call falls back to Spark, as it did in 1.0.0. Outside the
two test files below, every hunk is the same as upstream's, and `statics.scala`
is byte-identical to `main`'s.
Two test-only adaptations:
- `CometCodegenSuite`: the import block conflicted. #6497 is a test-only PR
that isn't on this branch, and it added `aggregate.{Final, Partial}`,
`CometFilterExec`, `CometHashAggregateExec` and `Row` to that block. I kept
this branch's imports and took only #6542's additions. I also added `Row`,
because #6542's new test uses it at suite scope.
- `CometIcebergSystemFunctionSuite`: the new Iceberg test calls
`withHadoopCatalog`, which `main`'s `CometIcebergTestBase` got from #6456. This
branch's backport of #6456 (#6486) left that test-base change out. So the suite
gets a private copy of the helper, and the test body is unchanged.
## How are these changes tested?
On this branch, with the default Spark 4.1 profile:
- `CometCodegenSuite` and `CometIcebergSystemFunctionSuite`: 119 passed,
including the four new #6425 tests. `main` runs 121. The other two are #6497's
and #6456's tests, which aren't on this branch.
- `CometExpressionSuite`, `CometStringExpressionSuite` and
`CometCodegenSourceSuite`: 267 passed, 4 ignored.
- `CometSqlFileTestSuite`, `CometCodegenFuzzSuite` and
`CometIcebergResidualPushdownSuite`: 596 passed.
- Without the fix, both new SQL tests fail on this branch. With the
`CometInvokeTargets` check in `emitJvmCodegenDispatch` disabled, the DataSource
V2 test returns #6425's wrong answers (`0.03` for `3.00`, and `1000000.00`
where Spark returns null). The Iceberg test finds no fallback.
I didn't run the other Spark profiles locally. CI runs them on this branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
