andygrove opened a new issue, #4830:
URL: https://github.com/apache/datafusion-comet/issues/4830
## Describe the bug
`CometArrayFilter.convert` has a fast path that lowers `filter(arr, x -> x
IS NOT NULL)` to the native `array_compact` serde, to avoid the per-batch JNI
cost of the codegen dispatcher. The guard only checks that the lambda body's
first child is an `IsNotNull`, not that the `IsNotNull` operand is the lambda
variable being iterated.
```scala
expr.function.children.headOption match {
case Some(_: IsNotNull) =>
// Fast path: `array_compact` lowers to `filter(arr, x -> x is not
null)`.
CometArrayCompact.convert(expr, inputs, binding)
...
}
```
As a result, a lambda whose body is `IsNotNull` of something other than the
element (for example a captured column) is incorrectly treated as
`array_compact`.
## To Reproduce
A query like:
```sql
SELECT filter(arr, x -> captured_col IS NOT NULL) FROM t
```
Spark semantics: keep every element of `arr` when `captured_col` is non-null
for that row, and produce an empty array when `captured_col` is null. The `x`
variable is unused.
Comet fast-path semantics: `array_compact(arr)`, i.e. drop the null elements
of `arr` regardless of `captured_col`.
These diverge whenever `arr` contains null elements or `captured_col` is
null.
## Expected behavior
The fast path should only fire when the `IsNotNull` operand is the lambda's
own variable, so the rewrite is a faithful `array_compact`. Any other shape
should fall through to the native higher-order-function path (once available)
or the codegen dispatcher / Spark.
## Additional context
This predates #4744 (introduced in #2536), but it became more visible while
reviewing the native lambda work in #4744. The fix is to tighten the guard in
`CometArrayFilter.convert`
(`spark/src/main/scala/org/apache/comet/serde/arrays.scala`) to confirm the
`IsNotNull` argument references the single `NamedLambdaVariable` of the lambda.
A regression test with a captured column and null elements would cover it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
