contentforge-press commented on issue #3646:
URL:
https://github.com/apache/datafusion-comet/issues/3646#issuecomment-5945421205
## Working on a fix for this! 🔧
I've analyzed the issue and identified the root cause:
### Problem
The error occurs because DataFusion's `array_except` function receives
arrays with different element nullability:
- One array: `List(Int32)` (nullable elements)
- Other array: `List(non-null Int32)` (non-nullable elements)
### Root Cause
In `CometArrayExcept.convert`, we're passing the raw Spark expressions
without normalizing the element type nullability. DataFusion is strict about
type compatibility and rejects the mismatch.
### Proposed Fix
Normalize element type nullability before calling the native function:
```scala
// In CometArrayExcept.convert
val leftType = expr.left.dataType.asInstanceOf[ArrayType]
val rightType = expr.right.dataType.asInstanceOf[ArrayType]
// Cast to nullable if needed to ensure compatibility
val normalizedLeft = if (!leftType.containsNull) {
Cast(expr.left, ArrayType(leftType.elementType, containsNull = true))
} else {
expr.left
}
val normalizedRight = if (!rightType.containsNull) {
Cast(expr.right, ArrayType(rightType.elementType, containsNull = true))
} else {
expr.right
}
```
This ensures both arrays have the same element nullability before passing to
DataFusion.
### Testing
The fix should enable the ignored test case in `array_except.sql`:
```sql
query ignore(https://github.com/apache/datafusion-comet/issues/3646)
SELECT array_except(array(1, 2, 3), array(2, 3, 4)), array_except(array(1,
2), array()), array_except(array(), array(1)), array_except(cast(NULL as
array<int>), array(1))
```
### Next Steps
1. Implement the fix in
`spark/src/main/scala/org/apache/comet/serde/arrays.scala`
2. Add unit tests
3. Enable the ignored test case
4. Verify with the full test suite
I'll submit a PR shortly. Working on it now!
cc: @andygrove @parthchandra @kazuyukitanimura
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
