zhangfengcdt opened a new issue, #25981:
URL: https://github.com/apache/datafusion/issues/25981
### Describe the bug
`SparkArrayContains` in the `datafusion-spark` crate
(`datafusion/spark/src/function/array/array_contains.rs`) delegates the element
comparison to `array_has`, which compares values by their bits. Spark's
`ArrayContains` compares elements with `genEqual`, under which `-0.0` equals
`0.0` and all NaNs are equal, at any depth of a struct or array element. So the
two disagree on those values:
| | Spark | `SparkArrayContains` |
| --- | --- | --- |
| `array_contains(array(-0.0D), 0.0D)` | `true` | `false` |
| `array_contains(array(double('NaN')), double('NaN'))` | `true` | `true`
only when the NaN bit patterns match |
The null semantics patched in on top of `array_has` are right; only the
equality differs.
### To Reproduce
```rust
use std::sync::Arc;
use arrow::array::{Float64Array, ListArray};
use arrow::datatypes::{DataType, Field};
use datafusion_common::ScalarValue;
use datafusion_expr::{ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl};
use datafusion_spark::function::array::array_contains::SparkArrayContains;
let array = ListArray::new(
Arc::new(Field::new("item", DataType::Float64, true)),
arrow::buffer::OffsetBuffer::new(vec![0, 1].into()),
Arc::new(Float64Array::from(vec![-0.0])),
None,
);
let result =
SparkArrayContains::default().invoke_with_args(ScalarFunctionArgs {
args: vec![
ColumnarValue::Array(Arc::new(array)),
ColumnarValue::Scalar(ScalarValue::Float64(Some(0.0))),
],
arg_fields: vec![],
number_rows: 1,
return_field: Arc::new(Field::new("result", DataType::Boolean, true)),
config_options: Default::default(),
})?;
// Spark returns true for array_contains(array(-0.0D), 0.0D); this returns
false.
```
### Expected behavior
`array_contains` matches Spark's `genEqual` for float and double elements:
`-0.0` equals `0.0`, and any NaN equals any other NaN, including inside struct
and array elements.
### Additional context
Found while working on apache/datafusion-comet#6520. Comet currently routes
`array_contains` on float elements away from this function because of the
mismatch, and is adding its own kernel with Spark's equality in the meantime; a
fix here would let it drop that kernel. `arrays_overlap` (#20781) in the same
crate likely has the same equality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
