andygrove opened a new issue, #6520: URL: https://github.com/apache/datafusion-comet/issues/6520
### What is the problem the feature request solves? `array_contains` on an array whose elements hold a `FLOAT` or `DOUBLE`, at any depth, reports `Incompatible`, because DataFusion's `array_contains` compares elements by their bits, while Spark's `genEqual` treats `-0.0` and `0.0` as equal and all NaNs as equal. `CometArrayContains` mixes in `CodegenDispatchFallback`, so by default these calls go through the JVM codegen dispatcher. That matches Spark, but the expression leaves the native path. With `spark.comet.expression.ArrayContains.allowIncompatible=true` they get the bitwise kernel, and the wrong answer for those values. ### Describe the potential solution A native `array_contains` for float elements built on `float_semantics::spark_equality`, the way `spark_array_remove` in #6518 is: for a constant value, one test per element in one pass over all the values, and for a value per row or nested elements, `spark_equality`. It has to keep Spark's three-valued result, which is null when nothing matches and the array holds a null element. `CometArrayContains` could then report `Compatible` for float elements and stop dispatching. `array_remove`, `array_position` and `array_contains` all find the elements of a row that equal a value under Spark's rules, today in three different ways, so one shared kernel that produces that match could serve all three. ### Additional context Part of #6385. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
