zhangfengcdt opened a new pull request, #6599: URL: https://github.com/apache/datafusion-comet/pull/6599
## Which issue does this PR close? Closes #6520 ## Rationale for this change `array_contains` on an array whose elements hold a `FLOAT` or `DOUBLE`, at any depth, reports `Incompatible`, because the native path compares elements by their bits while Spark's `genEqual` treats `-0.0` and `0.0` as equal and all NaNs as equal. By default those calls go through the JVM codegen dispatcher, which is correct but leaves the native path. With `allowIncompatible=true` they get the bitwise kernel and the wrong answer. This adds a native kernel with Spark's equality, built the way `spark_array_remove` was in #6518, so these calls stay native and report `Compatible`. ## What changes are included in this PR? - `spark_array_contains`, a Comet kernel for arrays with a float leaf at any depth. A constant float value is one test per element in a single pass over all the values. A value per row uses Spark's float comparison. Nested array and struct elements go through the shared `spark_equality`. - `CometArrayContains` routes float-element arrays to the kernel and reports them `Compatible`. Other element types keep the current path through datafusion-spark's `array_contains`. The `Incompatible` reason and the dispatcher fallback are removed. - The one-pass prefix bit count that `array_remove` used for its offsets moves to the module so both kernels share it. `array_remove`'s behavior is unchanged. - The float semantics page covers `array_contains`, and the audit entry for it is corrected. Benchmark, Comet kernel versus the native path other element types take, 8,192 rows, medians: | Case | len 8 | len 8, nulls | len 50 | len 50, nulls | | --- | --- | --- | --- | --- | | Constant value | 27 vs 31 µs | 35 vs 67 µs | 60 vs 67 µs | 76 vs 120 µs | | Value per row | 18 vs 30 µs | 32 vs 116 µs | 18 vs 52 µs | 58 vs 242 µs | This is only for fixing `array_contains`. `array_remove`, `array_position` and `array_contains`, which the issue mentions, is left for a follow-up. ## How are these changes tested? - Unit tests for the kernel: every edge value looked up in rows of edge values through both the constant and per-row paths, null rows, null values, empty arrays, scalar inputs, nested array elements, and struct elements. - `array_contains_floating_point.sql`, with every query marked `expect_native`, covers signed zeros, NaNs, null elements, a null array or value, a value per row, nested arrays and structs, and literals. - The two routing files now expect native execution on the float column in strict floating-point mode, with and without the dispatcher. They failed before this change. - The `array_contains` benchmark cases are added to `float_arrays`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
