andygrove opened a new issue, #6613: URL: https://github.com/apache/datafusion-comet/issues/6613
### Describe the bug Spark's `ArrayContains` is a null-intolerant `BinaryExpression`. When the array is NULL it returns NULL without evaluating the second argument. Comet's native `array_contains` evaluates both arguments over the whole batch first, because `ScalarFunctionExpr` evaluates every child before calling the function. So a second argument that throws under ANSI raises on rows Spark never evaluates it for, and a query that Spark completes fails under Comet. The same happens for `array_position`, `array_remove`, `arrays_overlap` and `map_contains_key`, which Spark rewrites to `array_contains(map_keys(m), k)`. `element_at` is not affected, because `CometElementAt` wraps the lookup in a `CASE WHEN <array> IS NOT NULL` guard (#5766). ### Steps to reproduce ```sql SET spark.sql.ansi.enabled=true; -- the default on Spark 4.x CREATE TABLE t(ai ARRAY<INT>, m MAP<INT, INT>, s STRING) USING parquet; INSERT INTO t VALUES (NULL, NULL, 'bad'), (array(1), map(1, 10), '1'); SELECT array_contains(ai, CAST(s AS INT)) FROM t; SELECT array_position(ai, CAST(s AS INT)) FROM t; SELECT array_remove(ai, CAST(s AS INT)) FROM t; SELECT arrays_overlap(ai, array(CAST(s AS INT))) FROM t; SELECT map_contains_key(m, CAST(s AS INT)) FROM t; ``` Each query runs as `CometProject` over `CometNativeScan` and fails with: ``` org.apache.spark.SparkNumberFormatException: [CAST_INVALID_INPUT] The value 'bad' of the type "STRING" cannot be cast to "INT" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead. SQLSTATE: 22018 ``` Reproduced on `main` at 98662215dc with Spark 4.1.3, the default profile, with one `CometSqlFileTestSuite` fixture per query. ### Expected behavior Spark returns NULL for the first row, and `true`, `1`, `[]`, `true` and `true` for the second. When the bad value sits on a row whose array is not NULL, Spark raises `CAST_INVALID_INPUT` too, and Comet already matches it there. ### Additional context - `branch-1.1` routes all five the same way for an `ARRAY<INT>`, so 1.1.0 should be affected too. I have not run it there. - Only ANSI mode is affected. With ANSI off the cast returns NULL, and `try_cast` is a workaround. - On `main`, `array_contains` over an array with a float leaf avoids this, because it goes through the codegen dispatcher, which runs Spark's generated code. #6599 would move those arrays to a native kernel and bring this back for them. The review there covers it. - This is one shape of the family tracked in #6006. `CometElementAt` (#5766) and `CometArrayJoin.orderInsensitive` show two existing fixes: a `CASE WHEN` guard that masks the lookup, or keeping any second argument other than a literal or column read on the dispatcher. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
