viirya opened a new issue, #6823: URL: https://github.com/apache/datafusion-comet/issues/6823
### Describe the bug Spark's `In` evaluates the candidates of each row in order and stops at the first one that matches. Comet's `IN` does not: DataFusion's `InListExpr::evaluate`, for a list it does not build as a static filter, and Comet's `NestedPredicate` evaluate each candidate over the whole batch, and stop only once every row of the batch has matched. A later candidate therefore runs for rows that an earlier candidate already matched. When that candidate can fail, as many expressions can with ANSI mode on, Comet raises an error for a row that Spark never evaluates it for. ### Steps to reproduce With `spark.sql.ansi.enabled=true`, and both rows in one file so that they reach the same batch: ```sql CREATE TABLE t (id BIGINT, s STRING) USING parquet; INSERT INTO t SELECT /*+ REPARTITION(1) */ * FROM VALUES (0L, 'bad'), (1L, '1') AS v(id, s); SELECT id, id IN (0L, CAST(s AS BIGINT)) FROM t; ``` Spark returns `[0, true]`, `[1, true]`. Comet fails with `CAST_INVALID_INPUT` for `'bad'`, which is only in the row where `id = 0`, a row that `0L` already matched. Since #6763, a CASE candidate that reads a column takes the same path, so `SELECT id, id IN (0L, CASE WHEN id = 0 THEN CAST(s AS BIGINT) WHEN id = 2 THEN id END) FROM t` fails the same way. ### Expected behavior The same results as Spark: a candidate is evaluated only for the rows that no earlier candidate matched, or a query whose candidates can fail falls back to Spark. ### Additional context Spark 4 runs with ANSI mode on by default, so this affects default configurations. This is one instance of the data-dependent errors tracked in #6006. Found while reviewing #6763. ### Willingness to contribute I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
