dwsmith1983 opened a new issue, #6970: URL: https://github.com/apache/datafusion-comet/issues/6970
### Describe the bug With ANSI mode on and subexpression elimination on (its default), Spark's projection evaluates a subexpression that several of its expressions share for every row, before any of their NULL checks. The native `element_at` on a map and the native `map_from_arrays` wrap their work in a `CASE WHEN <input> IS NOT NULL` guard, so on a row whose map or key array is NULL the shared subexpression is never evaluated. When that subexpression raises, Spark fails the query and Comet returns rows. The guards are in `CometElementAt.convert` (the `needsNullGuard` branch in `arrays.scala`) and `CometMapFromArrays.convert` (`maps.scala`). Neither consults `QueryPlanSerde.canShortCircuitNulls`, which `withNullShortCircuit` uses to keep the arguments eager when a later one holds an operator-shared subexpression. `GetMapValue` goes through `withNullShortCircuit` and already matches Spark. Found in #5867 while answering https://github.com/apache/datafusion-comet/pull/5867#discussion_r4266282723, which reported the same gap for the dispatched lookups that PR adds. ### Steps to reproduce On main (7d8d9b154), as a SQL file test: ```sql -- Config: spark.sql.ansi.enabled=true -- Config: spark.sql.subexpressionElimination.enabled=true statement CREATE TABLE t(m1 map<int, int>, m2 map<int, int>, k1 array<int>, k2 array<int>, x double, i int) USING parquet statement INSERT INTO t VALUES (CAST(NULL AS MAP<INT, INT>), CAST(NULL AS MAP<INT, INT>), CAST(NULL AS ARRAY<INT>), CAST(NULL AS ARRAY<INT>), 0.0D, 0), (map(1, 5), map(1, 6), array(1), array(2), 1.0D, 1) query expect_error(DIVIDE_BY_ZERO) SELECT element_at(m1, 1 div i), element_at(m2, 1 div i) FROM t query expect_error(DIVIDE_BY_ZERO) SELECT element_at(m1, CAST(1.0D / x AS INT)), element_at(m2, CAST(1.0D / x AS INT)) FROM t query expect_error(DIVIDE_BY_ZERO) SELECT map_from_arrays(k1, array(1.0D / x)), map_from_arrays(k2, array(1.0D / x)) FROM t ``` Run one per file (a file stops at its first failure), each of the three queries fails with "Expected Comet to throw an error matching 'DIVIDE_BY_ZERO' but query succeeded". The same shape with `m1[1 div i], m2[1 div i]` passes. ### Expected behavior Comet raises `DIVIDE_BY_ZERO` like Spark. One way is to build these guards through `withNullShortCircuit`, or to check `canShortCircuitNulls` before adding them, so a shared subexpression stays evaluated for every row. ### Additional context The dispatched paths that #5867 adds decline the same shapes and fall back to Spark. Existing JVM codegen dispatch serdes on main may have a similar gap. I have not reproduced that yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
