tlfua commented on issue #7246: URL: https://github.com/apache/datafusion-comet/issues/7246#issuecomment-6596774990
I re-ran the reproductions on current `main` (644a870883, default Spark 4.1 profile, ANSI on, AQE off). Each query was its own `CometSqlFileTestSuite` fixture, so every row arrives in one batch, as in the original report. | Query | Result on `main` | Plan | | --- | --- | --- | | `id = 1 OR CAST(txt AS INT) = 2` | ✅ matches Spark | `CometProject` | | `id <> 1 AND CAST(txt AS INT) = 2` | ✅ matches Spark | `CometProject` | | `id IN (1, CAST(txt AS INT))` | ✅ matches Spark | `CometProject` | | `id = (SELECT max(k) FROM nums) OR CAST(txt AS INT) = 2` | ✅ matches Spark | `CometProject` | | `n + CAST(txt AS INT)` (`n` NULL) | ❌ `CAST_INVALID_INPUT` | `CometProject` | | `CAST(txt AS INT) DIV n` (`n` NULL) | ❌ `CAST_INVALID_INPUT` | `CometProject` | | `concat(IF(id = 1, NULL, ...), CAST(CAST(txt AS INT) AS STRING))` | ❌ `CAST_INVALID_INPUT` | `CometProject` | | `ORDER BY a, b + 1 LIMIT 2` | ❌ `ARITHMETIC_OVERFLOW` | `CometTakeOrderedAndProjectExec` | All plans are fully native over `CometNativeScan`. The `OR`, `AND` and `IN` cases, including the scalar subquery form, are fixed by #7326 (`LazyLogical` / `lazy_in_list`), which landed after the `c4a0b1f2bc` the issue was reproduced on. What remains: 1. **NULL operands of binary arithmetic (`+`, `DIV`, and presumably `-`, `*`, `/`, `%`) and `concat`.** Spark skips the later operand on rows where an earlier one is NULL. The existing `NullShortCircuit` / `withNullShortCircuit` / `canSkipEvaluation` machinery seems to fit, gated on the skipped operand being able to fail. 2. **Later top-K sort keys.** Nothing covers these yet. I plan to take these as two PRs, (1) first. For top-K, would you prefer a lazy native evaluation of the later keys, or a fallback to Spark when ANSI is on and a later sort key can fail? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
