[
https://issues.apache.org/jira/browse/SPARK-60315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-60315:
-----------------------------
Affects Version/s: 3.5.8
4.2.0
4.1.3
4.0.4
> Filter raises an error with whole-stage codegen off but not on when an
> earlier conjunct is null
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-60315
> URL: https://issues.apache.org/jira/browse/SPARK-60315
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.8, 4.2.0, 5.0.0, 4.1.3, 4.0.4
> Reporter: Max Gekk
> Priority: Major
>
> The same query raises an error with whole-stage codegen off and returns rows
> with it on. It happens when a filter's condition is a conjunction, an earlier
> conjunct is NULL on a row, and a later conjunct raises on that row.
> h3. Reproducer
> Default (ANSI) mode, a Parquet table, local mode:
> {code:sql}
> CREATE TABLE t USING parquet AS
> SELECT * FROM VALUES (NULL, 1, 2147483647), (0, 1, 0) AS v(a, b, y);
> SET spark.sql.codegen.wholeStage=true;
> SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND y + 1 > 0;
> -- 0
> SET spark.sql.codegen.wholeStage=false;
> SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND y + 1 > 0;
> -- [ARITHMETIC_OVERFLOW] overflow. Use 'try_add' to tolerate overflow and
> return NULL instead. ... SQLSTATE: 22003
> {code}
> The same happens in legacy mode with year-month interval arithmetic, which
> raises on overflow in every mode:
> {code:sql}
> SET spark.sql.ansi.enabled=false;
> CREATE TABLE t2 USING parquet AS
> SELECT * FROM VALUES (NULL, 1, 178956970), (0, 1, 0) AS v(a, b, y);
> SELECT a FROM t2 WHERE NOT (a > 0 AND b > 0) AND make_ym_interval(y, 0) +
> INTERVAL '1' YEAR > INTERVAL '0' YEAR;
> -- codegen on: 0
> -- codegen off: [INTERVAL_ARITHMETIC_OVERFLOW.WITH_SUGGESTION] Integer
> overflow while operating with intervals. ... SQLSTATE: 22015
> {code}
> On the first row, {{NOT (a > 0 AND b > 0)}} is {{NOT (NULL AND TRUE)}}, which
> is NULL, and the second conjunct overflows. The second row keeps the Parquet
> row group from being skipped by its statistics; it passes both conjuncts on
> both paths. {{NOT (a > 0 AND b > 0)}} is written so that the optimizer infers
> no {{isnotnull}} filter that would drop the first row earlier on both paths.
> Checked on master at 5d17442dec4 (8 October 2026) with {{bin/spark-sql
> --master local[1]}}.
> h3. Cause
> The two paths evaluate the conjunction differently:
> * *Whole-stage codegen* ({{FilterExec.generatePredicateCode}}) generates each
> top-level conjunct separately and skips the row as soon as one is false *or
> null* ({{if (isNull || !value) continue;}}). A filter keeps only rows whose
> condition is true, so the remaining conjuncts are never evaluated on that row.
> * *Interpreted* ({{FilterExec.doExecute}} without codegen, via
> {{Predicate.create}} over the whole condition) evaluates the {{And}} tree
> with {{And.eval}}, which short-circuits only on {{false}}. A null left side
> still evaluates the right side, because {{NULL AND FALSE}} is false and
> {{NULL AND TRUE}} is null, and that evaluation raises.
> Both are valid evaluations of {{AND}} in isolation. Inside a filter, though,
> false and null both drop the row, so the distinction the interpreted path
> computes the right side for is never used.
> h3. Expected
> The query should behave the same with whole-stage codegen on and off.
> Whole-stage codegen is the default, and its behaviour (no error) is the one
> users see, so the interpreted filter should match it: evaluate the top-level
> conjuncts in the same order and stop at the first one that is false or null.
> The same applies to other interpreted evaluations of a filter condition that
> codegen splits into conjuncts.
> Found by differential testing of random queries against Spark's own row
> engine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
