Daniil Grbic created SPARK-59987:
------------------------------------
Summary: AssertionError on NOT CAST to BOOLEAN during V2 predicate
pushdown
Key: SPARK-59987
URL: https://issues.apache.org/jira/browse/SPARK-59987
Project: Spark
Issue Type: Bug
Components: Bug
Affects Versions: 4.2.0
Reporter: Daniil Grbic
With ANSI mode enabled, CAST(c AS BOOLEAN) becomes a scalar V2Cast
([pointer|https://github.com/apache/spark/blob/2f4102c8117a087aba1324d52b268f7966670136/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala#L152-L154]).
When NOT/OR/AND wraps that cast inside COALESCE, the builder recursively
translates NOT's child, then asserts that the result is already a V2Predicate -
the child is a V2Cast, so the assertion fails before predicate translation can
return None
([pointer|https://github.com/apache/spark/blob/2f4102c8117a087aba1324d52b268f7966670136/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala#L241-L245]).
A top-level WHERE NOT CAST(c AS BOOLEAN) follows a different path and does not
hit that assertion. DataSourceV2Strategy handles the outer NOT separately and
translates CAST(c AS BOOLEAN) as a leaf predicate
([pointer|https://github.com/apache/spark/blob/2f4102c8117a087aba1324d52b268f7966670136/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala#L936-L940]).
With the default alwaysCreateV2Predicate setting, buildPredicate wraps the
scalar V2Cast as BOOLEAN_EXPRESSION(V2Cast(...)), then the outer path produces
NOT(BOOLEAN_EXPRESSION(V2Cast(...)))
([pointer|https://github.com/apache/spark/blob/2f4102c8117a087aba1324d52b268f7966670136/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala#L41-L50]).
Issue repro:
{code:sql}
SET spark.sql.ansi.enabled=true;
CREATE NAMESPACE h2.test;
CREATE TABLE h2.test.boolean_cast_repro (c INT);
INSERT INTO h2.test.boolean_cast_repro VALUES (0), (1);
SELECT c FROM h2.test.boolean_cast_repro
WHERE coalesce(NOT CAST(c AS BOOLEAN), false);
{code}
Expected result: one row containing 0. If the filter cannot be pushed down,
Spark should evaluate it after the scan.
Actual result: optimization fails with org.apache.spark.SparkException:
[INTERNAL_ERROR], caused by java.lang.AssertionError: assertion failed at
V2ExpressionBuilder.scala:243.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
