Peter Toth created SPARK-60084:
----------------------------------
Summary: PlanMerger loses a reused filter attribute that an
earlier merge folded into another filter
Key: SPARK-60084
URL: https://issues.apache.org/jira/browse/SPARK-60084
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
With filter propagation through a join, a merged subplan can end up with an
aggregate {{FILTER}} on an attribute its child doesn't output. The query then
fails with {{INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND}}.
It needs both of these configs, both are off by default:
*
{{spark.sql.optimizer.mergeSubplans.filterPropagation.symmetricFilterPropagation.enabled=true}}
*
{{spark.sql.optimizer.mergeSubplans.filterPropagation.throughJoin.enabled=true}}
Repro:
{code:sql}
CREATE TABLE t USING parquet AS SELECT * FROM VALUES
(1, 1, 10), (1, 2, 3), (2, 1, 7), (2, 2, 9), (3, 1, 9), (3, 2, 6) AS v(a, b,
c);
SELECT
(SELECT sum(l.a) FROM t l LEFT JOIN t r ON l.a = r.a WHERE l.b = 1),
(SELECT max(l.a) FROM t l LEFT JOIN t r ON l.a = r.a WHERE l.b = 2 AND
coalesce(r.c, 10) > 5),
(SELECT count(*) FROM t l LEFT JOIN t r ON l.a = r.a WHERE l.b = 2);
{code}
Expected {{12, 3, 6}}, but it fails with:
{noformat}
[INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND] Could not find propagatedFilter_3#179 in
[sum#141L,max#182,count#145L,a#6,propagatedFilter_5#181,propagatedFilter_4#180].
SQLSTATE: XX000
{noformat}
What happens:
# Merging the 2nd subquery into the 1st creates a filter attribute for {{l.b =
2}} on the left side of the join.
# The 2nd subquery's post-join {{coalesce(r.c, 10) > 5}} folds that attribute
into a new {{AND}} alias. Only the new alias is carried up to the aggregate.
# The 3rd subquery reuses the {{l.b = 2}} attribute. {{PlanMerger}} flags a
reused attribute as already present in the {{Project}}s above, so it doesn't
append it there. Here it isn't present, so the 3rd subquery's {{FILTER}}
references a missing attribute.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
