Yang Jie created SPARK-59603:
--------------------------------
Summary: Memoize a subexpression repeated inside a conditional
branch
Key: SPARK-59603
URL: https://issues.apache.org/jira/browse/SPARK-59603
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yang Jie
Subexpression elimination cannot reach a subexpression that is repeated inside
a single branch of an {{if}} or {{case when}}, and the reason is structural
rather than an oversight: it evaluates its candidates before the projection, so
{{EquivalentExpressions}} only collects what is always evaluated
({{ConditionalExpression.alwaysEvaluatedInputs}}) plus what every branch of a
group shares ({{branchGroups}}, an intersection). A subexpression repeated
inside one branch body and nowhere else is in no group, so it is evaluated once
per occurrence for every row that reaches the branch.
Since SPARK-58818 a {{With}} is evaluable and memoizes per evaluation, which
covers exactly this: the definition sits inside the branch, so nothing is
computed for a row that does not reach it, and the references read one value.
This adds an optimizer rule, {{MemoizeCommonExpressionsInBranches}}, that
rewrites the tallest repeated subexpression of each conditionally evaluated
child of an {{if}} / {{case when}} into a {{With}}. It runs late (after the
simplification rules, so what it memoizes is what survives them) and leaves the
{{With}} in the branch, which is the shape {{RewriteWithExpression}} keeps
anyway.
Measured on {{select case when id < 5 then udf(id) + udf(id) else 0 end from
range(0, 10, 1, 1)}} with a counting UDF: 5 evaluations with the rule, 10
without.
Guarded by {{spark.sql.optimizer.memoizeCommonExpressionsInBranches.enabled}},
default false, so plans are unchanged unless it is turned on. A candidate is
skipped when it is cheap ({{CollapseProject.isCheap}}), when it holds a
common-expression reference or a lambda variable (neither can be evaluated
where the definition would sit), when it holds an aggregate, window or
generator expression (the planner looks for those where they stand), or when it
holds a subquery expression. A branch that already holds a {{With}} is left
alone, since {{RewriteWithExpression}} defers a nested {{With}} to a pass that
no longer runs by then.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
