Yang Jie created SPARK-59589:
--------------------------------
Summary: Hand Expand's varying output columns to the consumer as
globals
Key: SPARK-59589
URL: https://issues.apache.org/jira/browse/SPARK-59589
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yang Jie
{{ExpandExec.doConsume}} allocates a mutable state for each output column that
varies across the Expand's branches, and hands it to the consumer as
{{JavaCode.variable(value, dataType)}}.
{{CodegenContext.addMutableState}} returns a field of the generated class: a
name of its own when it can inline the type, otherwise a slot in a compacted
array, e.g. {{mutableStateArray_0[3]}}. Either form is reachable from every
method of that class, and neither is a local variable, which is what
{{JavaCode.global}} says and {{JavaCode.variable}} does not.
The difference is not cosmetic. {{CodeGenerator.getLocalInputVariableValues}}
collects the {{VariableValue}}s of an operator's input so that a caller can
pass them into a split function -- used by
{{AggregateCodegenSupport.splitAggregateExpressions}} and by
{{CommonExprSlots.fill}} (SPARK-59295). Taking a field for a local variable
adds a parameter that shadows the field, and for the compacted form the
parameter name is a slot expression, which does not compile. SPARK-59295 had to
add an "is this a Java identifier" check on the candidate parameters partly for
this source.
Whole-stage subexpression elimination already wraps its own {{addMutableState}}
results with {{JavaCode.global}} / {{JavaCode.isNullGlobal}}
({{CodeGenerator.subexpressionEliminationForWholeStageCodegen}}), so this makes
{{ExpandExec}} consistent with it.
No query is known to reach the bad parameter today: with a consume function per
operator (the default) the value is passed as an argument to a fresh parameter,
and the shapes tried without it -- CUBE over string columns, and a regular
aggregate over a varying string column under a distinct rewrite -- do not put
the slot in a parameter list. This is a latent trap rather than an observed
failure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
