[
https://issues.apache.org/jira/browse/SPARK-59589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59589:
-----------------------------------
Labels: pull-request-available (was: )
> Hand Expand's varying output columns to the consumer as globals
> ---------------------------------------------------------------
>
> Key: SPARK-59589
> URL: https://issues.apache.org/jira/browse/SPARK-59589
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Yang Jie
> Priority: Major
> Labels: pull-request-available
>
> {{ExpandExec.doConsume}} allocates a mutable state for each output column
> that varies across the Expand's branches, and hands it to the consumer as
> {{JavaCode.variable(value, dataType)}}.
> {{CodegenContext.addMutableState}} returns a field of the generated class: a
> name of its own when it can inline the type, otherwise a slot in a compacted
> array, e.g. {{mutableStateArray_0[3]}}. Either form is reachable from every
> method of that class, and neither is a local variable, which is what
> {{JavaCode.global}} says and {{JavaCode.variable}} does not.
> The difference is not cosmetic. {{CodeGenerator.getLocalInputVariableValues}}
> collects the {{VariableValue}}s of an operator's input so that a caller can
> pass them into a split function -- used by
> {{AggregateCodegenSupport.splitAggregateExpressions}} and by
> {{CommonExprSlots.fill}} (SPARK-59295). Taking a field for a local variable
> adds a parameter that shadows the field, and for the compacted form the
> parameter name is a slot expression, which does not compile. SPARK-59295 had
> to add an "is this a Java identifier" check on the candidate parameters
> partly for this source.
> Whole-stage subexpression elimination already wraps its own
> {{addMutableState}} results with {{JavaCode.global}} /
> {{JavaCode.isNullGlobal}}
> ({{CodeGenerator.subexpressionEliminationForWholeStageCodegen}}), so this
> makes {{ExpandExec}} consistent with it.
> No query is known to reach the bad parameter today: with a consume function
> per operator (the default) the value is passed as an argument to a fresh
> parameter, and the shapes tried without it -- CUBE over string columns, and a
> regular aggregate over a varying string column under a distinct rewrite -- do
> not put the slot in a parameter list. This is a latent trap rather than an
> observed failure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
