[
https://issues.apache.org/jira/browse/SPARK-58209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118390#comment-18118390
]
Anshul Baliga commented on SPARK-58209:
---------------------------------------
Opened [apache/spark#58999|https://github.com/apache/spark/pull/58999] to
supersede the stalled #57371. Per the review on that PR, only
CallMethodViaReflection needs the stateful flag: XPathExtract is
RuntimeReplaceable (its state lives in a Literal-captured evaluator, out of
reach of the fresh-copy mechanism, as with SPARK-58205), and
UserDefinedGenerator never reaches a fresh-copy site.
> Mark XPathExtract, CallMethodViaReflection, and UserDefinedGenerator stateful
> to prevent executor-side data races
> -----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58209
> URL: https://issues.apache.org/jira/browse/SPARK-58209
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.5.0, 4.1.0, 4.0.0, 4.2.0, 4.3.0
> Reporter: Vinod KC
> Priority: Major
> Labels: pull-request-available
>
> Three Catalyst expressions `XPathExtract` `CallMethodViaReflection` ,
> `UserDefinedGenerator` maintain mutable state internally but do not override
> stateful (which defaults to false).
> As a result, freshCopyIfContainsStatefulExpression() never makes independent
> copies of them. When two executor tasks share the same expression instance —
> which can happen when sibling QueryExecutions are derived from the same base
> DataFrame — they race on the shared mutable fields and silently produce
> incorrect results.
>
> Need to override def stateful: Boolean = true to XPathExtract,
> CallMethodViaReflection, and UserDefinedGenerator. This will cause
> freshCopyIfContainsStatefulExpression() to return a fresh instance per
> concurrent task, giving each its own independent evaluator, buffer, and lazy
> vars.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
