[
https://issues.apache.org/jira/browse/SPARK-59706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-59706.
-----------------------------------
Fix Version/s: 4.4.0
(was: 4.3.0)
Resolution: Fixed
Issue resolved by pull request 58879
[https://github.com/apache/spark/pull/58879]
> Hoist loop-invariant left row out of the AS-OF join inner scan
> --------------------------------------------------------------
>
> Key: SPARK-59706
> URL: https://issues.apache.org/jira/browse/SPARK-59706
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: David Mollitor
> Assignee: David Mollitor
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> h3. What changes were proposed in this pull request?
> In {{SortMergeAsOfJoinScanner}}
> ({{{}sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala{}}}),
> the per-left-row scan over the buffered right group
> ({{{}findBestBackwardForward{}}} and {{{}findBestForwardNearest{}}}) rebinds
> both sides of the shared {{JoinedRow}} on every iteration via
> {{{}joinedRow.withLeft(leftRow).withRight(rightRow){}}}. The left row is
> fixed for the duration of a scan, so this binds
> {{joinedRow.withLeft(leftRow)}} once before the loop and keeps only
> {{joinedRow.withRight(rightRow)}} inside it.
> h3. Why are the changes needed?
> The inner scan runs once per buffered right row for every left row, so
> {{withLeft}} was called redundantly on each iteration with an argument that
> never changes within a scan. JFR profiling of {{AsOfJoinBenchmark}} showed
> {{JoinedRow.withLeft}} / {{withRight}} among the hotter frames in the
> (interpreted) scanner. Binding the left side once removes the redundant
> per-row call.
> This is behavior-preserving: {{JoinedRow.withLeft}} only stores the left-row
> reference, nothing mutates the left side within a scan, and {{findNext}}
> re-binds both sides before projecting the output row.
> h3. Does this PR introduce any user-facing change?
> No.
> h3. How was this patch tested?
> Existing tests pass: {{{}SortMergeAsOfJoinSuite{}}},
> {{{}DataFrameAsOfJoinSuite{}}}, {{{}AsOfJoinSQLSuite{}}}, and
> {{AsOfJoinSortMergeSQLSuite}} (89 tests).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
