David Mollitor created SPARK-59706:
--------------------------------------
Summary: Hoist loop-invariant left row out of the AS-OF join inner
scan
Key: SPARK-59706
URL: https://issues.apache.org/jira/browse/SPARK-59706
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
Fix For: 4.3.0
h3. What changes were proposed in this pull request?
In {{SortMergeAsOfJoinScanner}}
({{{}sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala{}}}),
the per-left-row scan over the buffered right group
({{{}findBestBackwardForward{}}} and {{{}findBestForwardNearest{}}}) rebinds
both sides of the shared {{JoinedRow}} on every iteration via
{{{}joinedRow.withLeft(leftRow).withRight(rightRow){}}}. The left row is fixed
for the duration of a scan, so this binds {{joinedRow.withLeft(leftRow)}} once
before the loop and keeps only {{joinedRow.withRight(rightRow)}} inside it.
h3. Why are the changes needed?
The inner scan runs once per buffered right row for every left row, so
{{withLeft}} was called redundantly on each iteration with an argument that
never changes within a scan. JFR profiling of {{AsOfJoinBenchmark}} showed
{{JoinedRow.withLeft}} / {{withRight}} among the hotter frames in the
(interpreted) scanner. Binding the left side once removes the redundant per-row
call.
This is behavior-preserving: {{JoinedRow.withLeft}} only stores the left-row
reference, nothing mutates the left side within a scan, and {{findNext}}
re-binds both sides before projecting the output row.
h3. Does this PR introduce any user-facing change?
No.
h3. How was this patch tested?
Existing tests pass: {{{}SortMergeAsOfJoinSuite{}}},
{{{}DataFrameAsOfJoinSuite{}}}, {{{}AsOfJoinSQLSuite{}}}, and
{{AsOfJoinSortMergeSQLSuite}} (89 tests).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
