[
https://issues.apache.org/jira/browse/SPARK-59616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59616:
-----------------------------------
Labels: pull-request-available (was: )
> Avoid copying every candidate row in the AS-OF join best-match scan
> -------------------------------------------------------------------
>
> Key: SPARK-59616
> URL: https://issues.apache.org/jira/browse/SPARK-59616
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: David Mollitor
> Priority: Minor
> Labels: pull-request-available
>
> h2. Problem
> {{SortMergeAsOfJoinScanner
> }}({{{}sql/core/.../execution/joins/SortMergeAsOfJoinExec.scala{}}}) has no
> whole-stage codegen, so its inner right-buffer scan is always interpreted.
> For each left row it scans the buffered right group and materializes the best
> match with {{{}bestMatch = rightRow.copy(){}}}. Under a backward
> (last-match-wins) join this copies *every* as-of-satisfying candidate and
> discards all but the last; forward/nearest copies on each improvement.
> JFR profiling of {{AsOfJoinBenchmark}} (sort-merge cases isolated) showed
> {{UnsafeRow.copy()}} at ~18% of CPU and ~61% of allocation ({{{}UnsafeRow{}}}
> 36% + {{byte[]}} 25%): each {{copy()}} allocates a fresh {{UnsafeRow}} and
> {{{}byte[]{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
