David Mollitor created SPARK-59616:
--------------------------------------
Summary: Avoid copying every candidate row in the AS-OF join
best-match scan
Key: SPARK-59616
URL: https://issues.apache.org/jira/browse/SPARK-59616
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Problem
{{SortMergeAsOfJoinScanner
}}({{{}sql/core/.../execution/joins/SortMergeAsOfJoinExec.scala{}}}) has no
whole-stage codegen, so its inner right-buffer scan is always interpreted. For
each left row it scans the buffered right group and materializes the best match
with {{{}bestMatch = rightRow.copy(){}}}. Under a backward (last-match-wins)
join this copies *every* as-of-satisfying candidate and discards all but the
last; forward/nearest copies on each improvement.
JFR profiling of {{AsOfJoinBenchmark}} (sort-merge cases isolated) showed
{{UnsafeRow.copy()}} at ~18% of CPU and ~61% of allocation ({{{}UnsafeRow{}}}
36% + {{byte[]}} 25%): each {{copy()}} allocates a fresh {{UnsafeRow}} and
{{{}byte[]{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
