ranflarion commented on code in PR #24820:
URL: https://github.com/apache/datafusion/pull/24820#discussion_r3986304462
##########
datafusion/physical-plan/src/joins/nested_loop_join.rs:
##########
@@ -1014,8 +1013,16 @@ impl EmbeddedProjection for NestedLoopJoinExec {
/// Left (build-side) data
pub(crate) struct JoinLeftData {
- /// Build-side data collected to single batch
- batch: RecordBatch,
+ /// Build-side data as bounded chunks, in input order. Kept as chunks
rather than one
+ /// `concat_batches` result so buffering never needs input and output to
coexist, and a
+ /// chunk that already arrived at target size is retained without being
copied at all.
+ chunks: Vec<RecordBatch>,
+ /// Row index of the first row of each chunk, i.e. prefix sums over the
chunk lengths.
+ /// The visited-left bitmap is indexed by these global row numbers.
+ row_offsets: Vec<usize>,
+ total_rows: usize,
+ /// Build-side schema, kept so an empty chunk list still knows its shape
+ schema: SchemaRef,
Review Comment:
the operator now asks JoinLeftData::range(start, max_len) for the next run
of rows and gets back a ChunkRange with the batch to address, the local start,
the global start for the bitmap and the length, already clamped at the chunk
end, so the translation and the clamp live in one place and the probe and
unmatched-left paths just consume ranges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
