ranflarion commented on code in PR #24820:
URL: https://github.com/apache/datafusion/pull/24820#discussion_r3986308901
##########
datafusion/physical-plan/src/joins/nested_loop_join.rs:
##########
@@ -1071,10 +1114,14 @@ async fn collect_left_input(
with_visited_left_side: bool,
probe_threads_count: usize,
spill_manager: Option<SpillManager>,
+ target_batch_size: usize,
) -> Result<LeftLoad> {
let schema = stream.schema();
let metrics = join_metrics;
- let mut batches: Vec<RecordBatch> = Vec::new();
+ let mut chunks: Vec<RecordBatch> = Vec::new();
+ // Batches at or above half the target size pass through without being
copied.
+ let mut coalescer = BatchCoalescer::new(Arc::clone(&schema),
target_batch_size)
Review Comment:
The convention does not hold at this operator's build side. coalesce_batches
wraps only FilterExec, HashJoinExec and RepartitionExec, so aggregates, unions,
memory tables and row-group tails arrive at whatever size produced them, and
MemorySourceConfig forwards stored batches as they are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
