jizezhang commented on code in PR #19002:
URL: https://github.com/apache/datafusion/pull/19002#discussion_r2621550628
##########
datafusion/physical-plan/src/repartition/mod.rs:
##########
@@ -1438,7 +1445,10 @@ impl PerPartitionStream {
spill_stream: SendableRecordBatchStream,
num_input_partitions: usize,
baseline_metrics: BaselineMetrics,
+ batch_size: Option<usize>,
Review Comment:
Hi @Dandandan , this was mentioned in the PR description, but it was because
`RepartitionExec` has two variants: sort-preserving and non sort-preserving.
For the former, as I understand, merge sort does batching
https://github.com/apache/datafusion/blob/e4dcf0c85611ad0bd291f03a8e03fe56d773eb16/datafusion/physical-plan/src/sorts/merge.rs#L279-L293
Thus I made `batch_coalescer` and `batch_size` optional, e.g. only batch for
non sort-preserving case. But please let me know if this makes sense to you.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
