rluvaton commented on code in PR #22945:
URL: https://github.com/apache/datafusion/pull/22945#discussion_r3412904135
##########
datafusion/physical-plan/src/sorts/multi_level_merge.rs:
##########
@@ -182,7 +191,17 @@ impl MultiLevelMergeBuilder {
async fn create_stream(mut self) -> Result<SendableRecordBatchStream> {
Review Comment:
Can you please try to revert the batch size back to the original value for
the **last** stream if it is possible to stay in the limit?
so if the original very large batch was spread across many rows that are now
in different batches or files we may still keep the batch size as one last try
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
