rluvaton commented on code in PR #15610:
URL: https://github.com/apache/datafusion/pull/15610#discussion_r2041196922
##########
datafusion/common/src/config.rs:
##########
@@ -337,6 +337,13 @@ config_namespace! {
/// batches and merged.
pub sort_in_place_threshold_bytes: usize, default = 1024 * 1024
+ /// When doing external sorting, the maximum number of spilled files to
+ /// read back at once. Those read files in the same merge step will be
sort-
+ /// preserving-merged and re-spilled, and the step will be repeated to
reduce
+ /// the number of spilled files in multiple passes, until a final
sorted run
+ /// can be produced.
+ pub sort_max_spill_merge_degree: usize, default = 16
Review Comment:
@2010YOUY01 and @alamb I hope before you merge this PR to look at
https://github.com/apache/datafusion/pull/15700 to see what I mean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
