2010YOUY01 commented on code in PR #14823:
URL: https://github.com/apache/datafusion/pull/14823#discussion_r1966496849
##########
datafusion/physical-plan/src/sorts/sort.rs:
##########
@@ -446,21 +511,9 @@ impl ExternalSorter {
None => {
let sorted_size =
get_reserved_byte_for_record_batch(&batch);
if self.reservation.try_grow(sorted_size).is_err() {
- // Directly write in_mem_batches as well as all the
remaining batches in
- // sorted_stream to disk. Further batches fetched from
`sorted_stream` will
- // be handled by the `Some(writer)` matching arm.
- let spill_file =
-
self.runtime.disk_manager.create_tmp_file("Sorting")?;
- let mut writer = IPCWriter::new(spill_file.path(),
&self.schema)?;
- // Flush everything in memory to the spill file
- for batch in self.in_mem_batches.drain(..) {
- writer.write(&batch)?;
- }
- // as well as the newly sorted batch
- writer.write(&batch)?;
- spill_writer = Some(writer);
+ self.in_mem_batches.push(batch);
+ self.spill().await?;
Review Comment:
This refactor is not related to the PR, I did this along the way.
##########
datafusion/physical-plan/src/sorts/sort.rs:
##########
@@ -1425,7 +1478,7 @@ mod tests {
// Processing 840 KB of data using 400 KB of memory requires at least
2 spills
// It will spill roughly 18000 rows and 800 KBytes.
// We leave a little wiggle room for the actual numbers.
- assert!((2..=10).contains(&spill_count));
+ assert!((12..=18).contains(&spill_count));
Review Comment:
This is caused by the above refactor, the old implementation forget to
update the statistics, so we missed several counts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
