dwsmith1983 commented on code in PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#discussion_r4380839104
##########
native/shuffle/src/writers/local/local_partition_writer.rs:
##########
@@ -372,14 +452,45 @@ impl PartitionWriter for LocalPartitionWriter {
// The shuffle output is complete; nothing else encodes through this
context.
self.codec_context.release_zstd();
+ self.sync_buffer_reservation();
Ok(())
}
fn write_burst_complete(&mut self) {
- // A spill burst just ended and the next encode may be a long time
coming; the zstd
- // workspace is native memory no reservation tracks, so don't sit on
it.
+ // A spill burst just ended and the next encode may be a long time
coming, so don't
+ // sit on the zstd workspace.
self.codec_context.release_zstd();
+ self.sync_buffer_reservation();
Review Comment:
`write_burst_complete` now drops the drained encode scratch and syncs the
reservation before it touches the spill buffer, so one wide row no longer
leaves its scratch charged. The scratch regrows on the next encode.
`wide_row_scratch_is_released_after_the_spill` is your case: LZ4, a 512 KiB
pool, one 768 KiB incompressible value, then 128 one-row batches. It expects at
most two spills, where the head spilled 129, and every row back from the output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
