andygrove opened a new pull request, #2802: URL: https://github.com/apache/datafusion-ballista/pull/2802
# Which issue does this PR close? No issue. Split out of #2792. It is a prerequisite for making zstd the default shuffle codec. # Rationale for this change The sort shuffle spill manager keeps one Arrow IPC `StreamWriter` open for every output partition it has spilled, until the input partition finishes. Each open writer holds a file descriptor and its own compression context, and the contexts are not tracked by the memory pool. With LZ4 the contexts are small. With zstd each one is about 0.3–0.7 MB. Once an input spills, it usually spills every output partition, so a task with M local inputs and K output partitions holds up to M × K writers at once. At 16 inputs and 256 partitions that is 4,096 open files per task and about 2.9 GB of untracked zstd contexts per executor. # What changes are included in this PR? - `SpillManager::spill_batches` writes one spill event's batches for a partition as one complete IPC stream, appended to that partition's spill file, and closes the writer before returning. A spill file therefore holds one stream per spill event. The consolidated output already concatenates several streams per partition range, and the readers (`MultiStreamPartitionStream` and the remote fetch path) already read across them, so the output file format is unchanged. - `spill` stays, as a one-batch wrapper. `finish_writers` is removed, since no writer outlives a call. `open_spill_reader` is removed too: it read only the first stream, and nothing outside the tests used it. - New test `multiple_spilling_input_partitions_round_trip`: several input partitions each spill repeatedly and keep an in-memory remainder, and every row must come back exactly once with its value. The existing multi-input test never spilled, so nothing checked how each output partition's range stitches spill and in-memory bytes from every input together. On TPC-H SF100 (2 executors × 8 cores, 16 partitions, zstd) the suite total is unchanged: 132.8 s against 132.3 s. # Are there any user-facing changes? No behavior changes. `SpillManager` is public, and `finish_writers` and `open_spill_reader` are removed from it, so this has the `api-change` label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
