andygrove opened a new issue, #2672: URL: https://github.com/apache/datafusion-ballista/issues/2672
## Describe the issue The pipelined exchange operators (#2670, part of #2652) hold three kinds of buffer outside the task's DataFusion `MemoryPool`: - the per-output queues of `PipelinedShuffleWriterExec`: `ballista.pipelined.queue_capacity` batches per output partition per producer task; - the per-output `BatchCoalescer` of a hash writer, which holds up to a batch of rows per output partition; - the `ReplayBuffer` of a broadcast output, which keeps every batch of the partition until the executor removes the stage or job from its `PipelinedRegistry`. None of these is tracked, so an executor's memory limit does not see them, and a job can exceed it without a spill or an error. The size of a broadcast output relies on the planner's broadcast threshold, which the operator does not enforce. ## Proposed change - Give each output a `MemoryReservation` from the task context: grow it when a batch is queued or pushed into the coalescer and shrink it when the consumer takes the batch. - For a broadcast output, grow the reservation per batch and never shrink it until the partition is removed. A reservation that cannot grow fails the producer, which fails the job, and that gives broadcast outputs the bound they lack today. - The queue reservation needs care: the consumer, not the producer, releases memory, and the two can be on different executors only for the Flight path, where the batch is already out of the producer's memory once sent. ## Context Raised in the review of #2670. Pipelined jobs do not spill, so accounting is the only way an executor can protect itself from a wide pipelined job. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
