Satyr09 opened a new issue, #25992: URL: https://github.com/apache/datafusion/issues/25992
### Describe the bug When a column worker fails during a parallel Parquet write, the writer can continue consuming input before returning the error. The worker closes its input channel, but the dispatcher treats the failed send as normal shutdown and returns success from that operation. The underlying error is reported later, when the row group is finalized or input ends. This delays failure reporting and wastes work. With long-lived input and a large row-group limit, the write can appear stuck even though a column worker has already failed. This also happens when max_row_group_bytes is unset. ### To Reproduce 1. Enable parallel Parquet writing and configure a small memory pool, such as 128 KiB. 2. Use a repeating input stream containing batches of wide strings. 3. Set max_row_group_size to usize::MAX so a row-group boundary is not reached during the test. 4. Write through the public DataFrame write_parquet API. A column worker exceeds its memory limit, but the write continues consuming input instead of promptly returning that error. The [proposed regression test](https://github.com/Satyr09/datafusion/blob/450c5d779f3961f4823f616312d5f5e312c6110a/datafusion/core/tests/parquet/write_errors.rs) reproduces this: it times out on the original writer and passes with the fix. ### Expected behavior When the dispatcher encounters the failed worker’s closed channel, it should return that worker’s original error without waiting for a row-group boundary or the end of input. ### Additional context Discovered while working on [#25041](https://github.com/apache/datafusion/pull/25041), but independent of the byte-target support requested in [#22982](https://github.com/apache/datafusion/issues/22982). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
