andygrove commented on issue #15323: URL: https://github.com/apache/datafusion/issues/15323#issuecomment-2743795290
> Do you see too many threads when writing the spill files or when reading? This is when reading, during the merge operation. > In merge phase, each spill file will be wrapped by a stream backed by a blocking thread (see [read_spill_as_stream](https://github.com/apache/datafusion/blob/46.0.1/datafusion/physical-plan/src/spill.rs#L44-L55)), so we'll spawn at least 183 blocking threads when there are 183 spill files to merge spilled data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
