alamb opened a new issue, #15358: URL: https://github.com/apache/datafusion/issues/15358
### Is your feature request related to a problem or challenge? Breaking rationale from https://github.com/apache/datafusion/pull/14975#issue-2890626662 into its own ticket: For memory-limit queries, executors might write temporary results into the disk to reduce memory load. It's important to have a configuration option to limit the max disk usage, in case some query would bloat the disk and cause other issues. DuckDB provides a similar configuration: ``` max_temp_directory_size | The maximum amount of data stored inside the 'temp_directory' (when set) (e.g., 1GB) ``` ### Describe the solution you'd like Provide a way to limit the total disk usage of queries that spill to disk ### Describe alternatives you've considered Add a configuration option max_temp_directory_size to disk manager (default to 100GB), if the limit is reached for all spilled files, an error will be returned. ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
