milenkovicm commented on issue #17334: URL: https://github.com/apache/datafusion/issues/17334#issuecomment-3393054846
FWIW, we take an optimistic approach; we do not limit memory for non-spillable operators. Getting numbers right when non-spillable operators are accounted for was impossible for us. Actually, we do track the memory usage of them, but we let it grow unbounded. The other reason is that we can’t do much with them; from our experience, they freak out a bit too early before a spill can occur. Also, this makes the estimation of overall process memory usage much simpler, as after a few empirical runs the total memory needed is whatever spill memory is set, plus whatever is measured for non-spillable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
