comphead commented on PR #11218: URL: https://github.com/apache/datafusion/pull/11218#issuecomment-2233641279
@alamb @viirya I removed spilling into subbatches and now spilling the entire batch. Reasoning for that: - in some extreme batch sizes like 1, the problem with subbatches and mask can occur(added the test scenario as well) - even if subbatches makes the reader consume less memory before the spill we still need to get this batch allocated in memory, so not much benefit in this case. - to make more subbatching and less memory approach we need to make SMJ async and work with stream of batches internally, which is much more work involved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
