naman-modi commented on issue #23658: URL: https://github.com/apache/datafusion/issues/23658#issuecomment-5116028836
@2010YOUY01, I am planning to add a bounded memory pool knob to the fuzzer's `SessionContextGenerator`, so some randomized contexts run under a bounded `FairSpillPool` and hit the spill paths. Baseline stays unbounded for correct assertions. selecting the right "memory sizing" to bound to, is the tricky part; I first tried a fraction of the dataset size, but it did not work as aggregate memory tracks distinct groups, not rows, so any pool that does not instantly exhaust sits well above the real footprint and never spills. So I am experimenting with sizing it off the measured peak instead, which I will get when I run the baseline under a tracking pool, and then force the spill, on the basis of a fraction of this "peak memory". Let me know if you have any opinions/suggestion here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
