naman-modi opened a new pull request, #23979: URL: https://github.com/apache/datafusion/pull/23979
## Which issue does this PR close? - Part of #23658. ## Rationale for this change The aggregation fuzzer always runs under the default unbounded memory pool, so it never exercises the aggregate spill paths (including the ordered-aggregation spilling from #23657). ## What changes are included in this PR? - A memory-limit knob in `SessionContextGenerator`. A context is either unbounded (today's behavior) or bounded by a `FairSpillPool` that forces a spill. - The pool is sized from the aggregate's **measured peak, not the dataset size**, because aggregate memory scales with distinct groups, not rows. The baseline runs under a `PeakRecordingPool` to capture that peak, and bounded contexts use a range of fractions (2x, 0.5x, 0.4x, 0.334x) of it. - Bounded contexts use a single partition and a small batch size so the spill fits; a run still too tight returns `ResourcesExhausted` and is skipped and logged, not failed. - The unbounded baseline stays the correctness oracle, so every spilled result is checked against it. ## Are these changes tested? Yes, no regression, and added a new `test_generated_context_spills` asserts a spill actually happens. ## Are there any user-facing changes? No, it's Test-only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
