bharath-techie commented on issue #19216:
URL: https://github.com/apache/datafusion/issues/19216#issuecomment-3636780457
> Yes I think that would be a much better approach than trying to spill the
entire 4GB batch
I'm just spilling the topK state and I free up the record batches, so its
quite efficient - then i got to thinking why do i need to spill , I can just
force compact.
```
| Plan with Metrics | SortExec: TopK(fetch=10), expr=[c@1 DESC],
preserve_partitioning=[false],
filter=[c@1 IS NULL OR c@1 > 57340], metrics=[output_rows=10,
elapsed_compute=81.54ms, output_bytes=50.0 MB,
output_batches=1, spill_count=19, spilled_bytes=115.1 MB, spilled_rows=125,
row_replacements=189]
|
```
I'll raise the issue to look more into force compact in TopK.
@alchemist51 please raise the issue for large single allocation in GroupBy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
