jayzhan211 commented on PR #25428:
URL: https://github.com/apache/datafusion/pull/25428#issuecomment-5814503044
Could you add numbers for the widened path? The fixture shows fewer
rewritten rows, but `external_aggr` (SF1, 4 partitions) swings ~2× between
repeated runs of one build, so timing there shows nothing (Q2 @ 32 MB: base
225–1016 ms, PR 253–732 ms). I also couldn't tell whether it reaches the
widened path.
Please compare base vs PR with `EXPLAIN ANALYZE` on a query that needs ≥2
intermediate passes, e.g. Q2 at SF10 across a few memory limits. Report
`spill_count` / `spilled_bytes` / `spilled_rows`, how many passes were widened,
peak disk usage (the cost widening adds), and output batches per spill file
(the cost of boundary flushes). Add wall clock only for the settings where the
counters differ:
```sh
cargo run --release --bin external_aggr -- benchmark --partitions 4
--iterations 10 \
--path data/tpch_sf10 --query 2 --memory-limit 64M
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
