hhhizzz commented on issue #23318: URL: https://github.com/apache/datafusion/issues/23318#issuecomment-4889525626
I found a TPC-DS SF10 performance regression on current `main` that appears related to the partially ordered aggregation path. Environment: - DataFusion current main: `6a0e76ed692afaafd616e0d98d51bef63c1ff02d` - Dataset: TPC-DS SF10 Parquet - Query set: 99 queries - Parquet pushdown / filter reorder / pruning: enabled Result summary: | Version | DataFusion | Arrow | Avg full 99-query round | |---|---:|---:|---:| | Previous baseline | `4d898e93f0` | `7b335b7712` | `31090.78 ms` | | Current main | `6a0e76ed69` | `d8d980bdcf` | `32876.21 ms` | Current `main` is about `+1785 ms/round` slower. The main stable contributors are q39 and q78. Per-query comparison: | Query | Before | Current main | Delta | |---|---:|---:|---:| | q39 | ~`1274 ms` | ~`1962 ms` | +`688 ms` | | q78 | ~`655 ms` | ~`1121 ms` | +`466 ms` | Both q39 and q78 contain `AggregateExec ... ordering_mode=PartiallySorted(...)`. The likely cause is that after #23181, these partially sorted aggregate cases are routed through the new ordered partial/final aggregate streams. For these TPC-DS queries, that path appears slower than the previous regular hash aggregation path with early emit. Debug metrics also suggest the extra time is concentrated in the aggregate emitting path. This seems consistent with the point in this issue that partially ordered input should likely still use regular hash aggregation with the early-emit optimization, while fully ordered input may need a separate optimized path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
