2010YOUY01 commented on code in PR #13090:
URL: https://github.com/apache/datafusion/pull/13090#discussion_r1816410206
##########
benchmarks/bench.sh:
##########
@@ -524,7 +529,19 @@ run_imdb() {
$CARGO_COMMAND --bin imdb -- benchmark datafusion --iterations 5 --path
"${IMDB_DIR}" --prefer_hash_join "${PREFER_HASH_JOIN}" --format parquet -o
"${RESULTS_FILE}"
}
+# Runs the external aggregation benchmark
+run_external_aggr() {
+ # Use TPC-H SF1 dataset
+ TPCH_DIR="${DATA_DIR}/tpch_sf1"
+ RESULTS_FILE="${RESULTS_DIR}/external_aggr.json"
+ echo "RESULTS_FILE: ${RESULTS_FILE}"
+ echo "Running external aggregation benchmark..."
+ # Only parquet is supported
+ # External aggregation is not stable yet, set partitions to 4 to make sure
Review Comment:
I have updated the comment to be more specific, just to ensure this
benchmark can run successfully on different machines
```
# Since per-operator memory limit is calculated as (total-memory-limit /
# number-of-partitions), and by default `--partitions` is set to number
of
# CPU cores, we set a constant number of partitions to prevent this
# benchmark to fail on some machines.
```
Moreover, now memory-limited queries fail quite easily due to
`record_batch.get_array_memory_size()` over-count the memory usage
https://github.com/apache/datafusion/issues/13089, once it's fixed we can set
default partitions larger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
