wirybeaver commented on PR #2749: URL: https://github.com/apache/datafusion-ballista/pull/2749#issuecomment-6592398141
<details> <summary>Benchmark results — TPC-H SF10</summary> <!-- task2-tpch-sf10-results --> ## Experiment A = runtime filtering disabled; B = runtime filtering enabled. The same Task 2 binaries and controls are used within each comparison, with immediate probe dispatch in staged AQE execution. Repository TPC-H queries at SF10; Q16 is omitted as in CI. Each query was warmed and verified against a local DataFusion reference in both arms before timing. Visits are interleaved in `A → B → B → A → A → B` or `B → A → A → B → B → A` order, shown verbatim below. Mean Δ is `mean(B) / mean(A) - 1`; median Δ is `median(B) / median(A) - 1`, calculated from unrounded observations. Negative values mean lower latency. ## Key Results Q19 has the clearest benefit: **324.4 → 284.7 ms (−12.23% median, −13.30% mean)**. Q3 is **618.9 → 610.1 ms (−1.42% median, −1.23% mean)**; Q7 is **881.1 → 893.0 ms (+1.35% median, +2.08% mean)**. | Query | A median (ms) | B median (ms) | Mean Δ | Median Δ | Visits in executed order (ms) | |---|---:|---:|---:|---:|---| | Q1 | 530.6 | 531.4 | +0.18% | +0.15% | A 530.555 → B 531.687 → B 531.356 → A 527.948 → A 532.544 → B 530.931 | | Q2 | 151.7 | 147.5 | -3.50% | -2.78% | A 160.774 → B 146.626 → B 151.042 → A 151.730 → A 148.813 → B 147.506 | | Q3 | 618.9 | 610.1 | -1.23% | -1.42% | A 642.222 → B 626.470 → B 589.702 → A 649.065 → A 586.613 → B 584.917<br>B 610.065 → A 585.423 → A 618.878 → B 636.385 → B 636.746 → A 588.001<br>A 647.010 → B 589.665 → B 623.996 → A 611.693 → A 647.829 → B 609.961 | | Q4 | 273.2 | 272.5 | +2.50% | -0.24% | A 273.169 → B 298.036 → B 272.501 → A 281.223 → A 267.017 → B 271.396 | | Q5 | 419.1 | 418.6 | -0.37% | -0.13% | A 420.182 → B 407.464 → B 421.763 → A 413.152 → A 419.102 → B 418.553 | | Q6 | 178.3 | 179.9 | +0.67% | +0.87% | A 186.796 → B 183.999 → B 179.857 → A 172.580 → A 178.305 → B 177.421 | | Q7 | 881.1 | 893.0 | +2.08% | +1.35% | A 881.126 → B 885.616 → B 911.934 → A 875.887 → A 849.975 → B 898.463<br>B 924.255 → A 886.350 → A 895.543 → B 907.755 → B 892.990 → A 862.142<br>A 894.875 → B 890.742 → B 892.832 → A 844.851 → A 909.831 → B 860.631 | | Q8 | 446.6 | 443.0 | -0.50% | -0.81% | A 446.632 → B 442.992 → B 451.127 → A 450.370 → A 444.778 → B 440.896 | | Q9 | 937.8 | 938.4 | +0.29% | +0.07% | A 934.998 → B 961.816 → B 938.380 → A 953.712 → A 937.765 → B 934.406 | | Q10 | 641.5 | 633.6 | -1.51% | -1.23% | A 641.451 → B 626.121 → B 633.554 → A 659.835 → A 631.657 → B 644.139 | | Q11 | 73.6 | 74.3 | -1.28% | +0.90% | A 80.105 → B 74.100 → B 74.270 → A 73.604 → A 73.446 → B 75.888 | | Q12 | 421.7 | 406.0 | -2.85% | -3.73% | A 422.443 → B 404.143 → B 405.951 → A 410.384 → A 421.660 → B 408.690 | | Q13 | 432.5 | 427.3 | -1.03% | -1.22% | A 433.997 → B 427.291 → B 430.753 → A 426.766 → A 432.550 → B 421.995 | | Q14 | 211.7 | 208.3 | -0.21% | -1.60% | A 211.724 → B 208.343 → B 212.170 → A 211.805 → A 203.414 → B 205.094 | | Q15 | 315.4 | 310.6 | -1.27% | -1.54% | A 315.446 → B 305.379 → B 317.486 → A 312.110 → A 317.887 → B 310.580 | | Q17 | 1298.5 | 1288.2 | -4.12% | -0.79% | A 1457.564 → B 1312.296 → B 1270.810 → A 1281.470 → A 1298.540 → B 1288.218 | | Q18 | 2073.6 | 2077.7 | -0.84% | +0.20% | A 2164.787 → B 2077.653 → B 2084.203 → A 2073.591 → A 1992.074 → B 2016.289 | | Q19 | 324.4 | 284.7 | -13.30% | -12.23% | A 335.749 → B 288.994 → B 278.826 → A 323.190 → A 324.404 → B 284.744 | | Q20 | 442.8 | 440.6 | -0.74% | -0.50% | A 442.769 → B 435.798 → B 449.921 → A 436.172 → A 457.176 → B 440.556 | | Q21 | 1620.2 | 1605.9 | -1.22% | -0.88% | A 1620.215 → B 1611.336 → B 1605.885 → A 1612.819 → A 1641.135 → B 1597.505 | | Q22 | 173.7 | 168.2 | -1.85% | -3.19% | A 174.708 → B 166.875 → B 168.194 → A 173.740 → A 169.964 → B 173.749 | ## Conclusion Q19 retains approximately 59.60 million pruned rows. Q2 opportunistically prunes about 42,300 rows in some visits; other queries have no observed row pruning. Small timing changes without pruning should not be counted as selectivity gains. Stage counts are unchanged between arms. These results support a useful Q19 improvement, while Q3 is approximately flat and Q7 retains a small observed cost; they do not establish a uniform suite-wide speedup. ### Environment and reproducibility - Apple Silicon Mac, 10 reported CPUs and 24 GiB RAM; one local scheduler and two local executors, each with 4 vcores and a 4 GiB pool. Queries run sequentially over loopback. - Fresh `release-nonlto` build of the [measured PR revision](https://github.com/wirybeaver/datafusion-ballista/commit/e33c64db6dfeefd8d200a6816a9418c652fa67f5). SF10 Parquet generated with `tpchgen-cli 4.0.0`, 16 parts, Snappy, 128 MiB target row groups; warm cache. - 16 query partitions, batch size 8192, adaptive planning and hash joins enabled; DataFusion local dynamic-filter pushdown, coalescing and skew-join optimization disabled. Only `ballista.planner.runtime_filter.enabled` differs within a comparison. - Latency includes planning, execution and complete result fetch; client/session registration, reference validation, service startup and metrics collection are outside the reported timer. - Host activity is sampled every 0.5 s. External build/benchmark workloads or sustained desktop load invalidate a whole query group, which is rerun. Accepted timing groups had no detected external build/benchmark workloads; short desktop CPU peaks permitted by the policy remain. - Pruned-row counters are postprocessed to align metrics-producing nodes with collector slots (`metrics=None` nodes otherwise offset the current stage renderer). Timings and binaries are unchanged by this observation correction; displayed row counts can be rounded. - This is an exploratory single-host result. Small latency differences remain variable and do not establish multi-host throughput gains. No pooled suite score is inferred from the per-query table. </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
