This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
The following commit(s) were added to refs/heads/master by this push:
new 9033e9aa0 IMPALA-13781: Fix "Median Diff %" for
report_benchmark_results.py
9033e9aa0 is described below
commit 9033e9aa08dea4f014ffa775a4fd28d721d2eaa3
Author: Joe McDonnell <[email protected]>
AuthorDate: Fri Feb 21 13:23:52 2025 -0800
IMPALA-13781: Fix "Median Diff %" for report_benchmark_results.py
The calculation that report_benchmark_results.py uses for
"Median Diff %" is using the wrong divisor. For an AB test,
it should be dividing by the base median time (i.e. A), but
this is actually dividing by the new median time (i.e. B).
That gives the wrong range and can result in "Median Diff %"
values of >100%. This fixes the calculation to divide by the
base median time (i.e. A).
Change-Id: I6f9da14bc7d5b6ef6a795b9024760f8cae77d72f
Reviewed-on: http://gerrit.cloudera.org:8080/22532
Reviewed-by: Riza Suminto <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
---
tests/benchmark/report_benchmark_results.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/benchmark/report_benchmark_results.py
b/tests/benchmark/report_benchmark_results.py
index e7ddcc123..9df147b64 100755
--- a/tests/benchmark/report_benchmark_results.py
+++ b/tests/benchmark/report_benchmark_results.py
@@ -371,10 +371,10 @@ class Report(object):
self.delta_avg = float('-inf')
self.perf_change_str = ''
else:
- median = results[SORTED][int(len(results[SORTED]) / 2)]
+ base_median = ref_results[SORTED][int(len(results[SORTED]) / 2)]
all_diffs = [x - y for x in results[SORTED] for y in
ref_results[SORTED]]
all_diffs.sort()
- self.median_diff = all_diffs[int(len(all_diffs) / 2)] / median
+ self.median_diff = all_diffs[int(len(all_diffs) / 2)] / base_median
self.perf_change, self.zval, self.tval = (
self.__check_perf_change_significance(results, ref_results))
self.base_avg = ref_results[AVG]
