I written up a small statistics tool which is targeted at showing if
a benchmark proves anything or not.
critter phk> cat before
29195
29232
29226
29203
29211
critter phk> cat after
29179
29203
29195
29194
29196
critter phk> ./ministat before after
x before
+ after
+--------------------------------------------------------------------------+
|+ +*+ * x x x|
| |___________A_M___|_____|___________M__A_____________________| |
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 29195 29232 29211 29213.4 15.469324
+ 5 29179 29203 29195 29193.4 8.7920419
Difference at 95.0% confidence
-20 ± 18.3497
-0.0684617% ± 0.0628127%
(Student's t, pooled s = 12.5817)
In this case the benchmark shows a very small improvement has obtained
with 95% probability
Here is a more convincing result:
x before
+ after
+--------------------------------------------------------------------------+
| + |
| + |
|x x + |
|x x x + +|
||_________A__M______| |_____M____A_________| |
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 0.132 0.134 0.133 0.1328 0.00083666003
+ 5 0.136 0.138 0.136 0.1364 0.00089442719
Difference at 99.5% confidence
0.0036 ± 0.0024642
2.71084% ± 1.85558%
(Student's t, pooled s = 0.000866025)
The program is a <500 line .c program and I was wondering if it belongs
in the tree so we have an easy to use tool to point people at when they
run benchmarks.
Source: http://phk.freebsd.dk/patch/ministat
Poul-Henning
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
[EMAIL PROTECTED] | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
