Steve Kargl wrote:
Not if you don't include the standard deviation and/or variance. With only 16 samples, having 2 rather large outlier points will skew the mean.
Seems a bit of special pleading to me. Actually outliers are quite indicative. Often you don't care much if there is 5-10% difference in performance, but the possibility of outliers with really significant performance degradation is more of a problem. If I were them and were really trying to make the best case, I would not use a mean, but a statement like On bla bla benchmarks, our compiler provided performance up to xx% faster than bla bla compiler :-) Quite true, if perhaps misleading! You can complain about actual inaccuracies, but it's a bit useless to complain about accurate things that you think are misleading.
Additionally, they are comparing the compiler against gfortran 4.1.2. There have been a few releases and a few improvements since 4.1.2 was released.
Well that of course is something you can't always help. As long as they identify the version, we can't insist that they keep working to update the results for later versions.
