Marc-Andre Lemburg <m...@egenix.com> added the comment: Kristján Valur Jónsson wrote: >
Thanks for the patch. Here's a quick review (a bit terse, but hope you don't mind)... > The attached patch contains suggested fixes to pybench.py: > 1) add a processtime timer for windows I'm not sure why you added this: the systimes.py module already has a ctypes wrapper for kernel32.GetProcessTimes() ?! All you have to do is use this command line option to enable it: --timer systimes.processtime > 2) fix a bug in timer selection: timer wasn't put in 'self' Thanks for spotting this one. > 3) Collect 'gross' times for each round I'm assuming that "gross" = test time + overhead. Is that right ? I like the idea, but not the implementation :-) Don't like my own pybench statistics implementation much either. I plan to rewrite it for Python 2.7. > 4) show statistics for per-round sums, both for sum of "adjusted" times > for each round, and 'gross' time per round. > > The last one is important. Traditionally, the focus would be on > individual tests. The series of individual timings for each test would > be examined, min and afverage found. These minima and averages would > then be summed up to get a total time. These results are very noisy. > In addition, the sum of individual minima is not a linear operator that > has any meaning and is thus extremely noisy. > > Looking at the minimum and average of the sum is a much stabler > indication of total benchmark performance. I agree. For the minimum times, the minimum over all tests should be used. > Another thing that I found when working with this, is that using > "calibration" significantly adds to noise and can produce incorrect > results. It adds a level of non-linearity to the results that can > appear very strange. Typically the 'overhead' is so low that we should > consider skipping the calibration. The purpose of the benchmark must be > to measure the performance of python in context. The meaning of > individual operations in isolation are pretty meaningless. Although I > didn't change the default number of calibration runs from 10, I would > suggest documenting that it may be useful to turn it to 0 to increase > predictability and get numbers indicative of real performance. The idea behind the calibration is to hide away the overhead of setting up the test. You're right, that nowadays the tests run so fast, that the calibration causes more noise than do good. This is due to the fact that the number of iteration rounds and test "packs" were chosen in 2006. Back then, both Python and the CPUs were a lot slower. OTOH, timers have not gotten a lot more accurate. As a result, the measurements of pybench on todays machines have far more noise than they did in 2006. The only way to resolve this is to adjust all tests - something which I also plan to do in time for Python 2.7. ---------- nosy: +lemburg _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue7029> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
