New submission from Kristján Valur Jónsson <krist...@ccpgames.com>:
The attached patch contains suggested fixes to pybench.py: 1) add a processtime timer for windows 2) fix a bug in timer selection: timer wasn't put in 'self' 3) Collect 'gross' times for each round 4) show statistics for per-round sums, both for sum of "adjusted" times for each round, and 'gross' time per round. The last one is important. Traditionally, the focus would be on individual tests. The series of individual timings for each test would be examined, min and afverage found. These minima and averages would then be summed up to get a total time. These results are very noisy. In addition, the sum of individual minima is not a linear operator that has any meaning and is thus extremely noisy. Looking at the minimum and average of the sum is a much stabler indication of total benchmark performance. Another thing that I found when working with this, is that using "calibration" significantly adds to noise and can produce incorrect results. It adds a level of non-linearity to the results that can appear very strange. Typically the 'overhead' is so low that we should consider skipping the calibration. The purpose of the benchmark must be to measure the performance of python in context. The meaning of individual operations in isolation are pretty meaningless. Although I didn't change the default number of calibration runs from 10, I would suggest documenting that it may be useful to turn it to 0 to increase predictability and get numbers indicative of real performance. ---------- components: Demos and Tools files: pybench.patch keywords: patch messages: 93416 nosy: krisvale severity: normal status: open title: Improve pybench type: behavior versions: Python 2.7 Added file: http://bugs.python.org/file15016/pybench.patch _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue7029> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
