In article <kkuep8-nqd....@satorlaser.homedns.org>, Ulrich Eckhardt <ulrich.eckha...@dominolaser.com> wrote:
> Yes, this is surely something that is necessary, in particular since > there are no clear success/failure outputs like for unit tests and they > require a human to interpret them. As much as possible, you want to automate things so no human intervention is required. For example, let's say you have a test which calls foo() and times how long it takes. You've already mentioned that you run it N times and compute some basic (min, max, avg, sd) stats. So far, so good. The next step is to do some kind of regression against past results. Once you've got a bunch of historical data, it should be possible to look at today's numbers and detect any significant change in performance. Much as I loathe the bureaucracy and religious fervor which has grown up around Six Sigma, it does have some good tools. You might want to look into control charts (http://en.wikipedia.org/wiki/Control_chart). You think you've got the test environment under control, do you? Try plotting a month's worth of run times for a particular test on a control chart and see what it shows. Assuming your process really is under control, I would write scripts that did the following kinds of analysis: 1) For a given test, do a linear regression of run time vs date. If the line has any significant positive slope, you want to investigate why. 2) You already mentioned, "I would even wonder if you can't verify the behaviour agains an expected Big-O complexity somehow". Of course you can. Run your test a bunch of times with different input sizes. I would try something like a 1-2-5 progression over several decades (i.e. input sizes of 10, 20, 50, 100, 200, 500, 1000, etc) You will have to figure out what an appropriate range is, and how to generate useful input sets. Now, curve fit your performance numbers to various shape curves and see what correlation coefficient you get. All that being said, in my experience, nothing beats plotting your data and looking at it. -- http://mail.python.org/mailman/listinfo/python-list
