Am 16.11.2011 15:36, schrieb Roy Smith:
It's really, really, really hard to either control for, or accurately measure, things like CPU or network load. There's so much stuff you can't even begin to see. The state of your main memory cache. Disk fragmentation. What I/O is happening directly out of kernel buffers vs having to do a physical disk read. How slow your DNS server is today.
Fortunately, I am in a position where I'm running tests on one system (generic desktop PC) while the system to test is another one, and there both hardware and software is under my control. Since this is rather smallish and embedded, the power and load of the desktop don't play a significant role, the other side is usually the bottleneck. ;)
What I suggest is instrumenting your unit test suite to record not just the pas/fail status of every test, but also the test duration. Stick these into a database as the tests run. Over time, you will accumulate a whole lot of performance data, which you can then start to mine.
I'm not sure. I see unit tests as something that makes sure things run correctly. For performance testing, I have functions to set up and tear down the environment. Then, I found it useful to have separate code to prime a cache, which is something done before each test run, but which is not part of the test run itself. I'm repeating each test run N times, recording the times and calculating maximum, minimum, average and standard deviation. Some of this is similar to unit testing (code to set up/tear down), but other things are too different. Also, sometimes I can vary tests with a factor F, then I would also want to capture the influence of this factor. I would even wonder if you can't verify the behaviour agains an expected Big-O complexity somehow.
All of this is rather general, not specific to my use case, hence my question if there are existing frameworks to facilitate this task. Maybe it's time to create one...
While you're running the tests, gather as much system performance data as you can (output of top, vmstat, etc) and stick that into your database too. You never know when you'll want to refer to the data, so just collect it all and save it forever.
Yes, this is surely something that is necessary, in particular since there are no clear success/failure outputs like for unit tests and they require a human to interpret them.
Cheers! Uli -- http://mail.python.org/mailman/listinfo/python-list
