On 3 January 2014 11:26, James E. Blair <jebl...@openstack.org> wrote:
> If you are able to do this and benchmark the performance of a cloud > server reliably enough, we might be able to make progress on performance > testing, which has been long desired. The large ops test is (somewhat > accidentally) a performance test, and predictably, it has failed when we > change cloud node provider configurations. A benchmark could make this > test more reliable and other tests more feasible. In bzr we found it much more reliable to do tests that isolate and capture the *effort*, not the time: most [not all] performance issues have both a time and effort domain, and the effort domain is usually correlated with time in a particular environment, but itself approximately constant across environments. For instance, MB sent in a request, or messages on the message bus, or writes to the file system, or queries sent to the DB. So the structure we ended up with - which was quite successful - was: - a cron job based benchmark that ran several versions through functional scenarios and reporting timing data - gating tests that tested effort for operations - a human process whereby someone wanting to put a ratchet on some aspect of performance would write an effort based test or three to capture the status quo, then make it better and update the tests with their improvements. I think this would work well for OpenStack too - and infact we have some things that are in this general direction already. -Rob -- Robert Collins <rbtcoll...@hp.com> Distinguished Technologist HP Converged Cloud _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
