I had a debate with my team about how can I be sure the strategy I want to use to optimize my cluster with Watcher works well. Also, how a developer can guarantee its strategy works well?
After a lot of discussions, we found a consensus: - For the unit tests, a developer should be able to demonstrate that its strategy algorithm is correct, by mocking all data required by this algorithm (Cluster data model, metrics, scoring data, ...). A good test coverage is required to be confident about a given strategy. If you implement a deterministic algorithm, unit tests should be able to thoroughly check the algorithm outputs. - For the integration tests (Tempest), a developer/tester should be able to check that the strategy has been correctly implemented, according to the plugin implementation documentation available within the Watcher developer guide[0]. One should at least implement a Tempest scenario test that creates a executes a new audit with this strategy (and related goal). Note that with tempest gate job, there is not a single measurement being collected, so one should take into account such a case. Finally, we added a new section in the Watcher documentation to promote each strategy. This piece of documentation is very important, hence strategy developers should make sure they describe both how the strategy works and the goal it achieves. A complete documentation, with references, will make end users more confident about using it. Feel free to comment my post :) [0]: http://docs.openstack.org/developer/watcher/ BR, David __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
