On 18 September 2012 18:20, David Nalley <da...@gnsa.us> wrote: > On Mon, Sep 17, 2012 at 7:24 PM, Edison Su <edison...@citrix.com> wrote: >> First, thanks for working on test stuff, we really need more automate test >> cases. >> If you plan to work on the API test, we have a python test framework: >> https://cwiki.apache.org/CLOUDSTACK/testing-with-python.html. The most >> compelling feature is that it supports code completion for cloudstack API, >> very easy to write test case in Eclipse environment. And we are setting up a >> test infrastructure, which will execute all the test cases written for this >> framework. >> We already have around 500+ test cases, or maybe more, but apparently it's >> not enough. If you like to work on it, that will be great! >> > > While I like the functional/integration tests, as you note we have > 500+ currently. Those tests also take many hours to run at present. > Unit tests should be something fast, a sanity check, and we only have > 25 right now for the entire code base, so while I like > func/integration tests, I'd really like for our unit tests to be > runnable on every commit, and provide some decent level of coverage so > that we have a bit of confidence that code works well, and provide > good feedback to folks on whether their commit passes the sniff test. > > --David
We can speed up the tests. Just never got to have marvin drive everything asynchronously. It's been on my TODO list forever now as have the API tests. I for one would be extremely happy to see some. About the tool though, here's my opinion: An API test would be a great addition to the marvin suite. Marvin makes it easier to write them too. All you will need is compare a canned response from say listVirtualMachines with what is returned by the CS endpoint. Writing these as python's doctests would make it cleaner and easier to maintain too. Having one tool do all functional and sanity tests is a bonus as well. If you do choose to go the jUnit/java way I suggest you look at jClouds as well. It has a cloudstack-ec2 suite that does API tests already. They will need to be extended and brought to match with the changes done for 4.0. Yichi - I think if you begin testing with all the listXxx API calls and put them down as unit-tests the createXxx, addXxx, deleteXxx will fall in place automatically. HTH
