Thanks Edison for the proposal, this is something we really need. Now that CloudStack is a lot feature rich we want to make it easier for developers to build components and contribute stuff at the same time make sure different parts tie up without losing integrity.
On 07-Jan-2013, at 2:11 PM, Edison Su <edison...@citrix.com> wrote: > Hi All, > As Cloudstack is becoming more modularized, we can start thinking about > how to test these components. > The organization of maven projects in cloudstack. At lowest level, it's a > maven project. The modularized component, means a group of maven projects. > The subsystem, means a logical group of components. A system, will be build > on a lot of subsystems. For example, there will be a system called, > cloud-engine, which is a server, provides restful API service to other > systems(e.g API server). Cloud-engine will have a set of subsystems: > computing, storage, network etc. For each subsystem, will have a group of > components: e.g. storage subsystem will include volume/image/snapshot .. > components. For each component, will have a set of maven projects, for > example, volume component will have a maven project, called storage-volume, > and a lot of plugin projects, such as (storage-volume-plugin-solidfire, > storage-volume-plugin-netapp etc). > Based on above hierarchy, the integration test will happen at different > levels: at component level, at subsystem level, at system level etc. I want > to bring up the discuss about how to organize integration tests, and which > integration test framework should we use. > First, how to organize integration tests? The organization will be > affected/constrained by the dependency between maven projects you want to > test. > 1. Will the integration test at each level have its own maven > project? Or we put all the integration test cases into one giant maven > project? I'd prefer one maven project at one level, as it looks like more > cleaner. For example, there will be a storage-integration-test maven project, > which depends on storage-volume-integration-test, > storage-image-integration-test, and so on, while > storage-volume-integration-test will depend on storage-volume, > storage-volume-plugin-solidfire, and storage-volume-plugin-netapp and so on. > When you write test case at each level, you only focus on the limited area > you want to test: the design of test case and the configuration file of each > test case, will/can only be specific to the area. Hope it will make writing > test case easier. I think we should have integration tests place locally to a maven project, but these tests extend or implement integration tests for that module. For example, we'll have three top level modules, compute, storage and network. So, for storage for example a storage plugin developer should have the plugin as maven project on one level and the integration test along with it. If applicable these tests should extend/implement a common test class/framework/mechanism. It would be better to keep code and test together and would make it easier for a developer to work on things, putting everything in one giant maven project will make it difficult for a new developer to contribute their work. > 2. Which test framework we should use? Junit or testNG? I was sold > to testNG by its feature sets: parametrized test case, group test cases, > running test case in parallel at different level(method, class etc), test > case dependency etc. You can find out more information about testNG at > http://kaczanowscy.pl/tomek/sites/default/files/testng_vs_junit.txt.slidy_.html#(1). While we can have both, I too like testNG (similar to our pythonic integration smoke tests in a lot of ways :), the listener way in testNG can be useful to write your own processing and tricks. JUnit can test very well a particular class, but it's not very good at testing contracts and integrity with other piece of code. Which one should we use: testNG! Regards. > But in order to integrate testNG with Spring, need a little bit coding. > For example, may need to add a listener in testNG, to inject @DB annotation, > and need to make sure mockito will work. I created a base testNG class: > CloudStackTestNGBase, which can read configuration parameters from a testNG > config xml file, setup @DB context for each test case etc. > > Comments/feedback are welcome! > >
