On Fri, Apr 12, 2013 at 09:50:36AM -0700, Chiradeep Vittal wrote: > > On 4/12/13 6:55 AM, "Chip Childers" <chip.child...@sungard.com> wrote: > > >On Fri, Apr 12, 2013 at 06:55:56PM +0530, Prasanna Santhanam wrote: > >> On Fri, Apr 12, 2013 at 09:20:39AM -0400, Chip Childers wrote: > >> > On Fri, Apr 12, 2013 at 06:39:29AM +0000, Koushik Das wrote: > >> > > Currently there are no functional tests, did e2e tests manually. > >> > > There are some external setup that is required which then needs to > >>be added to Cloudstack like Nexus 1000v, VNMC, ASA 1000v appliance > >>details. I don't see any existing tests related to Vmware/Nexus stuff > >>which I can reuse, so implementing all these steps would need some time. > >> > > > >> > > So wanted to check if it is ok to merge now and add the functional > >>tests subsequently? > >> > > >> > Before we merge it in, can we first discuss how we want to hand this > >> > sort of thing? It seems that at least a marvin test, not run by > >> > default, would be helpful here. Certainly documentation on how to > >>setup > >> > an environment to support the test is also needed. > >> > > >> > Do other's have thoughts on this? > >> > >> +1 - I can spare some hardware to host virtual instances of the ASA > >> appliance. We do NetScaler VPXs in this fashion. But I'm not sure what > >> kind of pre-programming will be required on the ASA to run the test. > >> So some documentation would certainly help. > > > >Koushik, > > > >I think that it will require working with Prasanna to get a Marvin test > >suite for the feature ready to go. I'd personally like to see that > >*prior* to the merge into master. We need to "up our game" on automated > >testing of new features, so that we stop digging ourselves deeper and > >deeper into a hole. > > > >-chip > > But this approach isn't scalable either. > > Every new whiz bang network device will require infrastructure / licenses > / etc. > > > I think if there's component level tests of the code introduced (at least > the network elements) that should be a good first step. > > -- > Chiradeep
Agreed that it would be a good first step, absolutely. We really do need to think about how we deal with all of these integrations... and how we do ongoing testing. It's going to be a challenge either way. The point of having a marvin test, IMO, isn't necessarily tied to the CI infrastructure. Instead, it allows for a regression test to be run when and if there is an appropriate target infrastructure. I'm sort of breaking the problem into two parts: having tests to run if you have the required setup, and figuring out how we get the required setup. Make sense?
