Hi there Dan and all, after some trying around I have now more clarity regarding what is needed for my test setup and have a more specific question I can't answer from the Marvin doc page [1]
Once again what is the test setup and which goal do I have? If I was the Travis build system checking a release say 14.3.1.0 I would do the following: - visit GitHub repo, clone and checkout sources - proceed with Travis CI specification file .travis.yml, following all the installation scripts and do builds, and setup a test environment, and run integration tests using nose+Marvin Agree? Now I am not Travis; and I have already a test instance of CloudStack to play with. I just want to reuse existing test code and run one or two, or maybe more integration tests against this instance. So I have done so far the following, locally: - visit GitHub repo, clone and checkout sources - read installation scripts following specs in the Travis CI specs file travis.yml and get dependencies which appear mandatory for running the actual integration test code - these dependencies include Java, Maven, Python2.7, pip2 and some further Python2 packages - build Java components, and essentially build Marvin - install Marvin locally following the docs [1] Then, I can run a single integration test using code from [2], and not unexpectedly I get then an API connection error in the test log: nosetests --with-xunit --xunit-file=integration-test-results/$suite.xml --with-marvin --marvin-config=setup/dev/advanced.cfg test/integration/$suite.py -s -a tags=advanced,required_hardware=false --zone=ZONE_NAME --hypervisor=HYPERVISOR (where $suite is a test of my choice) Error excerpt: "...Max retries exceeded with url: /client/api?account=admin&command=listUsers&response=json ..." I do understand I need also to configure access to resources like database which goes then to the file "setup/dev/advanced.cfg" What I miss however, where to specify API client configuration, like the URL of the instance and access/secret keys? [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+-+Testing+with+Python#MarvinTestingwithPython-CompilingandPackaging [2] https://github.com/apache/cloudstack/blob/4.13.1.0/tools/travis/script.sh ________________________________________ Von: Daan Hoogland <daan.hoogl...@gmail.com> Gesendet: Donnerstag, 26. November 2020 10:11:20 An: dev Betreff: Re: [DISCUSS] python test framework Thanks PL, I have focussed on a nose2 on python3 implementation so far [1]. If anybody feels like contributing there, be welcome. I is only a PoC yet but looks promising. The total work seems like a moderately large job and pretty straight forward, with the most challenging part the redesign of the callbacks being provided by nose2 in comparison to nose. We still have to test verification through ssh and db connectors but these are so widely used by the larger python community that I'm not too scared of those. I hope that nose2 has a lively community as I see more issues than answers on github [2] serverspec seems like a good model, so I'll give that a look as well @Peter (peter.murysh...@zv.fraunhofer.de) I saw a separate mail [3] by you about your test setup, but couldn't grasp it yet. I hope we can have a ccc one day soon again to discuss in close quarters. [1] https://github.com/apache/cloudstack/pull/4479 [2] https://github.com/nose-devs/nose2/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc [3] https://lists.apache.org/thread.html/r0788da5cdbaaaa0cebf349b12cc5f488621c48e509e880216989cb35%40%3Cdev.cloudstack.apache.org%3E On Mon, 16 Nov 2020, 15:03 Pierre-Luc Dion, <pdion...@apache.org> wrote: > It would be interesting if we could come up with a catalogue of api tests, > as serverspec [1] did for infrastructure tests. > if we would have some kind of library of tests, then the only things we > would define in our tests suite would be the order of > unit tests and pretty much no code. > > I don't know if there is an equivalent of serverspec framework on Python, > serverspec is the predecessor to Chef-inspec, unit tests for infra. > This could be a reference starting point. > > Via nose2 seems to make sense. > > [1] https://serverspec.org/ > > > On Mon, Nov 16, 2020 at 5:34 AM <peter.murysh...@zv.fraunhofer.de> wrote: > > > Hi there Dan, > > > > we work to setup the test environment and will share relevant output > > fragments with the limitation we can't share complete logs publicly. > > > > kind regards > > Peter > > ________________________________________ > > Von: Daan Hoogland <daan.hoogl...@gmail.com> > > Gesendet: Montag, 16. November 2020 11:12:55 > > An: dev > > Betreff: Re: [DISCUSS] python test framework > > > > sounds like I am on my own. Any body cares to chime in? > > > > On Thu, Nov 12, 2020 at 10:21 PM Nicolas Vazquez < > > nicolas.vazq...@shapeblue.com> wrote: > > > > > LGTM > > > > > > > > > Regards, > > > > > > Nicolas Vazquez > > > > > > ________________________________ > > > From: Daan Hoogland <daan.hoogl...@gmail.com> > > > Sent: Thursday, November 12, 2020 1:05 PM > > > To: dev <dev@cloudstack.apache.org> > > > Subject: Re: [DISCUSS] python test framework > > > > > > I checked the source code and the last commit is from the 3rd of March. > > > nose2 might require some effort from the community @here as well. > > > > > > On Thu, Nov 12, 2020 at 12:58 PM David Jumani < > > david.jum...@shapeblue.com> > > > wrote: > > > > > > > ++1 and like Boris said, need to look into backward compatibility > > > > ________________________________ > > > > From: Boris Stoyanov <boris.stoya...@shapeblue.com> > > > > Sent: Thursday, November 12, 2020 2:26 PM > > > > To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> > > > > Subject: Re: [DISCUSS] python test framework > > > > > > > > +1 on nose2, as the marvin plugin is built on nose, it seems the > > natural > > > > choice to upgrade to. We need to investigate backwards compatibility > > > > though.. > > > > > > > > Bobby. > > > > > > > > On 11.11.20, 21:28, "Daan Hoogland" <daan.hoogl...@shapeblue.com> > > > wrote: > > > > > > > > devs, > > > > Investigating an new organization of test categories , I found > that > > > > our framework in use, nose, is no longer in maintenance. Its > maintainer > > > has > > > > stopped his support in 2017. As it is quite stable it has slipped our > > > > attention, but it is urgent that we discuss and choose a way forward. > > > There > > > > is a successor, nose2, which as the documentation tells me, has less > > > > support for underlying test types but is also extensible. This is my > > goto > > > > choice to investigate further the coming days, but I’d like input > from > > of > > > > you as many as possible. > > > > Regards, > > > > > > > > daan.hoogl...@shapeblue.com > > > > www.shapeblue.com<http://www.shapeblue.com> > > > > 3 London Bridge Street, 3rd floor, News Building, London SE1 > > 9SGUK > > > > @shapeblue > > > > > > > > > > > > > > > > > > > > > > > > boris.stoya...@shapeblue.com > > > > www.shapeblue.com<http://www.shapeblue.com> > > > > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > > > > @shapeblue > > > > > > > > > > > > > > > > > > > > david.jum...@shapeblue.com > > > > www.shapeblue.com<http://www.shapeblue.com> > > > > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > > > > @shapeblue > > > > > > > > > > > > > > > > > > > > > > -- > > > Daan > > > > > > nicolas.vazq...@shapeblue.com > > > www.shapeblue.com > > > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > > > @shapeblue > > > > > > > > > > > > > > > > -- > > Daan > > >
