On Tue, Jun 18, 2013 at 4:26 AM, Alexander Gorodnev <agorod...@mirantis.com> wrote: > Hi, > > I have the same problem with Quantum test. I don't even know what it depends > on, but I get the different result every time I run unittests using "tox > -epy27". > My latest result is: > > Ran 4548 (-721) tests in 205.026s (-63.932s) > PASSED (id=8, skips=201) > > As you can see amount of test decreased, but I haven't changed any files. > The best result is: > > Ran 5279 (+14) tests in 290.402s (+4.897s) > > Could it be some perfomance issue? And how can I fix it or try to debug? > > Thanks, > Alexander > > > On Mon, Jun 17, 2013 at 8:42 PM, Roman Bogorodskiy > <rbogorods...@mirantis.com> wrote: >> >> Hi, >> >> On a fresh checkout of quantum with new venv I have tox missing lots of >> tests (exact command is 'tox -epy27'). >> >> It reports: >> >> .... >> >> Ran 2368 tests in 470.017s (+2.755s) >> PASSED (id=10, skips=131) >> >> .... >> >> py27: commands succeeded >> congratulations :) >> >> I noticed that there should be 5000+ tests (e.g. in jenkins logs). >> >> What could be wrong here? I finally got around to looking into this today. I ran `tox -epy27` locally a couple times and noticed that test numbers did vary (but not as wildly as reported earlier). My next step was to source the tox py27 virtualenv then to `testr list-tests | wc -l` to get a rough number for the total number of tests (if tests are dynamically created at run time this number will be low). This gave me 5986 well above the ~5500-5600 that were actually running. At this point I wanted to remove parallel testing from the equation to see if all tests run properly in serial. `testr run` ran 2368 tests so removing parallel testing did not fix the problem.
Running the tests in this way did produce some interesting output in the subunit log [1]. Notice that the NEC agent is returning 0. This kills the test runner prematurely and tells testr that tests passed. This sys.exit(0) shouldn't be in the test path like this. I haven't looked at what fixing this will entail but it is probably possible to catch the SystemExit exception in the test and ignore it. I have a hunch the proper fix will involve making the NEC agent more properly unit testable. I will defer to those with more knowledge of OpenStack Networking for the actual fix. Clark [1] http://paste.openstack.org/show/38858/ _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
