Here, we mean test setup, not marvin setup ( or setup.py ) . Each test has their own dependency, we are having setup,flow,teardown for test module. By below content of bug description, we mean verifying the test dependencies under setup class of a test feature or module not the marvin setup. The test feature or module setup can take care to verify these dependencies and exit with better log and reporting as against reporting the test cases as failures.
Some dependencies at marvin setup can be verified, but the below purpose is to ensure test dependencies are met before running tests. The goal is to make less or zero % test script errors, against genuine product bugs as failures. Santhosh ________________________________________ From: Prasanna Santhanam [t...@apache.org] Sent: Monday, October 28, 2013 2:15 AM To: dev@cloudstack.apache.org Cc: iss...@cloudstack.apache.org Subject: [DISCUSS] (CLOUDSTACK-4971) Verifying Test Beds : prerequisites for launching tests I like the concept of a health check before launching tests. But it's a difficult distinction to make whether to run a specific test because its required hardware component/external dependency is not available/not ready in the deployment OR have the test ensure that the deployment is suitable for its run. So our current health check only ensures two things - SystemVMs are up and running - Default Templates are 'Ready' in the system Take the example of LDAP here that you quoted. In such a case I'd prefer that the test itself check for the deployment guarantees without pushing that to the framework or the deployment scripts to decipher. There is a counter case - some tests use the netaddr module to do subnet arithmetic. For these to run we need to ensure the virtualenv under which the test module runs contains netaddr. This is something that can be ensured by the deployment scripts. So this is a question of guidelines to test authors on what to expect and what not to expect when writing a test. Also how to have the dependencies added automatically before the test gets executed. I don't see a one-size-fits-all solution. On Mon, Oct 28, 2013 at 05:49:39AM +0000, Santhosh Kumar Edukulla (JIRA) wrote: > Santhosh Kumar Edukulla created CLOUDSTACK-4971: > > 1, Currently test scripts are running despite the test setup has > issues. EX: In one of the test scenario, where ldap server was down, > then test suite depending upon them was still running. > > 2, Ideally, the setup should check for these dependencies before > running the test script and exit wherever possible with proper log > and reporting . > > 3, We may not be able to cover all setup issues, but the purpose of > setup is also to check for its existence and dependencies. If > scripts depending upon particular server conditions, should have a > proper setup verifying the server existence before proceeding > further. > > 4, I believe running test cases when setup is down is also not > idealistic. > > 5. Purpose is to reduce script errors vs failures due to product > bugs. > > 6. Logging this bug to track the setup issue cases or script > failures. > -- Prasanna., ------------------------ Powered by BigRock.com
