Github user runseb commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-150178317
I think you guys agree on the problem but disagree on the solution.
To me 100% success on BVT, if it includes ~20% (or whatever percentage) of
skipped tests is not a real 100% success rate on BVT.
So @miguelaferreira prefers to see 80% success on BVT and vote +1 on a
release, if he actually knows what's happening with the 20% failure.
While @nitt10prashant and @pvr9711 you seem to want 100% success on BVT
even if it includes some tests that are actually not run.
The nicira test is actually the only one in there that's testing a third
party plugin. I don't see any tests for solidfire, netscaler, open contrail,
midokura etcâ¦.All theses tests will fail without the proper hardware/software
setup and hence will fail with simulator run.
If we really want 100% on integration tests (so called BVT) we need to
define a test suite that everyone can run without special hardware or software.
And those tests *cannot* be skipped.
I personally fail to see the benefit of skipped tests versus a failed test.
For this particular tests I propose we move it to a folder
/test/integration/plugins and have our base tests defining "BVT" as the test in
/test/integration/smoke
So -1 on this PR
+1 on a PR that would move this test to a separate folder.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
