1. We need to change the 75% test criteria for all test. That just makes
thing more complicated. Can we copy the tests we have an rewrite them so that
we don't need the criteria? Then we disable the first test and run them when
the bug is fixed upstream.
2. I think we should add a small check in the meta data of the VM is it
got a ip. Greping for it or something similar. Or checking if it has at least a
route to send the ping on. If it doesn't then we fail in any case. For a /24 ip
to ping it would be something like:
a. [[ $(route -n|awk '{print $1}') == *"192.168.122.0"* ]];
BR Nikolas
From: Skiadas Romanos [mailto:r...@intracom-telecom.com]
Sent: Friday, December 02, 2016 4:04 PM
To: BV Guravareddy
Cc: Nikolas Hermanns
Subject: RE: test Case fail criteria
The problem is that a VM not getting IP is not our issue per se. It causes us
problems, but what we test is VPN connectivity.
We could have an assertion before each test that all the VMs must get an IP and
then fail all of them, but then we fail to check if the VMs that got IPs had
the correct VPN connectivity.
We could fail all the tests the specific VM participates in.
Eg. Vm1 doesn't get an IP, we should fail:
- Vm1 can ping vm2
- Vm1 cannot ping vm2
But not vm2 can ping vm2.
I think the first would be the easiest thing to implement but I don't know
which is the saner thing to do. I've cc'd Niko so he can have a say in this.
In most tests we have 75% success criteria because isolation doesn't work. So
one flipped fail and one flipped pass would still give the same success %,
causing the overall tests to pass.
BR,
Romanos
From: BV Guravareddy [mailto:bv.guravare...@ericsson.com]
Sent: Friday, December 02, 2016 1:43 PM
To: Skiadas Romanos
Subject: test Case fail criteria
Hi Rom,
I have seen that some test cases are not having proper check for pass/fail
Criteria. Please see the examples below.
1. DHCP is not working, so instances will not get an IP address.
2. Ping of VM-1 to VM-2 will not be success.
3. But scenario like VM-1 should not be ping-able from VM-2 will PASS.
Without even having an IP address. My suggestion is to Fail ALL test cases if
VM is not having an IP address.
I am sure, we have success criteria of 100%. So it should not be an issue.
Please feel free to suggest.
Thanks & Regards,
Venkata Guravareddy.
_______________________________________________
opnfv-tech-discuss mailing list
opnfv-tech-discuss@lists.opnfv.org
https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
