I agree with Raja. BVT test case failures should be treated as blockers. Once the test issue is fixed, they might uncover new issues/regressions.
~Rajani On 30-Sep-2015, at 2:32 pm, Raja Pullela <raja.pull...@citrix.com> wrote: > The problem needs to be fixed - "have tested a solution that was discussed > with koushik" - we will have to create PR with that change. Once the PR is > pushed - we will have one less blocker/test failure. > > I agree this is not a cloudstack core code problem - but we should NOT make > that a criteria for priority of a bug - just my take. We should include BVT > failures in the Blocker criteria (whether the fix is in the core > functionality/test script). Otherwise, we will not hit 100% passrate on > BVTs. > > You are right - only a subset of these tests are being run on "travis" > otherwise we will catch more or some of these failures. > > -----Original Message----- > From: Sebastien Goasguen [mailto:run...@gmail.com] > Sent: Wednesday, September 30, 2015 2:09 PM > To: dev@cloudstack.apache.org > Subject: Re: [Blocker] test duplicated in test_scale_vm.py > > >> On Sep 30, 2015, at 10:28 AM, Raja Pullela <raja.pull...@citrix.com> wrote: >> >> Yup... btw, I am on top of this and goal is to get to 100% passrate on BVTs! >> >> > > Yes I got it, and we all have the same goal. > > It does appear that not all smoke tests are run by Travis, otherwise we would > catch any errors on simulator for every PR. > > My point is that if a failure in your BVT is due to the test itself it is not > a cloudstack failure per se and hence not a blocker. Which all said and done > is not a problem because it should be fixed now if I am understanding your > assessment and Koushik report of this failure correctly. > > > >> -----Original Message----- >> From: Sebastien Goasguen [mailto:run...@gmail.com] >> Sent: Wednesday, September 30, 2015 1:48 PM >> To: dev@cloudstack.apache.org >> Subject: Re: [Blocker] test duplicated in test_scale_vm.py >> >> >>> On Sep 30, 2015, at 10:15 AM, Raja Pullela <raja.pull...@citrix.com> wrote: >>> >>> Sebastien, any BVT tests failing is a blocker and needs to be fixed - which >>> is the reason for Blocker. We already have a fix. >>> >> >> Right, but that’s a fix in the test itself, no an issue with the cloudstack >> “core”. >> >> In any case, great, one less blocker. >> >>> Koushik, >>> >>> modified the test "test_01_scale_vm" with required_hardware=false on >>> simulator setup and it works. So, I think we can let the second method go. >>> I will also test this on XS. >>> Raja >>> >>> >>> root@localhost:~/cloudstack# nosetests --with-marvin >>> --marvin-config=/root/cloudstack/setup/dev/local1.cfg >>> --zone=Sandbox-simulator --hypervisor=simulator -a >>> tags=basic,required_hardware=false >>> /root/cloudstack/test/integration/smoke/test_scale_vm.py >>> >>> ==== Marvin Init Started ==== >>> >>> === Marvin Parse Config Successful === >>> >>> === Marvin Setting TestData Successful=== >>> >>> ==== Log Folder Path: /tmp//MarvinLogs//Sep_30_2015_08_06_32_AWNF1O. >>> All logs will be available here ==== >>> >>> === Marvin Init Logging Successful=== >>> >>> ==== Marvin Init Successful ==== >>> ===final results are now copied to: >>> /tmp//MarvinLogs/test_scale_vm_OPS7AD=== >>> root@localhost:~/cloudstack# cd /tmp//MarvinLogs/test_scale_vm_OPS7AD >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls >>> failed_plus_exceptions.txt results.txt runinfo.txt >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# vi results.txt >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls -al total 48 >>> drwxr-xr-x 2 root root 4096 Sep 30 08:07 . >>> drwxr-xr-x 8 root root 4096 Sep 30 08:06 .. >>> -rw-r--r-- 1 root root 0 Sep 30 08:06 failed_plus_exceptions.txt >>> -rw-r--r-- 1 root root 186 Sep 30 08:06 results.txt >>> -rw-r--r-- 1 root root 36164 Sep 30 08:06 runinfo.txt >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# cat results.txt >>> Test scale virtual machine ... === TestName: test_01_scale_vm | >>> Status >>> : SUCCESS === ok >>> >>> --------------------------------------------------------------------- >>> - >>> Ran 1 test in 23.455s >>> >>> OK >>> >>> -----Original Message----- >>> From: Sebastien Goasguen [mailto:run...@gmail.com] >>> Sent: Wednesday, September 30, 2015 1:04 PM >>> To: dev@cloudstack.apache.org >>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py >>> >>> >>>> On Sep 30, 2015, at 9:10 AM, Koushik Das <koushik....@citrix.com> wrote: >>>> >>>> Raja, >>>> The earlier scale vm test couldn't run on simulator because of the >>>> validation steps (IIRC it tries to connect to the real VM created during >>>> test). Since it wasn't possible to run with simulator, the new test was >>>> added to run just on the simulator. Check the required_hardware tag. >>>> >>> >>> Ok so that’s not really a blocker. >>> >>> Can we reduce the severity to major ? >>> Or send a PR that will add this to the Travis run and run the correct test >>> for simulator ? >>> >>> Or make you modifications in your BVT settings, re-run and close the issue ? >>> >>> >>> >>>> -----Original Message----- >>>> From: Raja Pullela [mailto:raja.pull...@citrix.com] >>>> Sent: Wednesday, 30 September 2015 12:12 >>>> To: CloudStack Dev (dev@cloudstack.apache.org) >>>> Subject: [Blocker] test duplicated in test_scale_vm.py >>>> >>>> Hi, >>>> >>>> Filed this blocker on test case "test_scale_vm.py" - >>>> https://issues.apache.org/jira/browse/CLOUDSTACK-8924 >>>> Fixing this will bring Simulator Basic and XS Basic to 100% passrate. >>>> >>>> Best, >>>> Raja >>>> >>> >> >
