On 03/26/2015 06:48 PM, Ryan Hsu wrote: > Exclusions are legitimate and will always be necessary at some point. In the > case of the linked bug, this was once a known issue for the VMware driver and > we had excluded affected tests so that CI could continue to run. This is the > same way we do it in Nova CI and oslo.vmware CI.
This is Cinder, Ryan, and Mike is the PTL. It is his decision. Thank you, Anita. > Thanks, > Ryan > > On Mar 26, 2015, at 3:30 PM, Mike Perez <thin...@gmail.com> wrote: > >> On 20:49 Thu 26 Mar , Ryan Hsu wrote: >>> Hi Mike, >>> >>> We (VMware CI) run "testr run tempest.api.volume" for our Cinder CI and this >>> runs about ~240 tests for us. I'm guessing that the rest of the ~60 tests >>> are >>> not being run due to skips and disabled features. For example, here is >>> a sampling of tests that are skipped in a recent run (note that this is >>> using >>> tempest.conf with no explicit disabling of Cinder services): >>> >>> tempest.api.compute.test_live_block_migration.LiveBlockMigrationTestJSON.test_iscsi_volume >>> ... SKIPPED: Block Live migration not available >>> setUpClass >>> (tempest.api.orchestration.stacks.test_volumes.CinderResourcesTest) ... >>> SKIPPED: Heat support is required >>> setUpClass >>> (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV2Test) ... >>> SKIPPED: Cinder multi-backend feature disabled >>> tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume >>> ... SKIPPED: SSH required for this test >>> setUpClass >>> (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV1Test) ... >>> SKIPPED: Cinder backup feature disabled >>> setUpClass >>> (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV2Test) ... >>> SKIPPED: Cinder backup feature disabled >>> setUpClass >>> (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV1Test) ... >>> SKIPPED: Cinder multi-backend feature disabled >>> tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern >>> ... SKIPPED: Skipped until Bug: 1373513 is resolved. >>> tempest.scenario.test_stamp_pattern.TestStampPattern.test_stamp_pattern ... >>> SKIPPED: Skipped until Bug: 1205344 is resolved. >>> setUpClass (tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest) >>> ... SKIPPED: The EC2 API is not available >>> setUpClass (tempest.thirdparty.boto.test_ec2_volumes.EC2VolumesTest) ... >>> SKIPPED: The EC2 API is not available >>> tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern >>> ... SKIPPED: Skipped until Bug: 1373513 is resolved. >>> >>> As we are actually running the volume suite according to the FAQ and the >>> above skipped tests are documented by our CI, would it be possible to add an >>> exception to the rule? I'm sure these numbers will be different for all CIs >>> and as long as people are not abusing and hiding skipped tests, I don't see >>> this as a problem. >> >> There will be no exceptions. Everyone must pass the same tests or you're not >> an >> approved volume driver for OpenStack Cinder. >> >> You should also take this bug [1] Vmware hit as a lesson of doing any >> excluding in your CI. The driver would've been seriously broken in Kilo if >> this >> wasn't caught. >> >> [1] - https://bugs.launchpad.net/cinder/+bug/1436603 >> >> -- >> Mike Perez >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
