Hello all! This email is about the way we manage testcases during milestone release testing cycles (also known as ISO testing). If you don't have experience with ISO testing you can read the documentation [1].
*A bit of background* >From Beta milestone, onwards, the release team requires the testcases in the ISO tracker to be covered in a 100% ratio, in order to approve the release. During the Lucid cycle we (Release team & QA) decided to move some of those testcases as "Optional" for release, as they were not that important, and we needed better coverage in the rest. The problem was that most of the optional testcases remained uncovered. During Maverick UDS, we had a session to rethink the testcases and move some of them back to mandatory, and the other way round. Also, the optional testcases were not that optional anymore, they had to be run at least once every milestone (as opposed to mandatory, that need to be run at least once in every image that contains the testcase in every milestone). During the Maverick cycle we have had an obvious problem with the naming. People not aware of that session still thought that the optional testcases were optional. Some others, when they discovered that the optional testcases were not that optional, missed having optional testcases. I think that those optional (really optional) tests have a value. Maybe they are not release critical but they are good to have around, so that people can manage better their testing processes. Some examples: the ARM team likes to have a Netboot testcase, but it is not critical if not covered; the Ubuntu Studio team would like to have some HW specific tests, but still make it to release, if they are not covered; etc. *Proposal* * Create a new category "Run Once" and move there those testcases that need to be run at least once every milestone. (Most of the tests that are now under the "optional" category.) * Use the "Optional" category for those tests that really are optional for release. If they are not covered, or they fail, the release can still go. * The "Mandatory" category would remain as it is. This is a proposal for the Natty release. If the release team agrees with those changes, I could implement them during the first week of the new cycle. If you think that this should be discussed during UDS, let me know. Thanks! Ara. [1] http://qa.ubuntu.com/testing/iso-testing/ -- Ubuntu-qa mailing list Ubuntu-qa@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa
