On Thu, Sep 30, 2010 at 01:12:58PM +0200, Ara Pulido wrote: > > Hello all! > > This email is about the way we manage testcases during milestone release > testing cycles (also known as ISO testing). If you don't have experience > with ISO testing you can read the documentation [1]. > > *A bit of background* > > From Beta milestone, onwards, the release team requires the testcases in > the ISO tracker to be covered in a 100% ratio, in order to approve the > release. During the Lucid cycle we (Release team & QA) decided to move > some of those testcases as "Optional" for release, as they were not that > important, and we needed better coverage in the rest. > > The problem was that most of the optional testcases remained uncovered. > During Maverick UDS, we had a session to rethink the testcases and move > some of them back to mandatory, and the other way round. Also, the > optional testcases were not that optional anymore, they had to be run at > least once every milestone (as opposed to mandatory, that need to be run > at least once in every image that contains the testcase in every milestone). > > During the Maverick cycle we have had an obvious problem with the > naming. People not aware of that session still thought that the optional > testcases were optional. Some others, when they discovered that the > optional testcases were not that optional, missed having optional testcases. > > I think that those optional (really optional) tests have a value. Maybe > they are not release critical but they are good to have around, so that > people can manage better their testing processes. Some examples: the ARM > team likes to have a Netboot testcase, but it is not critical if not > covered; the Ubuntu Studio team would like to have some HW specific > tests, but still make it to release, if they are not covered; etc. > > *Proposal* > > * Create a new category "Run Once" and move there those testcases that > need to be run at least once every milestone. (Most of the tests that > are now under the "optional" category.) > * Use the "Optional" category for those tests that really are optional > for release. If they are not covered, or they fail, the release can > still go. > * The "Mandatory" category would remain as it is. > > This is a proposal for the Natty release. If the release team agrees > with those changes, I could implement them during the first week of the > new cycle. If you think that this should be discussed during UDS, let me > know.
This makes a lot of sense to me and I don't any further discussion is necessary. -- Brian Murray
signature.asc
Description: Digital signature
-- Ubuntu-qa mailing list Ubuntu-qa@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa
