I'm overdue on posting something to the anaconda (maybe devel also?) list regarding installer testing, so this email is about flushing out some of the issues before I do that, so that hopefully it's not quite so messy as it is in my head right now and thus in this email.
So way back in January we had a QA meeting following some suggestion by adamw regarding priorities for testing the installer which I paraphrased like this: http://meetbot.fedoraproject.org/fedora-meeting/2013-12-23/fedora-qa.2013-12-23-16.00.log.html 16:37:49 <cmurf> The main thing is I agree with categorizing/prioritizing the test matrix: #1 critical must work, #2 in between, #3 bonus My essential argument, without taking Fedora.next into account, is that the installer's feature creep is incongruent with QA's available resources to test every possible outcome from the installer. In fact, we had trouble testing the latest Guided path partition scheme (LVM thinp) which shipped broken in Fedora 20. So far we've accepted the feature creep and simply admit we can't test everything, and leave it at that. But this process is essentially non-transparent to end users who certainly assume some minimal testing of every installer feature and outcome in both Guided and Manual partitioning. But then there's Fedora.next, and based on my reading of FESCo and WG meeting notes, it *seems* like there is some leaning toward raising the quality bar of Fedora to be biased more toward production/stability, rather than test bed oriented. I understand that's a continuum, not a binary condition, so I think the email needs to elicit a response by the WGs to be sufficiently clear on quality level expectations, if that is in fact going to change. If the bar is going to be raised, then my instinct is to be even more aggressive with how the installer should only present recommended or at least sane outcomes to users. It probably shouldn't ever crash. We probably should have *one* file system option for Guided partitioning, which is the recommended layout, and the user gets to choose a couple of variations: encryption, and a way to reuse an existing /home. Case in point, Guided partitioning in Fedora 20 permits some 80 possible outcomes. It's probably somewhere around several hundred, to infinity, for Manual partitioning. By contrast, Windows and OS X installers permit approximate two to a handful for Windows and one for OS X. A handful of additional outcomes are possible by using included partitioning utilities outside their installers. In any case, we aren't talking about even 80 outcomes let alone hundreds let alone infinite. Conclusion: I think the cows have almost all escaped because we didn't have a pen to put them in, and I'm suggesting that significant constraints are needed probably in the way of chopping or hiding features. If that's not going to happen then I think the fallback is to work with Docs to document our grand daddy test matrix to show users effectively what's recommended. The problem with that is that those results won't be consistent release to release, if we have a large percent of "bonus" or open ended tests that may or may not get done - and I'd argue that's incongruent with raising the bar. So really, I'm fairly convinced at this point that what's needed is feature chop, it's just a matter of how much which depends on what quality level expectations the WGs decide upon. Chris Murphy -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
