On Thu, 2012-08-09 at 05:49 -0400, Kamil Paral wrote: > > Hey, folks. So, as per https://fedorahosted.org/fedora-qa/ticket/307 > > , I > > think we need to add a process for validating the multi-live and > > multi-install images that we've been building for the last few > > releases. > > It is ironic that before F17 release I have talked about this to cwickert > (IIRC) and he assured me no QA help is needed and that nothing can go wrong. > When they build the image, they just ensure everything boots and that's it. > There's just a small syslinux glue, nothing else is changed. Well, it didn't > go according to the plan, apparently. > > > I've put together a draft matrix template for this validation here: > > > > https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Compendium_image_validation_results_template > > > > It's similar to the other validation matrices, and the changes are > > mostly obvious. The key bit is the layout of the matrix and the test > > cases themselves. I haven't actually written any of the specific test > > cases yet, but I tried to think of the best way to organize things. > > We > > obviously need to do the basic sanity tests on each image, then we > > need > > to test that each image boots and the menu that lets you select which > > sub-image to boot works properly, and also that each sub-image boots > > and > > installs properly. I think that set of tests ought to pretty much > > cover > > the intended functionality of the compendium images. When actually > > writing the test cases, we might be able to come up with ways to take > > advantage of some of the existing install/base/desktop validation > > test > > cases, but I haven't quite thought that far ahead yet. > > > > I tried to organize the planned test cases as efficiently as > > possible; > > this is the best layout I could come up with, but someone might be > > able > > to suggest improvements. In case it's not obvious, here's roughly > > what > > each test case does: > > > > QA:Testcase Mediakit_ISO_Size and QA:Testcase_Mediakit_ISO_Checksums > > - > > these test cases already exist and can apply without modification to > > the > > compendium images, they simply check that they're within required > > size > > limits (dual-layer DVD size for these images) and the published > > checksums match the images. > > > > QA:Testcase_multilive_boot and QA:Testcase_multidvd_boot - these will > > simply check that the compendium images themselves boot > > I don't understand what this means. > > To clarify, multidvd structure is this: > -> default DVD boot (auto-selects architecture) > -> select arch manually -> i386 > -> x86_64 > -> basic video -> both 2 options > -> memory test > -> local drive boot > > So you probably mean the "default DVD boot" here, right? But which > architecture? The result might differ on i386 and on x86_64 machine. > > But multilive structure is this: > -> default Live GNOME (auto-selects architecture) > -> default Live KDE (auto-selects architecture) > -> default Live LXDE (auto-selects architecture) > -> default Live SoaS (auto-selects architecture) > -> default Live XFCE (auto-selects architecture) > -> select arch manually -> i386 -> Live GNOME > -> Live KDE > -> Live LXDE > -> Live SoaS > -> Live XFCE > -> x86_64 -> Live GNOME > -> Live KDE > -> Live LXDE > -> Live SoaS > -> Live XFCE > -> basic video -> all 10 options here > -> memory test > -> local drive boot > > What does the test refer to for multilive?
> I have to say I'm somewhat confused from the current matrices structure. But > maybe when the test cases are written it'll clearer. > > My idea personally was: > > MultiLive matrix | GNOME | KDE | LXDE | SoaS | XFCE > ------------------------------------------------------------------ > multilive_auto_boot (i386) | > multilive_auto_boot (x86_64) | > multilive_manual_boot (i386) | > multilive_manual_boot (x86_64) | > multilive_install (anyarch) | > > "auto_boot" is when you select the default option and correct architecture is > chosen for you. To test this properly, you need both i386 and x86_64 machines > (or VMs, if accepted). > "manual_boot" is when you select the architecture manually from the submenu. > A single x86_64 machine is sufficient to check both cases. > > MultiDVD matrix | result > --------------------------------------- > multidvd_auto_boot (i386) | > multidvd_auto_boot (x86_64) | > multidvd_manual_boot (i386) | > multidvd_manual_boot (x86_64) | > multidvd_install (anyarch) | > > I am wondering whether we need installation tests at all. Theoretically the > ISOs are unchanged so there should be no difference at all. If they start to > boot, it should be fine. But according to Murphy's Law... OK, I revised the middle table somewhat, I tried to figure it out myself but it turns out fairly similar to yours: https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Compendium_image_validation_results_template#Boot_and_sub-image_selection so 'multi_auto' would just be 'leave it, and check it does the right thing', 'multi_manual' would be a test case that's something like 'try every entry besides basic video / memtest / local boot and make sure they respond appropriately', and the other three are obvious - we probably don't really need to do those three on both arches, we could split out a small table for those, I guess. The multi_sub_boot test now becomes redundant, and I suppose we could move sub_install up into this new big table; we can probably get away without testing install of every DE, since they all use anaconda. The case I can see where install might break on the compendium images is the case where anaconda, for whatever reason, examines the actual medium it's booting from. It _does_ do this, in some cases - like it tries to except the USB stick it's booting from (if it is) from the target device list. (This test may well not work on the compendium images, actually). I can possibly imagine a case where anaconda goes to look what image it's booting from, and gets confused if it's the compendium disc. But that shouldn't vary between desktops. This is a pretty tricky problem space and I'm not at all confident I'm wrapping my head around it perfectly, more comments and improvements of course welcome =) Thanks! -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
