On Wed, May 4, 2022 at 2:15 AM Adam Williamson <adamw...@fedoraproject.org> wrote:
> > 1. I know we have had GNOME Test days, but this > > stuff didn't come up. Presumably, it would have if someone had > happened > > to look at GNOME Photos. Can we formally go through > > https://www.fedoraproject.org/wiki/QA:Testcase_desktop_app_basic and > > > https://www.fedoraproject.org/wiki/QA:Testcase_desktop_app_basic_others > > much, much earlier (probably when the GNOME pre-release is available), > > either as part of a test day or some other formal thing? Or is there > > something else going on here that I'm not looking closely enough to > see? > > Well, kinda, yes. The thing going on is that we *did* go through that > test at several earlier points: > > > https://openqa.fedoraproject.org/testcase_stats/36/Desktop/QA_Testcase_desktop_app_basic_others_Release_blocking_desktops___lt_b_gt_x86___x86_64_lt__b_gt_.html > > but these bugs weren't discovered at that time. This is likely because > of your idea 2: "basic functionality" is a bit up for debate. You can > take an extremely minimalist approach to this (run the app, click a > couple of buttons, say it's OK if nothing explodes and no babies are > eaten) or a slightly more maximalist one (run the app, and actually try > and do something useful with it). In this case, before Final, when we > ran this test we mostly did the minimalist thing. At Final RC stage, we > went a bit more maximalist. > I wouldn't call that maximalist, that would mean testing everything possible. A *realistic* approach is more appropriate, I think. To "do something useful with it" is a great description. What good is it if it can start and survive a few button clicks, if you can't do useful tasks? The tasks it was actually designed for. What's the point of shipping a photo organizer where you can't create and organize albums? This is the realistic scenario, and ideally the thing we would always try to test. Of course, not always people have time to do that, and only some pathways can be broken while not others. Discovering bugs in the realistic scenarios requires time and also some luck. At the same time, bugs in realistic scenarios will very likely prevent actual users from actually using that app, even if just certain pathways are broken. Even if some of those bugs seem trivial to discover, they are not. If you edit any other field than the email address in gnome-contacts, you'll not see contacts duplication. So as a QA, you only have a certain chance to spot it. In the real world with regular usage, however, sooner or later you'll definitely edit someone's email address and then you'll see that contact doubled. (This is just a nice example, this particular bug was not accepted as a blocker, as it is not ground-breaking).
_______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
