On Mon, Feb 17, 2020 at 3:20 PM Kamil Paral <kpa...@redhat.com> wrote:
> This proposal intends to reduce the scope of the “*Default application > functionality*” release criterion [0]: > > All applications that can be launched using the standard graphical >> mechanism of a release-blocking desktop after a default installation of >> that desktop must start successfully and withstand a basic functionality >> test. >> > > *= Background =* > > The area which QA is responsible for has been growing and growing in > recent years. We used to have just a single Fedora release with two > blocking desktops (GNOME and KDE). Then Editions got introduced and we > started testing Workstation, Server, Cloud and the KDE Spin. Additional > architectures were introduced (fortunately i386 got obsolete) and we > started testing and blocking on specific images on armhfp and aarch64. > Currently there are 13 release blocking deliverables mentioned on the > ReleaseBlocking page [1]. That list is not complete, though. Fedora CoreOS > became an official edition lately, and even though its release cycle is not > tied to traditional Fedora release cycle (and that’s why it’s not mentioned > on that wiki page), as an official edition QA should care about it as well. > It is the same story with Fedora IoT, another recent release-blocking > addition [2]. Desktop testing is one of the most time-consuming jobs with > the most frequent bug occurrence. Right now, we’re supposed to be fully > testing and blocking on GNOME, KDE and XFCE (although XFCE might get > dropped in favor or aarch64 Workstation [3]). We cannot test all of this > and honestly claim that we verified basic functionality of all the included > apps on all these desktops. I believe we need to adjust the criterion and > align it closer with reality. In my eyes, it’s better to have a narrower > scope and perform it well than having a large scope and perform it poorly. > > *= Proposal =* > > Change the criterion to something along these lines: > > All applications that can be launched using the standard graphical >> mechanism after a default installation of Fedora Workstation on x86_64 >> architecture must start successfully and withstand a basic functionality >> test. >> >> For other release-blocking desktops (on any architecture), the >> requirements only apply to the following types of applications: >> >> * web browser (e.g. firefox) [4] >> >> * file browser (e.g. nautilus) >> >> * package manager (e.g. gnome-software) >> >> * image viewer (e.g. eog) >> >> * document viewer (e.g. evince) >> >> * text editor (e.g. gedit) >> >> * archive manager (e.g. file-roller) >> >> * terminal emulator (e.g. gnome-terminal) [4] >> >> * problem reporter (e.g. abrt) >> >> If there are multiple applications of the same type (e.g. several web >> browsers), only one of them needs to satisfy the requirements. >> > > As you can see, the original criterion was kept for Fedora’s flagship > desktop edition, the one that is most prominent on https://getfedora.org > and probably the one that most newcomers download. We would still verify > everything on Fedora Workstation on x86_64. But any other desktop > (including Workstation on aarch64) would get just reduced criteria, because > we simply can’t ensure the same quality bar for the smaller desktop > editions/spins. There are some high-profile types of applications that I > considered including in the list above, but didn’t in the end: > > * word editor (e.g. libreoffice-writer) > > * spreadsheet editor (e.g. libreoffice-calc) > > * video player (e.g. totem) > > * help viewer (e.g. yelp) > > I’d like to hear your thoughts on whether they should be included or not. > Of course from an end-user point of view, it would be beneficial. But the > question is whether we as QA can promise their testing. And also whether we > want to block the release e.g. if Gnumeric is broken on armhfp XFCE or if > totem doesn’t work on aarch64. Yes, it’s unpleasant, but people using > alternative desktops and architectures are usually far from beginners. It’s > usually not difficult to install a different application. Also note that > the apps included in x86_64 Workstation will be thoroughly tested so they > should be very likely to work well even on other architectures (minus some > arch-specific issues). > > I’d like to target Fedora 32 with this proposal, which means we should > decide on this proposal fast. (I wanted to propose it much sooner, but I > was waiting on clarifications about new release-blocking images and also on > some fesco tickets [3], some of which is still not clarified; so I’m sorry > about a late proposal). > > Please comment, thanks. > > > [0] > https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Default_application_functionality > > [1] https://fedoraproject.org/wiki/Releases/32/ReleaseBlocking > > [2] https://pagure.io/fedora-iot/issue/23 > > [3] https://pagure.io/fesco/issue/2339 > > [4] These two are required to work even in Basic release criteria [5], but > I decided to include them anyway to avoid confusion (“why is the web > browser missing?”), and to make it clear that they need to satisfy the > basic functionality (whereas for Basic release criteria just the barebone > functionality might be deemed enough) > > [5] > https://fedoraproject.org/wiki/Basic_Release_Criteria#Required_applications > > > I haven't received much feedback on this proposal. On test list and kde list, there was a short follow-up regarding the list of release-blocking applications. On kde list, Kevin disagreed because it lowers the guaranteed quality bar for KDE, but he didn't continue discussing it with me. I haven't received any alternative proposals either. On desktop list, there was silence, but Chris Murphy gave me a thumbs up on IRC. I'll give it a few more days, and unless somebody yells hard (and ideally also provides some constructive ways how to optimize desktop testing, because we need to cut it down /somehow/), I'll put the change into effect. I'll also probably move the help viewer into the list of release-blocking applications, based on the discussion I saw. Thanks, Kamil Fedora QA
_______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org
