Hi! Have you had issues with a tmt test not working due to a package installation issue? Maybe you had a big side-tag update not passing some gating test on bodhi or testing-farm? Good news, we are on our way to resolving these issues and your feedback would be greatly appreciated.
Recently we've started work on the new approach for installing tested artifacts (e.g. rpm packages) on the guest before the testing starts. We would like to gather early feedback on the proposed design as this is quite a significant change. Here we'll be demonstrating the approach on a koji build and included rpm packages but the same/similar might be applied for brew builds, copr builds, local rpms and other artifact sources. We also want to keep the implementation open for possible future extensions (e.g. container images). What are the most significant changes? Before: Testing Farm takes each requested koji build, downloads and installs all rpm packages included on the guest. There are some special hacks for the "exclude" key which allow handling conflicting rpms in a single build. After: Testing Farm hands over the koji build id to tmt which installs from the koji build **only** those packages which are explicitly requested by the test or plan. Which artifact sources will be supported? Currently we have the following sources on the list: - koji/brew build <https://github.com/teemtee/tmt/issues/253> (identified by build id, task id or nvr) - copr build <https://github.com/teemtee/tmt/issues/3993> (identified by <build-id>:<chroot-name>) - dnf repository <https://github.com/teemtee/tmt/issues/3994> (identified by baseurl of the repo) - dnf repository file <https://github.com/teemtee/tmt/issues/3995> (identified by url to the repo file) - rpm files <https://github.com/teemtee/tmt/issues/3992> (individual file paths or a directory, local or remote) But more are expected in the future. See the linked issues for more details and the outlined implementation. Feel free to ask questions or suggest changes directly there. How exactly will this work? All packages from the koji build are downloaded on the guest. List of the exact package nvrs is stored. A dnf repository is created from the packages and enabled with a suitable priority. Packages mentioned in test `require` or `recommend` keys, or in the plan prepare step are installed. Versions of the installed packages are verified to make sure the right packages were installed. Here’s an example prepare step config: prepare: summary: Prepare repository with packages how: artifact stage: prepare provide: - koji.build:1234 - koji.task:5678 - repo-file:https://another.url/repofile - repo-url:https://another.url/repo You will most likely not interact with it via the static files, instead testing-farm will take care of injecting this step via the command line: tmt run --all prepare --insert --how artifact --provide koji.build:1234 --provide koji.task:5678 What are the benefits? Consistent way to prepare artifacts for testing, e.g. when developing tests locally on your laptop you would get the very same result as when run in the Testing Farm. Simplified tmt reproducer line. This will also allow us to fully move to the new pipeline <https://issues.redhat.com/browse/TFT-3696> which supports multihost testing. No problems when conflicting packages are included in the koji build as only the right subset of packages is installed on the guest. Possibility to test minimal install or various combinations of packages which can reveal bugs hidden by always installing all packages. If a test needs to perform some advanced package handling actions (e.g. verifying that reinstalling a package is working as expected) it can make use of the dnf repository with the fresh packages to perform the desired dnf operations. No extra steps should be needed from either Testing Farm or the user. What is expected from the users? Make sure that all packages necessary for testing are included in the require <https://tmt.readthedocs.io/en/stable/spec/tests.html#require> or recommend <https://tmt.readthedocs.io/en/stable/spec/tests.html#recommend> key of the test or are explicitly installed using one of the prepare <https://tmt.readthedocs.io/en/stable/plugins/prepare.html> step plugins. What do you think? Does the proposed design look ok? Do you have any concerns? Are there any important use cases or corner cases we should keep in mind when working on the implementation? See the tracking issue for some more details about the design and the current progress: https://github.com/teemtee/tmt/issues/3990 We would be especially interested in suggestions regarding what would be the best way to set the correct priority for the repository so that it plays well with other defaults. Also, would you have any recommendations about what should happen if there are multiple nvrs for the same package included in the source repository? Looking forward to your feedback. psss...
-- _______________________________________________ devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
