Thanks Basil for bringing this topic to attention! I am all in for this new approach! I think when we spend time to write UI tests (which are expensive to write and maintain, and slow to run, see test pyramid of Mike Cohen) then we should consider to run them as real system tests using a real browser and a real Jenkins instance and not as part of our integration tests with a fake UI (HTMLUnit) and a fake Jenkins (started via JTH).
Since I already converted several plugins in the way you are suggesting I can give some more insights about the things that need to be done, the advantages, and disadvantages. I can only speak for plugins but in theory the same process should be possible for core as well. First of all, the credits for this approach go to Oliver Gondža! He created the building blocks in the acceptance test harness and his own plugins. If we follow this approach, a plugin will have three different types of tests: 1) Unit tests that test classes in isolation. 2) Integration tests that test one or more classes together in a running Jenkins instance (started by JTH). These tests will solely work on Jenkins' object model and will not use the browser HTML output anymore. 3) UI tests as end-to-end tests for one or more scenarios of the plugin use cases. These tests will then be started using maven-surefire (1) and maven-failsafe (2) in our normal pipelines (buildPlugin() or GitHub Actions). Those results will visible in the usual way in a pull request. The UI tests (3) need to be started as a separate process since they work on the created artifacts. I’m using GitHub Actions in my plugins to start the individual UI tests, but it should be possible to run on Jenkins as well. Those tests will run in a special docker container with a headless browser (chrome or firefox). All three types of tests are started for every pull request. So if a committer breaks something in the UI it will be visible as a PR check failure. Even if Jenkins core changes from 1.n to 1.(n+1), a Dependabot PR will be triggered and evaluated. If you want to use that approach in your plugin you need the following steps: 1) Change the folder structure of a plugin: we need a maven sub-module for the plugin and one for the corresponding ui-tests. Basically, all existing files need to be moved below the plugin folder. 2) Create a new ui-test project for the plugin. 3) Move existing Selenium page objects and test cases from the ATH to the plugin. Then you will have the previous ATH tests as part of your plugin. Here are some working examples: - a feature PR in the Warnings Plugin (https://github.com/jenkinsci/warnings-ng-plugin/pull/1206 <https://github.com/jenkinsci/warnings-ng-plugin/pull/1257>) - Jenkins upgrade PR in the Code Coverage Plugin (https://github.com/jenkinsci/code-coverage-api-plugin/pull/377 <https://github.com/jenkinsci/code-coverage-api-plugin/pull/377>) Here are two work-in-progress PRs: - JUnit Plugin (https://github.com/jenkinsci/junit-plugin/pull/355 <https://github.com/jenkinsci/junit-plugin/pull/355> or https://github.com/jenkinsci/junit-plugin/pull/342 <https://github.com/jenkinsci/junit-plugin/pull/342>) - PIT Plugin (https://github.com/jenkinsci/pitmutation-plugin/pull/129 <https://github.com/jenkinsci/pitmutation-plugin/pull/129>) So I think that it something that each plugin author can do without much effort in each plugin. So as a summary the advantages: - UI tests work with a real browser (and do not break due to HTMLUnit incompatibilities) - integration tests are fast and work with assertions on the Jenkins object model. - when you do not use HTMLUnit anymore you will also find better ways to test you Java side models. This will lead to a better Java model! The UI layer should be as thin as possible and the logic will be part of Java that can be tested with Unit tests in Java. - both types of tests are evaluated for each pull request The disadvantages: - not all plugins have UI tests yet, so there is an initial ramp up time required - ATH based UI tests are more fragile - build execution times of PRs will increase since the initialization is slow > Am 25.04.2022 um 21:19 schrieb Basil Crow <m...@basilcrow.com>: > > Hi all, > > It seems to me that a number of problems are caused by ATH tests being > in a separate repository from the code under test. When regressions > occur, they are not discovered until much later. This causes pain for > those who investigate ATH test failures. If the ATH tests were moved > to the same repository as the code under test (i.e., core or plugins), > then these issues would be caught prior to the problematic change > being merged and released. This would be a win for those who > investigate ATH failures. > > Related is the fact that ATH provides a more realistic execution > environment than JTH. ATH supports modern JavaScript frameworks, but > JTH does not. Being able to easily write ATH tests in core and plugins > would be a win for those who make frontend changes. New features could > be delivered with associated test automation at the time of feature > delivery. > > I think moving the ATH tests to the same repository as the code under > test would be a win on both fronts. ATH and JTH would then both be > test _harnesses_ (as their names imply) but not _collections_ of > tests: those would live in the same repository as the code under test, > and passing them would be a prerequisite for merge and release. > > Integrating such a system into the toolchain would be a significant > project. Providing first-class support for ATH tests in core and > plugins in the context of local builds, plugin CI builds, and BOM/PCT > builds requires a significant amount of work and likely carries > infrastructure prerequisites. Nevertheless, I think such a project > would be of large benefit. It would increase our confidence when > making UI changes, and it would increase our confidence that BOM bumps > do not contain regressions. These benefits would apply to multiple > parties across the Jenkins project. > > Note that this is a proposal for a project and a request for > discussion, but I am explicitly not volunteering to implement the > solution (at least not by myself). > > Basil > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkinsci-dev+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjoGK5XdfP38zE1WPH-pcVefRrOc7ndvWqApFMzEGEE0zg%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/220833BB-29A7-42CD-B920-1F9BD90241BB%40gmail.com.
