Going back to the earlier premise > It seems to me that a number of problems are caused by ATH tests being in a separate repository from the code under test
I would say the biggest problem is not so much that - rather that we do not run enough of these tests for the code that is affected, early enough, nor is there probably a good set of well written smoke test candidates. Moving the code is one possible solution to the first issue, but it also has other problems. For example If we update the PageObjects you then need to update N repos (or just not bother) to account for the change in API. (granted changes in API are hopefully minimal, but that is not always the case, esp wrt things like table-2-div). I discussed with directly with Mark Waite, but to me we need some new tests that better cover what we consider to be "critical functionality", (ie an Acceptance test), what we tend to have today is a bunch of limited tests where things are testing some things in isolation. I would love to see a test that was run on a core PR that at least created a credentials (in different scopes, stores), an agent a a pipeline. And check that the pipeline can checkout from a Git Repo (github would be nice, but attempting to minimize network operations and hence flakes and also brings security issues into play...) and build some PRs and a main branch. And something similar with a FreeStyleJob. (include in that Junit reporting). Currently the smoke tests for core PRs are too fine grained to me. I think we could make some quick wins with the following form of attack * extend buildPlugin so that you can specify a list of ATH tests to run (currently most of the tests are in the form of plugins.<plugin-name>Test * introduce a few new tests like discussed above that are run on Jenkins PRs. Whatever we have, if we end up having ATHs on core PRs to detect regressions early then we need to be prepared to maintain the ATH tests/framework or to not accept the changes. The maintenance has mostly fallen to a few individuals like Tim, Vincent and myself. We had a passing build last October, but it did not last (as things where not run with any associated changes). I am all in favour of improving the current situation however that happens and am happy to see this gaining traction. /James On Wednesday, April 27, 2022 at 7:33:44 PM UTC+1 ullrich...@gmail.com wrote: > > > Am 26.04.2022 um 18:28 schrieb Basil Crow <m...@basilcrow.com>: > > > > On Tue, Apr 26, 2022 at 7:45 AM Ullrich Hafner <ullrich...@gmail.com> > wrote: > > > >> Those tests will run in a special docker container with a headless > browser (chrome or firefox). > > > > Ideally, I think we would want to support both Dockerized and > > non-Dockerized builds if possible. > > You can run the build locally without Docker. Docker is just used in CI > (GitHub Action or Jenkins). I am locally using a different Maven profile > and run the tests without Docker in Firefox or Chrome. You can even run it > from IntelliJ and set breakpoints. > > > > >> […] run them as real system tests using a real browser and a real > Jenkins instance and not as part of our integration tests with a fake UI > (HTMLUnit) and a fake Jenkins (started via JTH) […] > > > > Regarding the use of a "real Jenkins instance," I wonder how the ATH > > implementation compares to RealJenkinsRule, which is at a high level > > doing something very similar; namely, starting Jenkins in a separate > > Java process and tearing it down at the end of the test. If the > > RealJenkinsRule implementation is more efficient while still remaining > > compatible with the goal of frontend testing with a headless browser, > > it may be worth looking into unifying the two implementations in order > > to reduce maintenance and/or increase performance. > > > >> not all plugins have UI tests yet, so there is an initial ramp up time > required > > > > I think if we build it they will come. > > > >> […] UI tests are more fragile > > > > Has this ever not been the case in the decades you and I have been > > programming? I guess the benefit of this idea depends on _how_ > > fragile. If the fragility is occasional and a retry chases the problem > > away, that could be tolerable. On the other hand, if the fragility is > > frequent and requires one or more retries, this could undermine the > > idea. > > > >> build execution times of PRs will increase since the initialization is > slow > > > > I wonder if some targeted optimization couldn't be done through the > > use of techniques such as the ones used in RealJenkinsRule. > > > > Basil > > > > -- > > You received this message because you are subscribed to the Google > Groups "Jenkins Developers" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to jenkinsci-de...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjphex3uXLfjsAfCpEaFvMNpXgBocgnX5tHaSuqTV1iZ7w%40mail.gmail.com. > > > > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/2ec3290f-93cc-40d1-926f-bedffa1c9d2fn%40googlegroups.com.
