I'm in favour of this although I can't dedicate time to it currently. > Whatever we have, if we end up having ATHs on core PRs to detect regressions early then we need to be prepared to maintain the ATH tests/framework or to not accept the changes.
If PRs break the ATH and they are in the same repository of course we would expect them to be fixed as part of the PR. > We had a passing build last October, but it did not last (as things where not run with any associated changes). We have fairly consistently passing builds, (apart from a number of agent disconnections requiring re-runs). Occasionally a core PR will be merged that wasn't run against ATH and breaks it, or a plugin changes and needs corresponding updates in ATH. Both of the above issues would be solved by moving the tests to corresponding plugins and core and then having an aggregator project that can run tests against all of them like BOM. If the framework was in core and we replaced existing HtmlUnit tests I think a lot more would be written. Recently both Jan and myself have hit a number of issues with HtmlUnit not supporting modern JavaScript. (You implement it and it works fine in the browser but then the PR test builder blows up and you need to re-implement it) Thanks Tim On Wed, 11 May 2022 at 15:11, 'jn...@cloudbees.com' via Jenkins Developers < jenkinsci-dev@googlegroups.com> wrote: > Going back to the earlier premise > > > It seems to me that a number of problems are caused by ATH tests being > in a separate repository from the code under test > > I would say the biggest problem is not so much that - rather that we do > not run enough of these tests for the code that is affected, early enough, > nor is there probably a good set of well written smoke test candidates. > > Moving the code is one possible solution to the first issue, but it also > has other problems. For example If we update the PageObjects you then need > to update N repos (or just not bother) to account for the change in API. > (granted changes in API are hopefully minimal, but that is not always the > case, esp wrt things like table-2-div). > > I discussed with directly with Mark Waite, but to me we need some new > tests that better cover what we consider to be "critical functionality", > (ie an Acceptance test), what we tend to have today is a bunch of limited > tests where things are testing some things in isolation. > > I would love to see a test that was run on a core PR that at least created > a credentials (in different scopes, stores), an agent a a pipeline. And > check that the pipeline can checkout from a Git Repo (github would be > nice, but attempting to minimize network operations and hence flakes and > also brings security issues into play...) and build some PRs and a main > branch. > And something similar with a FreeStyleJob. (include in that Junit > reporting). Currently the smoke tests for core PRs are too fine grained to > me. > > I think we could make some quick wins with the following form of attack > > * extend buildPlugin so that you can specify a list of ATH tests to run > (currently most of the tests are in the form of plugins.<plugin-name>Test > * introduce a few new tests like discussed above that are run on Jenkins > PRs. > > Whatever we have, if we end up having ATHs on core PRs to detect > regressions early then we need to be prepared to maintain the ATH > tests/framework or to not accept the changes. The maintenance has mostly > fallen to a few individuals like Tim, Vincent and myself. We had a passing > build last October, but it did not last (as things where not run with any > associated changes). > > I am all in favour of improving the current situation however that happens > and am happy to see this gaining traction. > > /James > > On Wednesday, April 27, 2022 at 7:33:44 PM UTC+1 ullrich...@gmail.com > wrote: > >> >> > Am 26.04.2022 um 18:28 schrieb Basil Crow <m...@basilcrow.com>: >> > >> > On Tue, Apr 26, 2022 at 7:45 AM Ullrich Hafner <ullrich...@gmail.com> >> wrote: >> > >> >> Those tests will run in a special docker container with a headless >> browser (chrome or firefox). >> > >> > Ideally, I think we would want to support both Dockerized and >> > non-Dockerized builds if possible. >> >> You can run the build locally without Docker. Docker is just used in CI >> (GitHub Action or Jenkins). I am locally using a different Maven profile >> and run the tests without Docker in Firefox or Chrome. You can even run it >> from IntelliJ and set breakpoints. >> >> > >> >> […] run them as real system tests using a real browser and a real >> Jenkins instance and not as part of our integration tests with a fake UI >> (HTMLUnit) and a fake Jenkins (started via JTH) […] >> > >> > Regarding the use of a "real Jenkins instance," I wonder how the ATH >> > implementation compares to RealJenkinsRule, which is at a high level >> > doing something very similar; namely, starting Jenkins in a separate >> > Java process and tearing it down at the end of the test. If the >> > RealJenkinsRule implementation is more efficient while still remaining >> > compatible with the goal of frontend testing with a headless browser, >> > it may be worth looking into unifying the two implementations in order >> > to reduce maintenance and/or increase performance. >> > >> >> not all plugins have UI tests yet, so there is an initial ramp up time >> required >> > >> > I think if we build it they will come. >> > >> >> […] UI tests are more fragile >> > >> > Has this ever not been the case in the decades you and I have been >> > programming? I guess the benefit of this idea depends on _how_ >> > fragile. If the fragility is occasional and a retry chases the problem >> > away, that could be tolerable. On the other hand, if the fragility is >> > frequent and requires one or more retries, this could undermine the >> > idea. >> > >> >> build execution times of PRs will increase since the initialization is >> slow >> > >> > I wonder if some targeted optimization couldn't be done through the >> > use of techniques such as the ones used in RealJenkinsRule. >> > >> > Basil >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Jenkins Developers" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to jenkinsci-de...@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjphex3uXLfjsAfCpEaFvMNpXgBocgnX5tHaSuqTV1iZ7w%40mail.gmail.com. >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkinsci-dev+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/2ec3290f-93cc-40d1-926f-bedffa1c9d2fn%40googlegroups.com > <https://groups.google.com/d/msgid/jenkinsci-dev/2ec3290f-93cc-40d1-926f-bedffa1c9d2fn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAH-3Bie8sN_Rg4xGtmzDbJQeL71U3SAcDZ76HgHG0vubPqNf%2Bg%40mail.gmail.com.
