>> Although ATH uses RESTish endpoints for a few purposes, for the most part >> the test setup (not just the actual assertions) uses the browser. In all >> cases the whole test run is “black-box”. This has its appeal (stronger >> coverage) but GUI setup makes tests much slower and either REST or GUI setup >> can also make it a lot more awkward to write tests than when using >> `[Real]JenkinsRule`, which are “white-box” and let you quickly set up an >> environment and run some assertions at the Java level. JTH also lets you >> write test-only extensions, which ATH does not. > > I see; very interesting. I wonder how many parts of the test setup > process are common to many tests/plugins versus unique to each > test/plugin. For example, I could imagine that things like going > through the setup wizard, creating a freestyle project, or adding an > agent are common to many tests (and therefore pointless from the > perspective of test coverage to repeat them multiple times), while > other parts of the test setup process, like configuring a specific > plugin, might be unique to each test/plugin. If true, and if the test > time is dominated by the parts of the test setup process that are > common to many tests/plugins, perhaps we could, for each portion of > the test setup process that is common to many tests/plugins, run that > portion with the GUI/REST API in one "primary" repository (e.g., core, > Workflow: Job, etc) and (without loss of test coverage at a macro > level) run that portion with JCasC/REST or even Java/JTH in the > "secondary" repositories as a way of speeding up and de-flaking the > setup process to get to the unique/interesting part.
This will depend on the type of the plugins. In my analysis plugins my tests start by creating a freestyle job or pipeline. There is no previous setup required. I don't think that this can be speed up with job-dsl or something different. > > In other words, I wonder if a middle ground between test coverage and > performance couldn't be found. Or maybe I am hopelessly naive and this > is an exercise in futility in the long term. I will discuss this with > my friend who has experience in this domain. > >> ATH installs plugins, and selects plugin versions, using user-mode tools. >> `[Real]JenkinsRule` follow the Maven test classpath. Possibly ATH could be >> given an option to define a mock UC based on a test classpath. > > Yes, a great example of the "middle ground" approach from the previous > paragraph! This is already possible. In my UI tests I am starting the Jenkins under test with the same set of plugins that I am using in the integration tests (JTH). These plugins are just copied from the target folder of the plugin maven module to the target folder of the ui module and the JUT starts with them without installing them. > >> Also whether the fragility is in the actual assertions, or test setup (point >> #1 above). > > I think your point is that fragility in the test setup would undermine > the idea at its core, while fragility in the actual assertions could > be tolerated on a case-by-case basis in individual test suites. If so, > I concur, and I think that finding a way to mitigate the fragility of > test setup by factoring out that logic as described in the preceding > paragraphs could be one solution. > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkinsci-dev+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjpmQLNRwhJSGYc3wT28P32gHvt2aF_CJtkDtX47iwBGzw%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CCF7B65B-3986-4C78-B66C-1417C01264F5%40gmail.com.
