On Tue, Apr 26, 2022 at 7:45 AM Ullrich Hafner <ullrich.haf...@gmail.com> wrote: > Thanks Basil for bringing this topic to attention!
You are welcome! > […] I already converted several plugins in the way you are suggesting […] I am pleased that this approach has been working well in your plugins. Assuming that the idea itself is desirable (and by all means, let me know if you think it is not!), let me focus on some practical concerns around the feasibility of adopting such an approach throughout the core and plugin ecosystem. One of the strengths of this ecosystem is that we expose a rich and polished framework to plugin developers that works out-of-the-box for most use cases. Adding this type of testing to the toolchain while maintaining existing levels of approachability could be a challenge, so there are a few points I want to discuss. > […] we need a maven sub-module for the plugin and one for the corresponding > ui-tests. Ideally, I think we would avoid the use of multi-module Maven projects if possible. The vast majority of Jenkins plugins are single-module Maven projects, and in my opinion the simplicity and low maintenance associated with this is desirable. I wonder if we could design a system such that for the common case only a single Maven project is needed, while still supporting multi-module Maven projects for more complex use cases. I am not sure if your existing multi-module Maven project is due to technical limitations or for structural/organizational reasons. > Those tests will run in a special docker container with a headless browser > (chrome or firefox). Ideally, I think we would want to support both Dockerized and non-Dockerized builds if possible. While most developers today are running Docker, it adds some complexity to the process of debugging local builds and adds some infrastructure requirements for CI builds. I think decreasing the number of requirements and the amount of complexity would make it easier for such a system to gain widespread adoption. Also, we need to think about how such changes would impact existing BOM/PCT builds. If we retain a single Maven module and avoid the requirement for a Docker daemon, then I think few if any changes would be needed here. But if we introduce additional complexity, there might be further infrastructure changes needed in BOM/PCT. > […] run them as real system tests using a real browser and a real Jenkins > instance and not as part of our integration tests with a fake UI (HTMLUnit) > and a fake Jenkins (started via JTH) […] Regarding the use of a "real Jenkins instance," I wonder how the ATH implementation compares to RealJenkinsRule, which is at a high level doing something very similar; namely, starting Jenkins in a separate Java process and tearing it down at the end of the test. If the RealJenkinsRule implementation is more efficient while still remaining compatible with the goal of frontend testing with a headless browser, it may be worth looking into unifying the two implementations in order to reduce maintenance and/or increase performance. > not all plugins have UI tests yet, so there is an initial ramp up time > required I think if we build it they will come. > […] UI tests are more fragile Has this ever not been the case in the decades you and I have been programming? I guess the benefit of this idea depends on _how_ fragile. If the fragility is occasional and a retry chases the problem away, that could be tolerable. On the other hand, if the fragility is frequent and requires one or more retries, this could undermine the idea. > build execution times of PRs will increase since the initialization is slow I wonder if some targeted optimization couldn't be done through the use of techniques such as the ones used in RealJenkinsRule. Basil -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjphex3uXLfjsAfCpEaFvMNpXgBocgnX5tHaSuqTV1iZ7w%40mail.gmail.com.
