I like this idea in general. It would be a good thing to consider for a
future test sprint. I have a few high-level questions on the specifics:
1. I've not used JUnit5 TestWatcher at all, but it seems like it might
do what we want. Does anyone else have practical experience with it? For
cases where a test passes or is skipped, is there any impact on the report?
2. Even if we do annotate all our JUnit5 headful robot tests with this,
I would want to qualify that on a system property -- probably off by
default. I presume this should be as easy as having the "onFailed" (or
whatever it's called) method of the watcher check this property and do
nothing if it isn't enabled?
3. Depending on how the test exits, it's quite possible that the JavaFX
runtime has shutdown. It might be a bit tricky to do this in a robust
manner without doing an exec of a new Java process. Any thoughts on this?
4. In answer to the problem of what happens if all too many tests fail
(flooding the log), maybe you could have a system property that sets a
max number of images to log? That might be hard to implement, though,
unless you store some information in the build directory of the system
tests, since each system test necessarily runs in a separate Java process.
5. Is writing to the log (e.g., System.err) the best way to capture the
data? Whatever we do, we will want some easy way to extract and display
the screenshots.
If we do move forward with this, and use the JUnit5 TestWatcher
facility, this will be one more motivation to drive the conversion of
tests from JUnit4 to JUnit5 -- at least in the system tests project. See
JDK-8328629 [2].
-- Kevin
[2] https://bugs.openjdk.org/browse/JDK-8328629
On 3/22/2024 1:49 PM, Andy Goryachev wrote:
Dear colleagues:
What do you think of the following idea:
The Problem
===========
Automated headful tests might fail intermittently or on specific
platform(s). After the fact, it's nearly impossible to understand why
this happened.
Solution
========
Implement a JUnit5 TestWatcher which captures the primary screen
pixels and dumps them as base64-encoded PNG images in stderr (or
stdout). The logs are typically preserved, so the screenshots can be
analyzed after the fact.
As far as I know, there is no way to configure JUnit to apply this to
each test using command line option, so each test needs to have the
following annotation added:
@ExtendWith(ScreenCaptureTestWatcher.class)
For more details please see this ticket [0] and the draft PR [1].
Possible Issues
===============
* the size of the image can be quite large, for example retina mac
increases the size of the log by approximately 4MB (for each image).
* a misconfiguration might cause every test to fail, resulting in
very large logs
References
==========
[0] https://bugs.openjdk.org/browse/JDK-8328716
[1] https://github.com/openjdk/jfx/pull/1415
What do you think?
Cheerio,
-andy
