juergencodes commented on issue #834:
URL: https://github.com/apache/maven-surefire/issues/834#issuecomment-2829810690
I experienced a similar problem: My maven build was successful though I had
a test failure. Since it is in a setup of a custom framework based on JBehave
it is interesting to see that others experience similar problems in cucumber
(this issue) or e.g. Arch (following the links in SUREFIRE-2229).
I was drilling down a bit deeper and found out the following reproducer that
just uses DynamicTest (which probably all the above mentioned frameworks use)
and plays with providing test source.
```
@TestFactory
DynamicTest withSource() {
return test("withSource", URI.create("pom.xml"));
}
@TestFactory
DynamicTest noSource() {
return test("noSource", null);
}
DynamicTest test(final String test, final URI source) {
return DynamicTest.dynamicTest(test, source, () -> {
System.out.println("Running '" + test + "'");
org.junit.jupiter.api.Assertions.assertNull(source);
});
}
```
Running the test (also see attachment for "full" project) with surefire
2.5.2 fails as expected with
```
[INFO] Running SurefireReproducerTest
Running 'withSource'
Running 'noSource'
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
0.049 s <<< FAILURE! -- in SurefireReproducerTest
[ERROR] withSource().withSource -- Time elapsed: 0.012 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <null> but was: <pom.xml>
at
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertNull.failNotNull(AssertNull.java:50)
at org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:35)
at org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:30)
at org.junit.jupiter.api.Assertions.assertNull(Assertions.java:279)
at
SurefireReproducerTest.lambda$test$0(SurefireReproducerTest.java:20)
at java.base/java.util.Optional.ifPresent(Optional.java:178)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] expected: <null> but was: <pom.xml>
[INFO]
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
```
Switching to 2.5.3 makes
```
[INFO] Running SurefireReproducerTest
Running 'withSource'
Running 'noSource'
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.049
s -- in SurefireReproducerTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
```
Mind that both tests are executed (see "Running..." output), but
- No error is reported
- The build does not fail
- The test count dropped from 2 to 1
(just in case it matters, i used Apache Maven 3.8.6)
[surefire-reproducer.zip](https://github.com/user-attachments/files/19905590/surefire-reproducer.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@maven.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
