[
https://issues.apache.org/jira/browse/SUREFIRE-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17690951#comment-17690951
]
Viktor Adam edited comment on SUREFIRE-2087 at 2/19/23 11:25 PM:
-----------------------------------------------------------------
Can we please reopen this ticket to also address issues with other repeated
runs? (in our case `@TestTemplate`).
We've put together a test case like this:
{code:java}
package demo.surefire2087;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestTemplate;
import org.junit.jupiter.api.extension.ExtendWith;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
@ExtendWith(DemoTestTemplateInvocationContextProvider.class)
public class TestSurefire2087 {
private static boolean flake = true;
private int demoValue = 1;
@ParameterizedTest
@ValueSource(ints = {0, 1})
public void testParameterised(int i) {
assertTrue(i == 1);
}
@TestTemplate
public void testDemoTemplate() {
assertEquals(demoValue, 1);
}
@Test
public void testPlain() {
assertTrue(false);
}
@Test
public void testFlakey() {
if (flake) {
flake = false;
assertTrue(false);
}
assertTrue(true);
}
} {code}
This uses the test template context provider:
{code:java}
package demo.surefire2087;
import org.junit.jupiter.api.extension.BeforeEachCallback;
import org.junit.jupiter.api.extension.Extension;
import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.lang.reflect.Field;
import java.util.Collections;
import java.util.List;
import java.util.stream.Stream;
public class DemoTestTemplateInvocationContextProvider implements
TestTemplateInvocationContextProvider {
@Override
public boolean supportsTestTemplate(ExtensionContext extensionContext) {
return
extensionContext.getRequiredTestMethod().getName().toLowerCase().contains("demotemplate");
}
@Override
public Stream<TestTemplateInvocationContext>
provideTestTemplateInvocationContexts(ExtensionContext extensionContext) {
return Stream.of(
new TestTemplateInvocationContext() {
@Override
public String getDisplayName(int invocationIndex) {
return "demo1--" +
TestTemplateInvocationContext.super.getDisplayName(invocationIndex);
}
},
new TestTemplateInvocationContext() {
@Override
public String getDisplayName(int invocationIndex) {
return "demo2--" +
TestTemplateInvocationContext.super.getDisplayName(invocationIndex);
}
@Override
public List<Extension> getAdditionalExtensions() {
return Collections.singletonList(new
BeforeEachCallback() {
@Override
public void beforeEach(ExtensionContext
extensionContext) throws Exception {
final Object testInstance =
extensionContext.getRequiredTestInstance();
final Field field =
extensionContext.getRequiredTestClass().getDeclaredField("demoValue");
field.setAccessible(true);
field.set(testInstance, 2);
}
});
}
}
);
}
} {code}
Executing this with `mvn verify -Dsurefire.rerunFailingTestsCount=2
-Dtest=TestSurefire2087#<testMethod>` yields the following results on 3.0.0-M9:
1. `testParameterised` (fails as expected)
{code:java}
[main] [INFO] --- surefire:3.0.0-M9:test (default-test) @ test-module ---
[main] [INFO] Using auto detected provider
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[main] [INFO]
[main] [INFO] -------------------------------------------------------
[main] [INFO] T E S T S
[main] [INFO] -------------------------------------------------------
[ThreadedStreamConsumer] [INFO] Running demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped:
0, Time elapsed: 0.069 s <<< FAILURE! - in demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR]
demo.surefire2087.TestSurefire2087.testParameterised(int)[1] Time elapsed:
0.034 s <<< FAILURE!
...
[main] [INFO]
[main] [INFO] Results:
[main] [INFO]
[main] [ERROR] Failures:
[main] [ERROR] com.atlassian.jira.TestSurefire2087.testParameterised(int)[1]
[main] [ERROR] Run 1: TestSurefire2087.testParameterised:21 expected: <true>
but was: <false>
[main] [ERROR] Run 2: TestSurefire2087.testParameterised:21 expected: <true>
but was: <false>
[main] [ERROR] Run 3: TestSurefire2087.testParameterised:21 expected: <true>
but was: <false>
[main] [INFO]
[main] [ERROR] demo.surefire2087.TestSurefire2087.testParameterised(int)[1]
[main] [ERROR] Run 1: TestSurefire2087.testParameterised:21 expected: <true>
but was: <false>
[main] [ERROR] Run 2: TestSurefire2087.testParameterised:21 expected: <true>
but was: <false>
[main] [ERROR] Run 3: TestSurefire2087.testParameterised:21 expected: <true>
but was: <false>
[main] [INFO]
[main] [INFO]
[main] [ERROR] Tests run: 4, Failures: 2, Errors: 0, Skipped: 0
[main] [INFO]
[main] [INFO]
------------------------------------------------------------------------
[main] [INFO] BUILD FAILURE
[main] [INFO]
------------------------------------------------------------------------{code}
2. `testPlain` (fails as expected)
{code:java}
[main] [INFO] --- surefire:3.0.0-M9:test (default-test) @ test-module ---
[main] [WARNING] Parameter 'systemProperties' is deprecated: Use
systemPropertyVariables instead.
[main] [INFO] Using auto detected provider
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[main] [INFO]
[main] [INFO] -------------------------------------------------------
[main] [INFO] T E S T S
[main] [INFO] -------------------------------------------------------
[ThreadedStreamConsumer] [INFO] Running demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped:
0, Time elapsed: 0.034 s <<< FAILURE! - in demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR] demo.surefire2087.TestSurefire2087.testPlain
Time elapsed: 0.018 s <<< FAILURE!
...
[main] [INFO]
[main] [INFO] Results:
[main] [INFO]
[main] [ERROR] Failures:
[main] [ERROR] com.atlassian.jira.TestSurefire2087.testPlain
[main] [ERROR] Run 1: TestSurefire2087.testPlain:31 expected: <true> but was:
<false>
[main] [ERROR] Run 2: TestSurefire2087.testPlain:31 expected: <true> but was:
<false>
[main] [ERROR] Run 3: TestSurefire2087.testPlain:31 expected: <true> but was:
<false>
[main] [INFO]
[main] [ERROR] demo.surefire2087.TestSurefire2087.testPlain
[main] [ERROR] Run 1: TestSurefire2087.testPlain:31 expected: <true> but was:
<false>
[main] [ERROR] Run 2: TestSurefire2087.testPlain:31 expected: <true> but was:
<false>
[main] [ERROR] Run 3: TestSurefire2087.testPlain:31 expected: <true> but was:
<false>
[main] [INFO]
[main] [INFO]
[main] [ERROR] Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
[main] [INFO]
[main] [INFO]
------------------------------------------------------------------------
[main] [INFO] BUILD FAILURE
[main] [INFO]
------------------------------------------------------------------------{code}
3. `testFlakey` (marked as flake, does NOT mark the build failed, but this is
expected and is OK)
{code:java}
[main] [INFO] --- surefire:3.0.0-M9:test (default-test) @ test-module ---
[main] [WARNING] Parameter 'systemProperties' is deprecated: Use
systemPropertyVariables instead.
[main] [INFO] Using auto detected provider
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[main] [INFO]
[main] [INFO] -------------------------------------------------------
[main] [INFO] T E S T S
[main] [INFO] -------------------------------------------------------
[ThreadedStreamConsumer] [INFO] Running demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped:
0, Time elapsed: 0.031 s <<< FAILURE! - in demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR] demo.surefire2087.TestSurefire2087.testFlakey
Time elapsed: 0.016 s <<< FAILURE!
...
[ThreadedStreamConsumer] [INFO] Running com.atlassian.jira.TestSurefire2087
[ThreadedStreamConsumer] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped:
0, Time elapsed: 0.001 s - in com.atlassian.jira.TestSurefire2087
[ThreadedStreamConsumer] [INFO] Running demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped:
0, Time elapsed: 0.001 s - in demo.surefire2087.TestSurefire2087
[main] [INFO]
[main] [INFO] Results:
[main] [INFO]
[main] [WARNING] Flakes:
[main] [WARNING] com.atlassian.jira.TestSurefire2087.testFlakey
[main] [ERROR] Run 1: TestSurefire2087.testFlakey:38 expected: <true> but
was: <false>
[main] [INFO] Run 2: PASS
[main] [INFO]
[main] [WARNING] demo.surefire2087.TestSurefire2087.testFlakey
[main] [ERROR] Run 1: TestSurefire2087.testFlakey:38 expected: <true> but
was: <false>
[main] [INFO] Run 2: PASS
[main] [INFO]
[main] [INFO]
[main] [WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2
...
[main] [INFO]
------------------------------------------------------------------------
[main] [INFO] BUILD SUCCESS
[main] [INFO]
------------------------------------------------------------------------{code}
4. `testDemoTemplate` (should fail but does NOT, marked as flake but one
parameterized execution always fails)
{code:java}
[main] [INFO] --- surefire:3.0.0-M9:test (default-test) @ test-module ---
...
[main] [INFO] -------------------------------------------------------
[main] [INFO] T E S T S
[main] [INFO] -------------------------------------------------------
[ThreadedStreamConsumer] [INFO] Running demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped:
0, Time elapsed: 0.048 s <<< FAILURE! - in demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR]
demo.surefire2087.TestSurefire2087.testDemoTemplate Time elapsed: 0.003 s <<<
FAILURE!
...
[main] [INFO]
[main] [INFO] Results:
[main] [INFO]
[main] [WARNING] Flakes:
[main] [WARNING] com.atlassian.jira.TestSurefire2087.testDemoTemplate
[main] [INFO] Run 1: PASS
[main] [ERROR] Run 2: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 3: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 4: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [INFO]
[main] [WARNING] demo.surefire2087.TestSurefire2087.testDemoTemplate
[main] [INFO] Run 1: PASS
[main] [ERROR] Run 2: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 3: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 4: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [INFO]
[main] [INFO]
[main] [WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2
...
[main] [INFO]
------------------------------------------------------------------------
[main] [INFO] BUILD SUCCESS
[main] [INFO]
------------------------------------------------------------------------{code}
If I apply the patch from my previous comment to tell apart different
parameterized runs of the same test, then it fails as expected:
{code:java}
[main] [INFO] --- surefire:3.0.0-M10-patch-vadam:test (default-test) @
test-module ---
[main] [WARNING] Parameter 'systemProperties' is deprecated: Use
systemPropertyVariables instead.
[main] [INFO] Using auto detected provider
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[main] [INFO]
[main] [INFO] -------------------------------------------------------
[main] [INFO] T E S T S
[main] [INFO] -------------------------------------------------------
[ThreadedStreamConsumer] [INFO] Running demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped:
0, Time elapsed: 0.046 s <<< FAILURE! - in demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR]
demo.surefire2087.TestSurefire2087.testDemoTemplate() demo2--[2] Time elapsed:
0.003 s <<< FAILURE!
...
[main] [INFO]
[main] [INFO] Results:
[main] [INFO]
[main] [ERROR] Failures:
[main] [ERROR] com.atlassian.jira.TestSurefire2087.testDemoTemplate() demo2--[2]
[main] [ERROR] Run 1: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 2: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 3: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [INFO]
[main] [ERROR] demo.surefire2087.TestSurefire2087.testDemoTemplate() demo2--[2]
[main] [ERROR] Run 1: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 2: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 3: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [INFO]
[main] [INFO]
[main] [ERROR] Tests run: 4, Failures: 2, Errors: 0, Skipped: 0
[main] [INFO]
[main] [INFO]
------------------------------------------------------------------------
[main] [INFO] BUILD FAILURE
[main] [INFO]
------------------------------------------------------------------------{code}
Can you please suggest a way forward here for fixing that? Would you prefer
reopening this issue, or opening a new one for TestTemplate specifically?
(unsure if there are more/other cases like that)
was (Author: vadam):
Can we please reopen this ticket to also address issues with other repeated
runs? (in our case `@TestTemplate`).
We've put together a test case like this:
{code:java}
package demo.surefire2087;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestTemplate;
import org.junit.jupiter.api.extension.ExtendWith;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
@ExtendWith(DemoTestTemplateInvocationContextProvider.class)
public class TestSurefire2087 {
private static boolean flake = true;
private int demoValue = 1;
@ParameterizedTest
@ValueSource(ints = {0, 1})
public void testParameterised(int i) {
assertTrue(i == 1);
}
@TestTemplate
public void testDemoTemplate() {
assertEquals(demoValue, 1);
}
@Test
public void testPlain() {
assertTrue(false);
}
@Test
public void testFlakey() {
if (flake) {
flake = false;
assertTrue(false);
}
assertTrue(true);
}
} {code}
Executing this with `mvn verify -Dsurefire.rerunFailingTestsCount=2
-Dtest=TestSurefire2087#<testMethod>` yields the following results on 3.0.0-M9:
1. `testParameterised` (fails as expected)
{code:java}
[main] [INFO] --- surefire:3.0.0-M9:test (default-test) @ test-module ---
[main] [INFO] Using auto detected provider
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[main] [INFO]
[main] [INFO] -------------------------------------------------------
[main] [INFO] T E S T S
[main] [INFO] -------------------------------------------------------
[ThreadedStreamConsumer] [INFO] Running demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped:
0, Time elapsed: 0.069 s <<< FAILURE! - in demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR]
demo.surefire2087.TestSurefire2087.testParameterised(int)[1] Time elapsed:
0.034 s <<< FAILURE!
...
[main] [INFO]
[main] [INFO] Results:
[main] [INFO]
[main] [ERROR] Failures:
[main] [ERROR] com.atlassian.jira.TestSurefire2087.testParameterised(int)[1]
[main] [ERROR] Run 1: TestSurefire2087.testParameterised:21 expected: <true>
but was: <false>
[main] [ERROR] Run 2: TestSurefire2087.testParameterised:21 expected: <true>
but was: <false>
[main] [ERROR] Run 3: TestSurefire2087.testParameterised:21 expected: <true>
but was: <false>
[main] [INFO]
[main] [ERROR] demo.surefire2087.TestSurefire2087.testParameterised(int)[1]
[main] [ERROR] Run 1: TestSurefire2087.testParameterised:21 expected: <true>
but was: <false>
[main] [ERROR] Run 2: TestSurefire2087.testParameterised:21 expected: <true>
but was: <false>
[main] [ERROR] Run 3: TestSurefire2087.testParameterised:21 expected: <true>
but was: <false>
[main] [INFO]
[main] [INFO]
[main] [ERROR] Tests run: 4, Failures: 2, Errors: 0, Skipped: 0
[main] [INFO]
[main] [INFO]
------------------------------------------------------------------------
[main] [INFO] BUILD FAILURE
[main] [INFO]
------------------------------------------------------------------------{code}
2. `testPlain` (fails as expected)
{code:java}
[main] [INFO] --- surefire:3.0.0-M9:test (default-test) @ test-module ---
[main] [WARNING] Parameter 'systemProperties' is deprecated: Use
systemPropertyVariables instead.
[main] [INFO] Using auto detected provider
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[main] [INFO]
[main] [INFO] -------------------------------------------------------
[main] [INFO] T E S T S
[main] [INFO] -------------------------------------------------------
[ThreadedStreamConsumer] [INFO] Running demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped:
0, Time elapsed: 0.034 s <<< FAILURE! - in demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR] demo.surefire2087.TestSurefire2087.testPlain
Time elapsed: 0.018 s <<< FAILURE!
...
[main] [INFO]
[main] [INFO] Results:
[main] [INFO]
[main] [ERROR] Failures:
[main] [ERROR] com.atlassian.jira.TestSurefire2087.testPlain
[main] [ERROR] Run 1: TestSurefire2087.testPlain:31 expected: <true> but was:
<false>
[main] [ERROR] Run 2: TestSurefire2087.testPlain:31 expected: <true> but was:
<false>
[main] [ERROR] Run 3: TestSurefire2087.testPlain:31 expected: <true> but was:
<false>
[main] [INFO]
[main] [ERROR] demo.surefire2087.TestSurefire2087.testPlain
[main] [ERROR] Run 1: TestSurefire2087.testPlain:31 expected: <true> but was:
<false>
[main] [ERROR] Run 2: TestSurefire2087.testPlain:31 expected: <true> but was:
<false>
[main] [ERROR] Run 3: TestSurefire2087.testPlain:31 expected: <true> but was:
<false>
[main] [INFO]
[main] [INFO]
[main] [ERROR] Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
[main] [INFO]
[main] [INFO]
------------------------------------------------------------------------
[main] [INFO] BUILD FAILURE
[main] [INFO]
------------------------------------------------------------------------{code}
3. `testFlakey` (marked as flake, does NOT mark the build failed, but this is
expected and is OK)
{code:java}
[main] [INFO] --- surefire:3.0.0-M9:test (default-test) @ test-module ---
[main] [WARNING] Parameter 'systemProperties' is deprecated: Use
systemPropertyVariables instead.
[main] [INFO] Using auto detected provider
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[main] [INFO]
[main] [INFO] -------------------------------------------------------
[main] [INFO] T E S T S
[main] [INFO] -------------------------------------------------------
[ThreadedStreamConsumer] [INFO] Running demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped:
0, Time elapsed: 0.031 s <<< FAILURE! - in demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR] demo.surefire2087.TestSurefire2087.testFlakey
Time elapsed: 0.016 s <<< FAILURE!
...
[ThreadedStreamConsumer] [INFO] Running com.atlassian.jira.TestSurefire2087
[ThreadedStreamConsumer] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped:
0, Time elapsed: 0.001 s - in com.atlassian.jira.TestSurefire2087
[ThreadedStreamConsumer] [INFO] Running demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped:
0, Time elapsed: 0.001 s - in demo.surefire2087.TestSurefire2087
[main] [INFO]
[main] [INFO] Results:
[main] [INFO]
[main] [WARNING] Flakes:
[main] [WARNING] com.atlassian.jira.TestSurefire2087.testFlakey
[main] [ERROR] Run 1: TestSurefire2087.testFlakey:38 expected: <true> but
was: <false>
[main] [INFO] Run 2: PASS
[main] [INFO]
[main] [WARNING] demo.surefire2087.TestSurefire2087.testFlakey
[main] [ERROR] Run 1: TestSurefire2087.testFlakey:38 expected: <true> but
was: <false>
[main] [INFO] Run 2: PASS
[main] [INFO]
[main] [INFO]
[main] [WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2
...
[main] [INFO]
------------------------------------------------------------------------
[main] [INFO] BUILD SUCCESS
[main] [INFO]
------------------------------------------------------------------------{code}
4. `testDemoTemplate` (should fail but does NOT, marked as flake but one
parameterized execution always fails)
{code:java}
[main] [INFO] --- surefire:3.0.0-M9:test (default-test) @ test-module ---
...
[main] [INFO] -------------------------------------------------------
[main] [INFO] T E S T S
[main] [INFO] -------------------------------------------------------
[ThreadedStreamConsumer] [INFO] Running demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped:
0, Time elapsed: 0.048 s <<< FAILURE! - in demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR]
demo.surefire2087.TestSurefire2087.testDemoTemplate Time elapsed: 0.003 s <<<
FAILURE!
...
[main] [INFO]
[main] [INFO] Results:
[main] [INFO]
[main] [WARNING] Flakes:
[main] [WARNING] com.atlassian.jira.TestSurefire2087.testDemoTemplate
[main] [INFO] Run 1: PASS
[main] [ERROR] Run 2: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 3: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 4: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [INFO]
[main] [WARNING] demo.surefire2087.TestSurefire2087.testDemoTemplate
[main] [INFO] Run 1: PASS
[main] [ERROR] Run 2: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 3: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 4: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [INFO]
[main] [INFO]
[main] [WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2
...
[main] [INFO]
------------------------------------------------------------------------
[main] [INFO] BUILD SUCCESS
[main] [INFO]
------------------------------------------------------------------------{code}
If I apply the patch from my previous comment to tell apart different
parameterized runs of the same test, then it fails as expected:
{code:java}
[main] [INFO] --- surefire:3.0.0-M10-patch-vadam:test (default-test) @
test-module ---
[main] [WARNING] Parameter 'systemProperties' is deprecated: Use
systemPropertyVariables instead.
[main] [INFO] Using auto detected provider
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[main] [INFO]
[main] [INFO] -------------------------------------------------------
[main] [INFO] T E S T S
[main] [INFO] -------------------------------------------------------
[ThreadedStreamConsumer] [INFO] Running demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped:
0, Time elapsed: 0.046 s <<< FAILURE! - in demo.surefire2087.TestSurefire2087
[ThreadedStreamConsumer] [ERROR]
demo.surefire2087.TestSurefire2087.testDemoTemplate() demo2--[2] Time elapsed:
0.003 s <<< FAILURE!
...
[main] [INFO]
[main] [INFO] Results:
[main] [INFO]
[main] [ERROR] Failures:
[main] [ERROR] com.atlassian.jira.TestSurefire2087.testDemoTemplate() demo2--[2]
[main] [ERROR] Run 1: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 2: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 3: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [INFO]
[main] [ERROR] demo.surefire2087.TestSurefire2087.testDemoTemplate() demo2--[2]
[main] [ERROR] Run 1: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 2: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [ERROR] Run 3: TestSurefire2087.testDemoTemplate:26 expected: <2> but
was: <1>
[main] [INFO]
[main] [INFO]
[main] [ERROR] Tests run: 4, Failures: 2, Errors: 0, Skipped: 0
[main] [INFO]
[main] [INFO]
------------------------------------------------------------------------
[main] [INFO] BUILD FAILURE
[main] [INFO]
------------------------------------------------------------------------{code}
Can you please suggest a way forward here for fixing that? Would you prefer
reopening this issue, or opening a new one for TestTemplate specifically?
(unsure if there are more/other cases like that)
> rerunFailingTestsCount incorrectly marks failed parametrized test as pass
> -------------------------------------------------------------------------
>
> Key: SUREFIRE-2087
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2087
> Project: Maven Surefire
> Issue Type: Bug
> Affects Versions: 3.0.0-M4
> Reporter: Ramanan Rajendran
> Priority: Major
>
> rerunFailingTestsCount is not working as expected with junit5
> `@ParameterizedTest`
> If 1st iteration of the test passes and the 2nd one fails, it still marks the
> test as flaky passed.
> Eg: In the following run, iteration 1 with input A passed but iteration 2
> with input B failed. Iteration 2 was rerun 2 times and failed in all the
> attempts. Instead of failing TestA, it's marked as flaky giving an overall
> pass for the test suite. I believe it's using the Run 1 PASS as a gate to
> mark the whole test as a pass.
>
> [INFO] Run 1: PASS
> [ERROR] Run 2: TestA.iteration2:101 1 expectation failed.
> Expected status code <400> doesn't match actual status code <401>.
> [ERROR] Run 3: TestA.iteration2:101 1 expectation failed.
> Expected status code <400> doesn't match actual status code <401>.
> [ERROR] Run 4: TestA.iteration2:101 1 expectation failed.
> <400> doesn't match actual status code <401>.
>
> Tests run: 23, Failures: 0, Errors: 0, Skipped: 2, Flakes: 1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
