![]() |
|
|
|
|
|
|
Change By:
|
Julien Carsique
(10/Sep/14 11:34 PM)
|
|
Summary:
|
Maven Job use of testFailureIgnore option should be configurable
or avoided
|
|
Issue Type:
|
Bug
Improvement
|
|
Description:
|
I understood from various emails, issues, discussions and source code that it is a wanted behavior to set a build UNSTABLE when (surefire or failsafe) tests fail. That is consistent with
Following the Jenkins [Terminology|
https://wiki.jenkins-ci.org/display/JENKINS/Terminology
However
]
,
when (surefire or failsafe) tests fail,
the
current behavior
Jenkins build status must be UNSTABLE:
<< A build
is
error-prone
unstable if it was built successfully
and
inconsistent with a command line
one
or
Freestyle job Maven execution
more publishers report it unstable
.
I guess that
For example
if the
UNSTABLE status was that much important, it would have been implemented on Freestyle jobs too. Especially since Freestyle are so often recommended rather than the useful Maven job.
h3. Problem
First main issue
JUnit publisher
is
configured and a test fails then
the build
continuation
will be marked unstable
.
>>
Second issue is the behavior difference between Maven and Freestyle jobs.
h4.
Freestyle
Maven
job
The result
That
is the
same as an execution from command line
default case on Maven jobs; the code of https
:
//github.com/jenkinsci/maven
-
Maven execution is interrupted
plugin/blob/maven
-
Maven build is FAILURE
plugin
-
Jenkins build is FAILURE
Reproducing the Maven job behavior is partially possible with {{-Dmaven
2
.
test
6/src/main/java/hudson/maven/reporters/SurefireArchiver
.
failure.ignore=true}} option and
java#L88 uses
the
use of 'Publish JUnit test result report' post build step (to change the build result from SUCCESS to UNSTABLE).
h4. Maven job
The result is similar to using the
{{-Dmaven.test.failure.ignore=true}} option:
- *the Maven execution is not interrupted
by a test failure
*
- Maven build is SUCCESS
- Jenkins build is UNSTABLE
Responsible code
However, that
is
there: https://github
a few error-prone and inconsistent with the command line or Freestyle job Maven execution
.
com/jenkinsci/maven
That would be highlighted by a Jenkins option to switch on or off that feature, rather than the need to know one must pass {{
-
plugin/blob/maven-plugin-2
Dmaven
.
6/src/main/java/hudson/maven/reporters/SurefireArchiver
test
.
java#L88
failure.ignore=false}} to deactivate it.
The
main issue is
error-prone aspect comes from
the build continuation, which will for instance install the artifacts:
{noformat}
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ test-jenkins-failures ---
[ERROR] There are test failures.
[INFO] --- maven-failsafe-plugin:2.17:integration-test (default) @ test-jenkins-failures ---
[INFO] --- maven-failsafe-plugin:2.17:verify (default) @ test-jenkins-failures ---
[ERROR] There are test failures.
[INFO] --- maven-install-plugin:2.5.1:install (default-install) @ test-jenkins-failures ---
[INFO] Installing ...
[INFO] BUILD SUCCESS
Finished: UNSTABLE{noformat}
h3. Solutions
h4. Add an option for ignoring
So, it is recommended in a Maven job to call the {{package} phase rather than {{install}}
or
not
{{deploy}} and use
the
test failures during the
post-
build
steps to deploy the artifacts at the end (the new deployAtEnd Maven option is an alternative)
.
An option
It
would
highlight that behavior and allow to easily switch from
be nice if there was
a
testFailureIgnore mode
way
to
a standard mode.
h4. Rather
alternatively
use
for instance the
{{-fae}} Maven 3 option
, or improve current implementation not to use the testFailureIgnore property hack
{noformat}-fae,--fail-at-end
("
Only fail the build afterwards; allow all non-impacted builds to continue
{noformat}
That would not be the exact equivalent to the current implementation but would still allow to get a status on multiple modules and would only require to
"
catch"
) and get
the
Maven failure at the end of the build.
Having a
final Maven build FAILED translated to Jenkins build UNSTABLE
would avoid the unwanted execution of other plugins after surefire or failsafe detected the failure
.
..
h4.
Consistency with
Freestyle job
For consistency purpose
About the Maven build step in Freestyle jobs
,
they do not behave that way by default. The result is
the
same as an execution from command line:
- Maven execution is interrupted by tests failures
- Maven build is FAILURE
- Jenkins build is FAILURE
Reproducing the Maven job behavior is possible with the {{-Dmaven.test.failure.ignore=true}} option and the use of
'Publish JUnit test result report' post build step
should
(to change the build result from SUCCESS to UNSTABLE).
As for the Maven job, it would
be
great if the 'Publish JUnit test result report' post build step would be
able to switch the Jenkins build result from FAILURE to UNSTABLE when the BUILD failure is only due to Maven
tests failure
test failures
.
|
|
|
|
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
|
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit
https://groups.google.com/d/optout.
EDIT
Changed from bug to improvement: that is not a bug. Changed the title and description. My mistake came from the following output in the console after having encountered false-positive errors due to the TASK and DRY plugins upgrade (I first though it was another similar issue):