JDevlieghere created this revision. JDevlieghere added reviewers: labath, davide, jingham, clayborg. JDevlieghere added a project: LLDB.
Unexpected successes should be considered failures because they can hide regressions when not addressed. When a test is fixed and not re-enabled, it can easily regress without us noticing. I couldn't find a good way to make this change other than changing it in the unittest2 framework. I know this is less than optimal but since we have the dependency checked in and the change is pretty fundamental to the framework I think it's not unreasonable. Repository: rLLDB LLDB https://reviews.llvm.org/D55835 Files: lit/Suite/lldbtest.py third_party/Python/module/unittest2/unittest2/result.py Index: third_party/Python/module/unittest2/unittest2/result.py =================================================================== --- third_party/Python/module/unittest2/unittest2/result.py +++ third_party/Python/module/unittest2/unittest2/result.py @@ -148,7 +148,9 @@ def wasSuccessful(self): "Tells whether or not this result was a success" - return (len(self.failures) + len(self.errors) == 0) + return (len(self.failures) + + len(self.errors) + + len(self.unexpectedSuccesses) == 0) def stop(self): "Indicates that the tests should be aborted" Index: lit/Suite/lldbtest.py =================================================================== --- lit/Suite/lldbtest.py +++ lit/Suite/lldbtest.py @@ -94,11 +94,10 @@ litConfig.maxIndividualTestTime)) if exitCode: - return lit.Test.FAIL, out + err - - unexpected_test_line = 'XPASS' - if unexpected_test_line in out or unexpected_test_line in err: - return lit.Test.XPASS, '' + if 'FAIL' in out or 'FAIL' in err: + return lit.Test.FAIL, out + err + if 'XPASS' in out or 'XPASS' in err: + return lit.Test.XPASS, out + err passing_test_line = 'RESULT: PASSED' if passing_test_line not in out and passing_test_line not in err:
Index: third_party/Python/module/unittest2/unittest2/result.py =================================================================== --- third_party/Python/module/unittest2/unittest2/result.py +++ third_party/Python/module/unittest2/unittest2/result.py @@ -148,7 +148,9 @@ def wasSuccessful(self): "Tells whether or not this result was a success" - return (len(self.failures) + len(self.errors) == 0) + return (len(self.failures) + + len(self.errors) + + len(self.unexpectedSuccesses) == 0) def stop(self): "Indicates that the tests should be aborted" Index: lit/Suite/lldbtest.py =================================================================== --- lit/Suite/lldbtest.py +++ lit/Suite/lldbtest.py @@ -94,11 +94,10 @@ litConfig.maxIndividualTestTime)) if exitCode: - return lit.Test.FAIL, out + err - - unexpected_test_line = 'XPASS' - if unexpected_test_line in out or unexpected_test_line in err: - return lit.Test.XPASS, '' + if 'FAIL' in out or 'FAIL' in err: + return lit.Test.FAIL, out + err + if 'XPASS' in out or 'XPASS' in err: + return lit.Test.XPASS, out + err passing_test_line = 'RESULT: PASSED' if passing_test_line not in out and passing_test_line not in err:
_______________________________________________ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits