[Lldb-commits] [PATCH] D127258: [lldb] Mark API tests as XFAIL if they have expected failures and no passing tests

[Jonas Devlieghere via Phabricator via lldb-commits]

JDevlieghere created this revision.
JDevlieghere added reviewers: aprantl, mib.
Herald added a project: All.
JDevlieghere requested review of this revision.

Currently, when an API test has no passing tests but at least one unsupported 
test, we report the test as unsupported. This patch slightly changes that 
behavior for tests that do not contain any passing tests, but contain both 
unsupported tests and expected failures. In that case, I think it's more useful 
to report the test as an expected failure rather than being unsupported.

For example, the output below now result in an "expected failure" rather than 
"unsupported":

  RESULT: PASSED (0 passes, 0 failures, 0 errors, 1 skipped, 3 expected 
failures, 0 unexpected successes)


https://reviews.llvm.org/D127258

Files:
  lldb/test/API/lldbtest.py


Index: lldb/test/API/lldbtest.py
===================================================================
--- lldb/test/API/lldbtest.py
+++ lldb/test/API/lldbtest.py
@@ -95,8 +95,12 @@
             # Otherwise this is just a failure.
             return lit.Test.FAIL, output
 
-        has_unsupported_tests = 'UNSUPPORTED:' in out or 'UNSUPPORTED:' in err
         has_passing_tests = 'PASS:' in out or 'PASS:' in err
+        has_expected_failures = 'XFAIL:' in out or 'XFAIL:' in err
+        if has_expected_failures and not has_passing_tests:
+            return lit.Test.XFAIL, output
+
+        has_unsupported_tests = 'UNSUPPORTED:' in out or 'UNSUPPORTED:' in err
         if has_unsupported_tests and not has_passing_tests:
             return lit.Test.UNSUPPORTED, output
 



Index: lldb/test/API/lldbtest.py
===================================================================
--- lldb/test/API/lldbtest.py
+++ lldb/test/API/lldbtest.py
@@ -95,8 +95,12 @@
             # Otherwise this is just a failure.
             return lit.Test.FAIL, output
 
-        has_unsupported_tests = 'UNSUPPORTED:' in out or 'UNSUPPORTED:' in err
         has_passing_tests = 'PASS:' in out or 'PASS:' in err
+        has_expected_failures = 'XFAIL:' in out or 'XFAIL:' in err
+        if has_expected_failures and not has_passing_tests:
+            return lit.Test.XFAIL, output
+
+        has_unsupported_tests = 'UNSUPPORTED:' in out or 'UNSUPPORTED:' in err
         if has_unsupported_tests and not has_passing_tests:
             return lit.Test.UNSUPPORTED, output
 

_______________________________________________
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits