On 7/16/21 9:32 AM, Thomas Schwinge wrote:
[much snipped] Of course, we shall assume a certain level of quality in the XFAILed test cases: I'm certainly not suggesting we put any random junk into the testsuite, coarsely XFAILed. (I have not reviewed Sandra's test cases to that effect, but knowing here, I'd be surprised if that were the problem here.)
FWIW, Tobias already did an extensive review of an early version of the testsuite patches in question and pointed out several cases where failures were due to my misunderstanding of the language standard or general confusion about what the expected behavior was supposed to be when gfortran wasn't implementing it or was tripping over other bugs. :-S I hope I incorporated all his suggestions and rewrote the previously-bogus tests to be more useful for the version I posted for review on the Fortran list, but shouldn't the normal patch review process be adequate to take care of any additional concerns about quality?
My previous understanding of the development process and testsuite conventions is that adding tests that FAIL is bad, but XFAILing them with reference to a PR is OK, and certainly much better than simply not having test coverage of those things at all. Especially in the case of something like the TS29113 testsuite where the explicit goal is to track standards compliance and/or the completeness of the existing implementation. :-S So it seems to me rather surprising to take the position that we should not be committing any new test cases that need to be XFAILed. :-S
-Sandra
