Marek Polacek via Gcc-patches <gcc-patches@gcc.gnu.org> writes: > On Thu, Jul 30, 2020 at 11:54:23AM +0200, Jakub Jelinek via Gcc-patches wrote: >> On Tue, Jul 28, 2020 at 05:44:47PM -0400, Marek Polacek via Gcc-patches >> wrote: >> > We will still have a surfeit of bugs that we've given short shrift to, but >> > let's at least automate what we can. The initial addition of the relevant >> > old(-ish) tests won't of course happen automagically, but it's a price I'm >> > willing to pay. My goal here isn't merely to reduce the number of open >> > PRs; >> > it is to improve the testing of the compiler overall. >> > >> > Thoughts? >> >> Looks useful to me, but I'd think it might be desirable to use separate >> directories for those tests, so that it is more obvious that it is a >> different category of tests. Now that we use git, just using git mv >> to move them to another place once they are fixed for good (together with >> some dg-* directive tweaks) wouldn't be that much work later. >> >> So having gcc.dg/unfixed/ , g++.dg/unfixed/ , c-c++-common/unfixed/ >> and their torture/ suffixed variants (or better directory name for those)? > > Thanks. I was afraid that it would cause too much friction when you happen > to fix one of the unfixed tests: you will have to find the correct directory > to put the test in and perhaps even rename the test to avoid conflicts with > tests with the same name in the final destination. But it's also true that > git is much better at moving files, and the extra clarity might be worth the > occasional hassle. It would also make it easy to skip testing unfixed tests. > dg-ice tests are easy to spot/grep for, but accepts-invalid/rejects-valid are > a different story.
FWIW, I agree with Mike that it might be better to put tests in the location that they'd have normally. Some reasons: - As already mentioned, it'd give more stable names. That's not an issue for things like git log, but it does mean that comparing one set of test results with another gives a meaningful XFAIL->PASS transition rather than a “removed test” + “added test” transition. It's also more convenient when comparing different branches. - There are a lot of specialised .exp harnesses, and I don't think it would be a good idea to encourage all of them to have unfixed/ variants, or unfixed/ subdirectories. E.g. vect.exp is not something we'd want to duplicate or subclass, but there are others. - There's a temptation to remove empty directories/harnesses when they have no associated tests, so we might oscillate between having unfixed/ directories and not. (The alternative is for the directory or harness to stick around based on the fact that it was useful at some point in the past.) - It's inconsistent with existing tests that are already XFAILed for all targets. Having a separate directory for completely-XFAILed tests might create a requirement (or the impression of a requirement) to move tests to an unfixed/ directory when XFAILing them. - IMO it draws an artificial distinction between tests that are completely XFAILed on all targets and tests that are either partly XFAILed for all targets, or XFAILed only for some targets. Thanks, Richard
