On Wed, Aug 05, 2020 at 08:58:40AM +0100, Richard Sandiford wrote: > Marek Polacek via Gcc-patches <gcc-patches@gcc.gnu.org> writes: > > On Thu, Jul 30, 2020 at 11:54:23AM +0200, Jakub Jelinek via Gcc-patches > > wrote: > >> On Tue, Jul 28, 2020 at 05:44:47PM -0400, Marek Polacek via Gcc-patches > >> wrote: > >> > We will still have a surfeit of bugs that we've given short shrift to, > >> > but > >> > let's at least automate what we can. The initial addition of the > >> > relevant > >> > old(-ish) tests won't of course happen automagically, but it's a price > >> > I'm > >> > willing to pay. My goal here isn't merely to reduce the number of open > >> > PRs; > >> > it is to improve the testing of the compiler overall. > >> > > >> > Thoughts? > >> > >> Looks useful to me, but I'd think it might be desirable to use separate > >> directories for those tests, so that it is more obvious that it is a > >> different category of tests. Now that we use git, just using git mv > >> to move them to another place once they are fixed for good (together with > >> some dg-* directive tweaks) wouldn't be that much work later. > >> > >> So having gcc.dg/unfixed/ , g++.dg/unfixed/ , c-c++-common/unfixed/ > >> and their torture/ suffixed variants (or better directory name for those)? > > > > Thanks. I was afraid that it would cause too much friction when you happen > > to fix one of the unfixed tests: you will have to find the correct directory > > to put the test in and perhaps even rename the test to avoid conflicts with > > tests with the same name in the final destination. But it's also true that > > git is much better at moving files, and the extra clarity might be worth the > > occasional hassle. It would also make it easy to skip testing unfixed > > tests. > > dg-ice tests are easy to spot/grep for, but accepts-invalid/rejects-valid > > are > > a different story. > > FWIW, I agree with Mike that it might be better to put tests in the > location that they'd have normally. Some reasons: > > - As already mentioned, it'd give more stable names. That's not an > issue for things like git log, but it does mean that comparing one > set of test results with another gives a meaningful XFAIL->PASS > transition rather than a “removed test” + “added test” transition. > It's also more convenient when comparing different branches. > > - There are a lot of specialised .exp harnesses, and I don't think it > would be a good idea to encourage all of them to have unfixed/ > variants, or unfixed/ subdirectories. E.g. vect.exp is not something > we'd want to duplicate or subclass, but there are others. > > - There's a temptation to remove empty directories/harnesses when > they have no associated tests, so we might oscillate between having > unfixed/ directories and not. (The alternative is for the directory > or harness to stick around based on the fact that it was useful at > some point in the past.) > > - It's inconsistent with existing tests that are already XFAILed for all > targets. Having a separate directory for completely-XFAILed tests > might create a requirement (or the impression of a requirement) to > move tests to an unfixed/ directory when XFAILing them. > > - IMO it draws an artificial distinction between tests that are > completely XFAILed on all targets and tests that are either partly > XFAILed for all targets, or XFAILed only for some targets.
Those are all great points. I have to reconsider my position once again ;-). Perhaps we should add an // UNFIXED comment then to easily disambiguate between tests that are not fixed. But I fear that people won't always use this consistently. Marek
