On Tue, Aug 04, 2020 at 03:33:23PM -0700, Mike Stump wrote:
> I think the read of the room is that people think it would be generally
> useful, so let approve the general plan.
Cool.
> So, now we are down to the fine details. Please do see just how far you can
> stretch the existing mechanisms to cover what you need to do. I think the
> existing mechanisms should be able to cover it all; but the devil is in the
> details and those matter.
At this point I'm only proposing one new directive, dg-ice. I think we can't
really do without it. The other one was a matter of convenience.
> For the suggestion to isolate the tests into their own area easily
> distinguished by filename, I think it is better to choose an original home
> for them using the existing naming scheme as much as possible, that way when
> fixed, they are already in the right spot. We in theory can move them
> around, but, there is a beauty in having a long term stable name that just
> doesn't change any. You can look through time to see all state changes. git
> log file.C, show you all the history nicely, and so on.
What you say makes sense. I'm still pretty wishy-washy about this.
But since git tracks renaming files well (you see the whole history of the
renamed file, even with its old name), I'm currently of the mind that having
a dedicated directory is preferable.
> You can experiment with dg-prms-id and see if that lets you tag in bug report
> numbers in a more meaningful way. Anyway, would be good to always include
> the bug report number in the test case, and in the bug report, add the name
> of the added test case (so others don't add yet another instance of the bug).
Absolutely, the PR number should be in the test, and the monitored PR ought
to say what test it's covered by.
I've looked at dg-prms-id in dejagnu, but I don't readily see how that
could help us.
> So with that as a backdrop, I think it's reasonable to self-approve additions
> of this sort to the test suite. If you have test cases that can go in with
> existing mechanisms, feel free to start adding them in.
Sounds good. As I mentioned, they should be of high quality, just like the
test we normally add when fixing bugs.
> As you find it difficult to express a test using the existing mechanisms,
> let's talk about those and see if anyone has a good idea on how to express
> it. I think ICEs are the most annoying to manage, but, I think excess and
> prune should be able to handle them. I think should get an error or warning,
> or should not get an error or warning are more trivial to manage.
I experimented with
// { dg-prune-output ".*internal compiler error.*" }
// { dg-xfail-if "" { *-*-* } }
but it's a mouthful and the results were poor (when the ICE is fixed but we
generate errors instead). dg-ice is convenient, handles even the different
kind of ICE (when the diagnostic routines were re-entered), and generates
nice XPASSes when the ICE goes away.
I've also played games with dg-regexp but it was too ugly.
(I honestly don't see why new directives are such a big deal, if they're
properly documented.)
> A word of caution, if we produce core files, before you add tons of core file
> producing test cases, you'll want to submit a, ulimit -c 0 patch that can
> avoid the issue. corefile writing is slow and consumes disk. I can't recall
> at the moment if the current infrastructure will reliably avoid core files.
I thought we'd already set ulimit -c 0, but I don't see that now. I definitely
agree that we don't want core dumps. It probably needs some hack that sets
ulimit -c to 0 when we're running a test in */unfixed/. :/ Which also argues
for a separate directory for unfixed tests.
> If test cases infinitely loop, timeout, consume all available ram, fill the
> disk, crash the host machine, or do other really nasty stuff, please, let's
> avoid those for now. It is mazing host fast testing goes on a 200 thread
> count box, and how slow it can go if a single test case needs to time out.
> If you start with the idea that any individual test case should only take
> 2-10 seconds, you won't go wrong.
I agree, but that should be true for all tests. Really, only the expect-ice
tests are novel.
Marek
