Quoting Ilia Mirkin (2016-05-10 17:45:49) > On Tue, May 10, 2016 at 8:05 PM, Dylan Baker <[email protected]> wrote: > > Quoting Ilia Mirkin (2016-05-10 14:32:59) > >> On Tue, May 10, 2016 at 5:32 PM, Dylan Baker <[email protected]> wrote: > >> > Quoting Ilia Mirkin (2016-05-10 06:41:33) > >> >> Shouldn't the test's result just be included into the "max" no matter > >> >> what? > >> >> > >> > > >> > Well, the problem is that a lot of tests were retrofitted with subtests, > >> > and now the overall status is bunk, and for new tests we generally don't > >> > add an overall result when there's subtests. > >> > >> It's surprising if the status of a test says fail but piglit-summary > >> reports it as pass. In the case where one of the subtests fails but > >> the overall status is pass, this would still say fail, no? > >> > >> What situation are you protecting against by only incorporating the > >> overall status on crash? > > > > I have this gut feeling that says skip or notrun might play havok here. > > But I guess I should look test it and see if that's the case. > > > > The other thing I guess is that there's only about 100 tests (or, c/c++ > > files, really) using subtests. So maybe it would just be easier to audit > > them to make sure that they do the right thing and just add the overall > > result to the max call. > > I dunno... I think in at least some cases some "stuff" happens between > subtests that can fail. I don't think it's correct to look at a test > as a shell for a bunch of disconnected subtests, and thus anything the > shell itself does is meaningless. The shell can also fail. Feel free > to audit everything :) > > -ilia
You know. Thinking back there was a point where the framework defaulted to FAIL instead of NOTRUN. This might be an artifact of that time. Either way I guess just seeing what happens is the easiest way to figure out if it's going to work or not
signature.asc
Description: signature
_______________________________________________ Piglit mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/piglit
