Am Mittwoch, 4. November 2015 um 09:30:03, schrieb Guenter Milde <mi...@users.sf.net> > On 2015-11-02, Kornel Benko wrote: > > Am 2. November 2015 um 08:36:05, schrieb Guenter Milde <mi...@users.sf.net> > >> On 2015-11-02, Scott Kostyshak wrote: > >> > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote: > >> >> Am 1. November 2015 um 21:27:11, schrieb Guenter Milde > > >> >> > Could we introduce a new status "suspended" - meaning just skip the > >> >> > test until a known bug is solved as we know the result is > >> >> > insignificant until then? > > >> >> We already have such (specified in file "ignoredTests"). But as this > >> >> tests are never executed, nobody cares for them anymore. > >> >> The tests here are such, that we know, we never resolve them. > > But even here, there are two cases: > > a) cases that fail for a good reason and should always fail > (e.g. export document with non-TeX fonts using 8-bit LaTeX) > > b) cases that should not fail but do (for reasons we cannot change). > > While a) should be "permanently inverted" (to give a warning if a change > makes the export pass but we expect the result to be faulty), b) is > correctly placed in "ignored" -- the test case does not help detect LyX > buts in any way. > > > >> Suggestion: > > >> Specified in file "suspendedTests") with the reason for suspending > >> (bug report, commit that turned hidden problems into export failure, ...) > > >> These tests are normally skipped, but they are not forgotten. > > >> The tests here are such, that we know, we can resolve them but their > >> failure is a minor issue that can be postponed (comparable to enhancement > >> requests in Trac). > > > >> Candidates for "suspended" tests are > > >> * "fragile" documents (many packages, ERT, heavy preamble, ...), > >> * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), > >> * non-default export routes > > >> and especially combinations of these. > > > > OK, here is my suggestion > > 1.) We add the appropriate tests into revertedTests > > Why? This would undo one important benefit of "suspended" tests:
I was not clear about this. These tests will not be run with e.g. 'ctest -L
export' (see the '-L' param).
In this sense we get 'clean' export tests.
> >> Suspending instead of reverting also frees us from the need to reassess
> >> them if a change in the "real life documents" or a fix makes them pass
> >> again. Instead, they could/should be revised at a time we have fixed
> >> major known problems and no release pressure...
>
> I'd only add the test to "revertedTests", too, if it is established that
> the correct result for this test is a failure.
>
> > The content in suspendedTests may be (in our case) e.g.
> > pdf4_texF
>
> > 1.a)
> > If a test is to be inverted, we check suspendedTest,
> > and if there, we ignore the testcase.
>
> Here, I'd change the priority:
>
> * with a normal run, all suspended tests are ignored,
>
> * with "run suspended tests" or "include suspended tests",
> suspended test are run, taking into consideration their
> "inversion state".
>
> > or
> > 1.b)
> > Such a test may get a label "suspended" instead of
> > "export", so that 'ctest -L export' will be clean, but
> > we also have the possibility to use 'ctest -L
> > suspended'.
>
> > This does neither effect non-inverted nor ignored.
>
> My suggestion would affect non-inverted tests (the ones with the label
> "suspended").
>
> This means for failing test you will have 3 options:
>
> 1. if failing is the expected outcome: invert
> 2. if failing for permanent reasons that are none of our business: ignore
> 3. if failing for minor reasons or a known bug: suspend
>
> Motivation:
>
> Some bugs only lead to failures depending on document content. We do
> "road testing" with real life documents (manuals, examples). Work on the
> documents could easily change the export status of tests without any
> progress on fixing the underlying problem! (Examples: adding/removing a
> character not available with non-default export route, adding non-ASCII
> character to the PDF Header Information with "inputenc=ascii",
> Writing my name as G\"unter in the PDF Header Information before solving
> #9830, ...)
>
> However, editing manuals or examples should not require immediate
> revision of test cases using this document if the basic problem is not
> solved.
So what do you propose for such tests (84% of export tests)?
ctest -L export -N | wc => 3719
ctest -L export -N |egrep '/(doc|examples)/'| wc => 3153
3153 / 3719 => 84.78%
> OTOH, if bugs that triggered suspension of some test cases are solved,
> it is time to revisit the set of suspended tests, reactivate the ones
> that pass and re-label the ones that still fail.
>
>
>
> > I prefer 1.b.
>
> Whatever works best in praxi.
We will see.
For now committed 1.b at 538228d
> Günter
Kornel
signature.asc
Description: This is a digitally signed message part.
