Am Donnerstag, 5. November 2015 um 12:43:12, schrieb Guenter Milde <mi...@users.sf.net> > Dear Kornel, > > thank you for the explanation. I come a bit nearer to a generic > understanding now. > > On 2015-11-05, Kornel Benko wrote: > > Am Donnerstag, 5. November 2015 um 07:34:47, schrieb Guenter Milde > > <mi...@users.sf.net> > > > ... > >> > It is suspended _only_ if you select testcases with the '-L' parameter. > > >> OK. My idea was that suspended testcases are skipped by default. > > > Let me sketch our use of '-L' parameter. > > > We have a number of lyx-files under lib/doc, lib/examples etc directories. > > There are export types like lyx16, xhtml, pdf, dvi etc > > TeX or non TeX fonts. > > For each combination we create a testname containing a hint to > > a.) the relative path of the lyx file (without extension) > > b.) export type (restricted to the specified output format(in > > lyx-file), if not 'default') > > c.) TeX or non TeX font (texF, systemF) > > (Is there a documentation of the testnames? If not, could this be added to > Development.lyx, maybe or in a README for the test machinery? With examples?) >
I was too lazy. (But probably I try to omit documenting because of lack of
enough English knowledge)
> Then: for my concept of "suspension":
>
> > If the name
>
> > i. matches regex in ignoreTests, the testcase is discarded. Handle to
> > next testname.
>
> ii. matches regex in suspendedTests: Prepend SUSPENDED to testname. If
> there can be more than one label, the test gets the label "suspended".
>
> iii. Not matching revertedTests, the test gets the label "export".
> Handle next testname.
>
> > (from here on it matches revertedTests)
> iv. Prepend INVERTED... to testname. Revert test verification.
>
>
> > Now calling ctest with '-L export' selects tests with label 'export',
> > thus skipping suspended tests. The same is valid for '-L reverted'.
>
> > The other possibilities, e.g. using '-R' parameter, does only check for
> > testnames. But the suspended tests must also have a name, so we cannot
> > skip them automatically.
>
> Then, with a regexp not starting with SUSPENDED, -R would not include
> suspended tests, right?
No, the tests do not have any special naming. As they are a subset of
revertedTests their name
starts with 'INVERTED'.
Of course, it could be arranged.
But one can use e.g.
#ctest -R someregex -LE suspended
to get the desired.
> > This should answer also your remark in a previous post (
> > > So what do you propose for such tests (84% of export tests)?
> > > ctest -L export -N | wc => 3719
> > > ctest -L export -N |egrep '/(doc|examples)/'| wc => 3153
> > > 3153 / 3719 => 84.78%
>
> >> I don't have any clue which tests are hidden behind these commands.
>
> This means we have about 3000 export tests with "real life" documents in doc
> and examples?
>
> Basically, I would look at currently failing tests
> (i.e. return value != expected return value) and
>
> * invert/uninvert test based on the "correct" return value
I expect the suspended to pass (120 cases). If some of them fails (good news),
it will be the minority
listed by ctest.
This is really easier to check.
> * suspend test failing for a known reason, so they are not run nor reported
> by a "normal" run.
>
> However, this is basically what you currently do with "inverted tests" there
> is basically just a different naming scheme.
>
> The "measure for deviation from a clean state" would be the number of
> "suspended" tests with my proposal and "inverted" tests in yours.
>
> The only difference are export tests we want to return an error.
> How are they managed in your scheme?
I am not sure I understand.
ATM export tests are expected to pass. All other tests are coverer be
revertedTests.
We (Scott and me) would add them to revertedTests.
> Günter
Kornel
signature.asc
Description: This is a digitally signed message part.
