Am Donnerstag, 5. November 2015 um 16:56:32, schrieb Guenter Milde <mi...@users.sf.net> > On 2015-11-05, Kornel Benko wrote: > > Am Donnerstag, 5. November 2015 um 12:43:12, schrieb Guenter Milde > > <mi...@users.sf.net> >
...
> > No, the tests do not have any special naming. As they are a subset of
> > revertedTests their name starts with 'INVERTED'.
> > Of course, it could be arranged.
>
> I was still explaining how my concept of "suspension" could work.
Ah, sorry.
> Arranging "suspended" tests to start with SUSPENDED would ensure the -R
> parameter normally skips them as well...
>
> > But one can use e.g.
> > #ctest -R someregex -LE suspended
> > to get the desired.
>
> OK
>
> ...
>
> >> However, this is basically what you currently do with "inverted tests"
> >> there is basically just a different naming scheme.
>
> >> The "measure for deviation from a clean state" would be the number of
> >> "suspended" tests with my proposal and "inverted" tests in yours.
>
> >> The only difference are export tests we want to return an error.
> >> How are they managed in your scheme?
>
> > I am not sure I understand.
>
> > ATM export tests are expected to pass. All other tests are coverer be
> > revertedTests. We (Scott and me) would add them to revertedTests.
>
> I see that there is one more ambiguity:
>
> * I divide LyX testing into
>
> unit tests:
> testing specific functions and classes
>
> currently missing
We will see, how it evolves.
> functional tests:
> testing processing (import/export) against a known reference
>
> currently implemented as tex2lyx tests
There is also a label "lyx2lyx", but I agree that are only rudimentary tests.
> export tests:
> testing the export status of sample documents (most of them taken from
> documentation and examples, i.e. with a different main purpose).
>
> * In contrast, the label "export" inside the "automatic export test framework"
> applies only to a subset of test cases.
> The others are "inverted/reverted" or "ignored".
>
> -> "export test" can be either the complete set of the "autotests" or just
> the subset where we currently expect the export to pass.
>
> I understand that this naming is historically motivated und "natural" for
> the ones that work with the automatic tests from start. However, it makes
> understanding what is going on complicated for newbies.
>
>
> The next problem is, that we must be very careful when speaking about a
> test that fails:
>
> -> inverted tests pass, when the export fails (i.e. returns an error)
>
>
> So I'll try to re-state my question:
>
> I would like to have a simple number measuring the "cleanness" of LyX
> code and shipped documents. For this, I propose to use the number
> of test cases that do not give the "correct" result.
>
> With the current implementation, how can I distinguish:
>
> 1. correct results:
>
> a) tests where the export should succeed and really does
>
> b) tests where the export fails for "good resons" (i.e. correct error
> reporting for "impossible" actions)
Yes.
> 2. wrong results:
>
> a) tests where the export should succeed but fails
There is no such test known. But of course, might exist. If it fails, then
something in
our preparing lyx file for export is wrong (perl script
useSystemFonts.pl+lyxStatus.pm)
> b) tests where the export does not report an error but we know it should
> (because the exported document is corrupt).
This is the hard part. There are so many tests, that no one had time (or
pleasure) to check them all visually.
> As far as I understand, "revertedTests" reflect the currently expected
> result, not the desired result. The number of inverted tests includes
> 1b) and 2a).
>
> Conclusion:
>
> The result of running the tests is good at telling about regressions or
> "surfaced bugs" (i.e. old problems that lead to export errors after changing
> either code or sample documents).
Yes.
> However, neither the number of "failing tests", nor the number of
> "inverted" tests can be used to tell how "good" LyX currently is.
>
Yes.
> Günter
Kornel
signature.asc
Description: This is a digitally signed message part.
