Dear Kornel,
thank you for the fast reply.
On 2015-12-02, Kornel Benko wrote:
> Am, 2. Dezember 2015 um 12:20:14, schrieb Guenter Milde <mi...@users.sf.net>
>> On 2015-12-01, Kornel Benko wrote:
...
>> When tagging tests that fail,
>> we want tags for
>> a) the reason of failure (TexBug, LyxBug, incompatibility, nonstandard, ...)
>> b) our treatment of the test case (ignored, inverted, suspended, ...)
> Both.
Fine.
...
When I issue
>> #> ctest --print-label,
I get a long list of all existing labels in the ctest-autotests.
It would make it easier to work with the labels, if we write the
complete "hierarchy" into sublabels.
> ... In this scenario 'export:reverted:chemgreek' would be a label,
> while still possible to select ' -L chemgreek'.
Exactly. 'export:reverted:chemgreek' could be selected by '-L export',
by '-L reverted' and any other regular expression that matches, too.
> Implementation problem is sorting of labels.
> ('export:unreliable:erratic' vs 'export:erratic:unreliable', (bad
> example, but you get the idea)) From our POV it is the same label, from
> ctests POV they are different.
>From my POV, they are different, too: The scheme is
<class-label>:<label>:<sublabel>:<subsublabel>...
In the example, as erratic is a sublabel of unreliable only
'export:unreliable:erratic' is correct.
> Something like depth-flag for a label would be good. Sorting labels
> with the same depth could be done alphabetically.
Besides the hierarchical labels, there can be "orthogonal" ones (i.e.
outside the hierarchy, like "unstable" (proposed below).
For these, no sorting is required.
>> ........................................................................
>> When creating the tests, I propose the following logic:
>> 1. filter: sort out test cases to ignore
>> ignoredTests -> Export combinations matching here are withdrawn
>> 2. categorization: label the remaining tests
>> a) *all* export tests get the label "export" to make them easy to
>> distinguish from lyx2lyx, roundtrip, mathmacros, ...
> Unhappy about this although it feels logical.
When we add later sublabels to "export", it is still easy to select the "good"
export tests via '-L "export$"'.
- b) Set specific labels: test names are matched against regexps in
+ b) Add sublabels: test names are matched against regexps in
>> all "*Tests" files.
>> Matches regexp in Label Sublabels
>> ================= ========== ========================================
>> InvertedTests¹ inverted² wontfix, minor, TeXBug, assertError, ...
>> UnreliableTests unreliable nonstandard, ...
>> SuspendedTests¹ suspended
A test matching a regexp under the sublabel "minor" in
InvertedTests would get the label "export:inverted:minor".
Unlikely but not impossible:
>> A test matching a regexp in UnreliableTests and
- InvertedTests, say, would get the labels "unreliable" and "inverted".
+ InvertedTests, say, would get the labels "export:unreliable"
+ and "export:inverted".
>> .....................................................................
>> Choice of names:
Pattern files:
The files with the regular expressions are currently named using the
scheme "<label>Tests". However, strictly speaking they do not list
inverted tests (etc.) but patterns which are used to determine which tests
should get the label. A naming scheme like "tests-to-<label>" would make
this function more clear.
>> suspicious / inverted/reverted
...
>> Is there a reason the label cannot be called "inverted"?
> No valid reason.
Could you rename the label to "inverted" then, please?
For the pattern, I suggest "tests-to-invert", but maybe there is a better
name...
>> .....................................................................
suspended
>> The effect of this label is that if a "suspended" test works again, there
>> is no feedback when running inverted export tests. Anything else?
>> I don't see the advantage.
> The tests _can_ be done if using the label 'suspended'. In this sense
> they behave like what you wanted for 'ignore'.
I see.
>> For problems that are unlikely to be fixed soon, I propose one of
>> inverted:minor # problems that may be eventually solved
>> inverted:wontfix # problems that
>> # - minor and are hard to solve (not worth the effort),
>> # - can't be solved due to systematic limitations, or
>> # - are bugs in "historic" packages no one works on.
>> However, not with "catchall" regular expressions but on a casewise basis,
>> and with explaining comment. This could be implemented via sublabels.
Alternatively, "suspended" could be kept as an "orthogonal" label
(outside the hierarchy).
>> tests with low signal/noise ratio
>> ================================
>> The export with Unicode-aware TeX engine but TeX-fonts
>> (dvi3|pdf4|pdf5)_texF is rather an example for a class of "unreliable"
>> tests -- however, the unreliability is not "fail/pass depending on the
>> site or the phase of the moon".
>> The inverted test cases fail for every developer running the tests, but
>> > Many other tests with the same signature are not failing.
>> Rather, export success depends on the document content:
>> Adding or removing one character (which depends on some autoloaded
>> package), can make a compilable document uncompilable and vice
>> versa. Also, the probability of hidden problems (erroneous output despite
>> non-failing export) is high.
>> This means that "(dvi3|pdf4|pdf5)_texF"-tests have a low signal/noise
>> ratio.
...
>> The questions are
>> Do we want a label for all tests with low signal/noise ratio?
> I want.
>> How to name this label?
Suggestions: fragile, delicate, unstable
>> How should we handle tests carrying this label?
* Keep outside the hierarchy as additonal label (orthogonal).
A failing test can have, e.g., the labels
export:inverted:minor unstable
and a currently working test the labels
export unstable
* Add label to the log-lines, so that failing tests read something like
Test #2109: UNSTABLE.UNRELIABLE:ERRATIC_export/examples/seminar_pdf4_TeXF
or
Test #2xx: UNSTABLE_export/doc/Math_pdf5_SystemF
+1 New fails of "unstable" tests are marked, so that we know the reason
may rather be an incompatibility than a regression.
Günter
