On 2015-12-04, Kornel Benko wrote:
> Am 3. Dezember 2015 um 22:08:35, schrieb Guenter Milde <mi...@users.sf.net>
>> On 2015-12-02, Kornel Benko wrote:
>> > Am 2. Dezember 2015 um 22:07:36, schrieb Guenter Milde <mi...@users.sf.net>
Dear Kornel,
>> Generally, labels are sensible to select subsets that cannot easily be
>> selected via regular expressions.
>> Are labels for the (sub-)locations really required?
>> Why not use, e.g. `-R "export/examples/"`
>> instead of `-L export:examples` in case there is need to select just
>> examples (and not manuals or templates).
> Because -R does not categorize.
> If you want to test examples, simply use '-L examples' Since sublabels
> are are (and hopefully stay) unique, there is no problem. Using '-R
> examples' dos all test having 'examples' in testname. This is not the
> same. E.g. (hypothetica) test 'check_example_urls'.
My example was `-R export/examples/`
This excludes 'check_example_urls' and similar "bycatch" without requiring a
special label.
I agree that labels make sense for categories that are difficult to get with
regular expressions (or not at all).
However, labels for categories that are easy discernible with a regexp just
add noise and make it harder to understand the test machinery.
>> If you prefer to keep them, I'd prefer separate labels for place and
>> status and also clearly mark that "suspended" is a subset of "inverted"
>> (this is especially important for the output: clearly indicate whether a
>> fail is an export fail or an fail to fail).
> It is.
>> cmplyx
>> export
>> export:autotests
>> export:examples
>> export:manuals
>> export:mathmacros
>> export:templates
>> inverted
>> inverted:suspended
>> suspended:suspended:chemgreek
> What is this?
> Sublabels should not be doubled.
Sorry, a typo. Should have been
inverted:suspended:chemgreek
>> layout
>> load
>> lyx2lyx
>> lyxlyx
>> module
>> roundtrip
>> unreliable:erratic
>> unreliable:nonstandard
>> If the labels "inverted" and "unreliable" are only intended for export
>> tests, the could be used with "export:" suffix:
> No, also other tests may be inverted. This is a feature from ctest, not ours.
> To our convenience the tests are also labeled.
OK.
> Yes, 'suspended' means also inverted. But we do NOT WANT to test
> 'suspended' if we test only '-L inverted' Therefore it is NOT labeled
> as 'inverted'
I understand your reasoning. However, we have to make a compromise:
explicit: inverted:suspended
+1 clear test output:
906 - INVERTED:SUSPENDED_export/doc/... (Failed)
-1 longer test invocation command when only testing non-suspended cases:
ctest -L inverted -LE suspended
implicit: suspended
+1 simpler test invocation command when only testing non-suspended cases:
ctest -L inverted
-1 the simpler invocation command obscures the fact that some tests are
suspended, so they are easier forgotten.
-2 hard to understand output:
906 - SUSPENDED_export/doc/... (Failed)
Without reading the docs (or without our long discussion here),
I would not realize that this is a fail to fail!
...
>> > Everyone is free to add sublabels.
>> Yes. But my idea is to use "minor" and/or "wontfix" *instead* of the
>> "suspended" pattern file.
> Not my idea.
I know, it is my idea.
> For me 'suspended' means don't care, rare use case, ignore
> for a while.
Here, we agree. However, we don't agree on the implementation:
...
> ... 'suspended' is meant as a more general filter
...
> We have many tests failing because of combination pdf4_texF. We do not
> want to bother with them now.
> So in suspendedTests we choose '.*pdf4_texF'.
> We normally do not choose a single test here.
> ATM there are exacly 2 (two) regexes there.
I cannot see the advantage in this approach:
+0 In case of a regression with .*pdf4_texF, we still have to investigate
and eventually add a regular expression in "suspiciousTests" to invert
and suspend the test case.
It would be no more effort to sort it in a section with sublabel
"suspended" in "suspiciousTests".
-1 With the 2-files approach, the logic is more complicated:
a test case is suspended, if it matches a regular expression in
"suspiciousTests" AND a regular expression in "suspendedTests".
Using a sublabel in "suspiciousTests" instead of a separate file would
result in:
+1 all suspended test patterns in one place in "suspiciousTests".
-1 the one-time effort to sort the affected regular expressions in
"suspiciousTests" into the correct section.
+1 simpler logic
+1 shorter and easier to understand documentation
> ...
>> >> >> Do we want a label for all tests with low signal/noise ratio?
>> >> > I want.
Did you realize, that the content of a "delicateTests" file would be the
same as what we have now in "suspendedTests"?
The only difference is that "suspendedTests" is only applied to inverted
tests.
Günter
