Am Freitag, 4. Dezember 2015 um 11:54:37, schrieb Guenter Milde <mi...@users.sf.net> > On 2015-12-04, Kornel Benko wrote: > > Am 3. Dezember 2015 um 22:08:35, schrieb Guenter Milde <mi...@users.sf.net> > >> On 2015-12-02, Kornel Benko wrote: > >> > Am 2. Dezember 2015 um 22:07:36, schrieb Guenter Milde > >> > <mi...@users.sf.net> > > Dear Kornel, > > > >> Generally, labels are sensible to select subsets that cannot easily be > >> selected via regular expressions. > > >> Are labels for the (sub-)locations really required? > > >> Why not use, e.g. `-R "export/examples/"` > >> instead of `-L export:examples` in case there is need to select just > >> examples (and not manuals or templates). > > > Because -R does not categorize. > > > If you want to test examples, simply use '-L examples' Since sublabels > > are are (and hopefully stay) unique, there is no problem. Using '-R > > examples' dos all test having 'examples' in testname. This is not the > > same. E.g. (hypothetica) test 'check_example_urls'. > > My example was `-R export/examples/` > > This excludes 'check_example_urls' and similar "bycatch" without requiring a > special label. > > I agree that labels make sense for categories that are difficult to get with > regular expressions (or not at all). > > However, labels for categories that are easy discernible with a regexp just > add noise and make it harder to understand the test machinery. > > > >> If you prefer to keep them, I'd prefer separate labels for place and > >> status and also clearly mark that "suspended" is a subset of "inverted" > >> (this is especially important for the output: clearly indicate whether a > >> fail is an export fail or an fail to fail). > > > It is.
... > > Yes, 'suspended' means also inverted. But we do NOT WANT to test > > 'suspended' if we test only '-L inverted' Therefore it is NOT labeled > > as 'inverted' > > I understand your reasoning. However, we have to make a compromise: > > explicit: inverted:suspended The (implementation) problem hare is, that I use a subset of labels to create a prefix for a testname. So I would have to assign 'inverted' label first, ant then remove again. Or a special handling in a routine which does creates the prefix. > +1 clear test output: > > 906 - INVERTED:SUSPENDED_export/doc/... (Failed) Yes, maybe. But if you _know_, that 'suspended' implies 'inverted', then there is no more info. > -1 longer test invocation command when only testing non-suspended cases: > > ctest -L inverted -LE suspended Yes. > implicit: suspended > > +1 simpler test invocation command when only testing non-suspended cases: > > ctest -L inverted > > -1 the simpler invocation command obscures the fact that some tests are > suspended, so they are easier forgotten. I would use -0.0001 here. > -2 hard to understand output: > > 906 - SUSPENDED_export/doc/... (Failed) Hm, yes. But only '-1' > Without reading the docs (or without our long discussion here), > I would not realize that this is a fail to fail! ... > > We have many tests failing because of combination pdf4_texF. We do not > > want to bother with them now. > > So in suspendedTests we choose '.*pdf4_texF'. > > > We normally do not choose a single test here. > > ATM there are exacly 2 (two) regexes there. > > I cannot see the advantage in this approach: > > +0 In case of a regression with .*pdf4_texF, we still have to investigate > and eventually add a regular expression in "suspiciousTests" to invert > and suspend the test case. If a test always fails, then it belongs to suspiciousTests. Not more. > It would be no more effort to sort it in a section with sublabel > "suspended" in "suspiciousTests". True. > -1 With the 2-files approach, the logic is more complicated: > a test case is suspended, if it matches a regular expression in > "suspiciousTests" AND a regular expression in "suspendedTests". > > Using a sublabel in "suspiciousTests" instead of a separate file would > result in: > > +1 all suspended test patterns in one place in "suspiciousTests". > > -1 the one-time effort to sort the affected regular expressions in > "suspiciousTests" into the correct section. > > +1 simpler logic For the user, not for the implementer. -1 Longer (time consuming) scan phase. > +1 shorter and easier to understand documentation > > > > ... > > >> >> >> Do we want a label for all tests with low signal/noise ratio? > > >> >> > I want. In the meantime I do not so sure that I want. If latex exits '0', then we have no automatic test-result signalling the output is really correct or wrong. Examining the output has to be done manually each time. That is different to all other test results. We could move such a test to 'unreliable' with some extra label. It cannot be done on whole groups like .*texF. > Did you realize, that the content of a "delicateTests" file would be the > same as what we have now in "suspendedTests"? No, I am not realizing it. > The only difference is that "suspendedTests" is only applied to inverted > tests. > I see it differently. > Günter Kornel
signature.asc
Description: This is a digitally signed message part.