On 2015-11-28, Kornel Benko wrote:
> Originally, this file should have regexes to select export combinations
> which fail. Therefore the name 'reverted'.
Unfortunately, this name is confusing: we revert a commit but invert a test.
> Soon we (Scott and I) found, that some tests worked on Scott's side,
> but not on mine (or reverse).
>
> Now, this file includes many tests which are not at all reverted. The
> main purpose is now to select tests which do not pass everywhere even
> with full TL/MikTeX installed.
>
> Günter Milde proposed 'problematicTests'.
>
> ATM it is the main filter to select suspicious test combinations.
We do not just suspect anything in these tests, we list tests where we
*know* that these tests fail (at least at one site).
OTOH, the term "suspicious" could be used instead of "fragile" for what I
proposed as a tag for test combinations that are "unstable" or "working by
chance", e.g. unsave export-font combination like XeTeX-texF.
> This filters 319 tests (out of 3716 export tests).
> From them 270 are inverted (means they always fail) and 49 unreliable
> (means that they fail or pass depending on some installed data)
>
> Our filtering is
>
> ignoredTests: (small file)
> Input: any export combination
> Tests selected here are withdrown.
> Output: All but selected.
>
> revertedTests:
> Input: Each test which passed 'ignoredTests'
> Tests not selected here gets the label 'export'. They should pass.
> Output: Tests selected by its regexes.
>
> suspendedTests:
> Input: Each Test selected by 'revertedTests'
> Tests selected here gain the label 'suspended' but _not_ 'export' or
> 'reverted'
> Output: Tests _not_ selected
>
> unreliableTests:
> Input: Each Test _not_ selected by 'suspendedTests'
> Tests selected either pass or fail, but that is dependant on system
> where the test is run.
> Selected tests gain the label 'unreliable'
> Not selected tests gain the labels 'reverted' and 'export'
Why do you use a filter chain?
Wouldn't it be easier to maintain separate filters?
Currently, an unreliable test must get an entry in "revertedTests" AND
"unreliableTests". If the problem is solved, both entries should be removed
again (but after removing the one from revertedTests we will not be flagged
to remove the other entry as well).
We have 2 options:
a) put all in one file "invertedTests" and use the new subtag mechanism:
... # normal inverted tests
...
subtag: unreliable
... # unreliable tests
...
subtag: suspended
... # suspended tests
...
b) change the filtering mechanism:
> ignoredTests: (small file)
> Input: any export combination
> Tests selected here are withdrown.
Not selected tests gain the label 'export'
> Output: All but selected.
> revertedTests:
> Input: Each test which passed 'ignoredTests'
Selected tests gain the labels 'reverted'.
> suspendedTests:
Input: Each test which passed 'ignoredTests'
Tests selected here gain the label 'suspended'
# eventually remove the labels 'reverted' and 'export'
> unreliableTests:
Input: Each test which passed 'ignoredTests'
> Tests selected either pass or fail, but that is dependant on system
> where the test is run.
> Selected tests gain the label 'unreliable'
> Not selected tests gain the labels 'reverted' and 'export'
> I think that most files used to filter export tests has appropriate naming.
> Remains only 'revertedTests'.
> I'd like to rename it to something like 'suspicious', 'suspect' or
> 'suspected'-Tests.
I am not happy with the naming "suspicious" (we know for sure there is a
problem and the test fail).
I am not happy with the naming "suspended" (what is its use case?)¹
Günter
¹ I know I once proposed this category, but I wanted something different from
what became implemented.
