On 2016-01-28, Scott Kostyshak wrote: > On Thu, Jan 28, 2016 at 07:07:43AM +0000, Guenter Milde wrote:
... >> > 3365 - TEMPLATES_export/templates/ACM-siggraph_dvi (Failed) >> ... >> It works with the new version. However, as these templates are for a >> not-on-CTAN class, I vote to add them to >> unreliable:nonstandard >> in any case. > Fine with me. When you add the comment, to be consistent, should we say > "not-on-TeXLive class" ? > The documentation currently states: > nonstandard: Documents with additional requirements, e.g. a class or > package file not in TeXLive. > They are probably closely related, but apparently not equivalent: > http://tex.stackexchange.com/questions/185495/how-can-a-package-be-listed-on-ctan-but-not-be-available-in-tex-live TeXLive packages are a subset of CTAN packages, so "not on CTAN" implies "not in TeXLive". > I don't have a strong preference on this so if you think something else > makes more sense, go ahead. We could either use "not in TeXLive" consistently (as TeXLive is the documented requirement for running the tests) or indicate that the package is not even on CTAN. I don't have a preference but would prefer a consistent approach. >> The docs are docs, if using them directly as test samples makes problems, >> we should use a copied version for the tests. > We should write down the advantages and disadvantages of this. One > advantage is that we are more likely to catch LyX bugs. For example, > when Uwe added English to the Japanese docs, they started to fail (I > think you refer to this above). This is probably a LyX bug that we would > not catch if we just fixed the documents that we test. Some > disadvantages are that there are more noise and frustration. > My opinion is that we should not use fixed versions of the documentation > to test. We should have unit tests that have fixed versions, and I hope > that we simulataneously build up our unit tests. But for these export > tests I think we will miss bugs if we only test the fixed versions. I > think it is worth the extra noise and that in the long-run we will > develop a policy that is agreed upon, not too invasive, and thus we will > eventually improve the signal-to-noise ratio. The idea was: * working on the documentation must not be complicated by having to consider non-standard exports. -> successful compiling/testing an edited documentation document with pdflatex suffices to ensure it can be commited, not tests with other exports are required. * if failures with other exports due to "half-tested" documentation updates are a problem for the test maintainer, the test suite should use copies that are * copied to a cache dir (autotests/samples/doc/, say) but not changed. * updated regularely (but on a time chosen by the test suite maintainer) from the originals in lib/doc/ This way, * no test will fail due to ongoing work on documentation, * the documentation is still tested in full (with some delay) * failures with non-default export can be examined and handled accordingly in one run with the cache update. * "interesting failures" (like the nested-language+polyglossia problem in es/Customization can be separated and moved into dedicated test samples. Günter
