On Fri, Jan 29, 2016 at 09:53:36AM +0000, Guenter Milde wrote: > On 2016-01-28, Scott Kostyshak wrote: > > On Thu, Jan 28, 2016 at 07:07:43AM +0000, Guenter Milde wrote: > > ... > > >> > 3365 - TEMPLATES_export/templates/ACM-siggraph_dvi (Failed) > >> ... > > >> It works with the new version. However, as these templates are for a > >> not-on-CTAN class, I vote to add them to > >> unreliable:nonstandard > >> in any case. > > > Fine with me. When you add the comment, to be consistent, should we say > > "not-on-TeXLive class" ? > > > The documentation currently states: > > > nonstandard: Documents with additional requirements, e.g. a class or > > package file not in TeXLive. > > > They are probably closely related, but apparently not equivalent: > > > http://tex.stackexchange.com/questions/185495/how-can-a-package-be-listed-on-ctan-but-not-be-available-in-tex-live > > TeXLive packages are a subset of CTAN packages, so "not on CTAN" implies > "not in TeXLive". > > > I don't have a strong preference on this so if you think something else > > makes more sense, go ahead. > > We could either use "not in TeXLive" consistently (as TeXLive is the > documented requirement for running the tests) or > indicate that the package is not even on CTAN. I don't have a preference > but would prefer a consistent approach.
I agree about consistency. I don't have a preference between the two. > >> The docs are docs, if using them directly as test samples makes problems, > >> we should use a copied version for the tests. > > > We should write down the advantages and disadvantages of this. One > > advantage is that we are more likely to catch LyX bugs. For example, > > when Uwe added English to the Japanese docs, they started to fail (I > > think you refer to this above). This is probably a LyX bug that we would > > not catch if we just fixed the documents that we test. Some > > disadvantages are that there are more noise and frustration. > > > My opinion is that we should not use fixed versions of the documentation > > to test. We should have unit tests that have fixed versions, and I hope > > that we simulataneously build up our unit tests. But for these export > > tests I think we will miss bugs if we only test the fixed versions. I > > think it is worth the extra noise and that in the long-run we will > > develop a policy that is agreed upon, not too invasive, and thus we will > > eventually improve the signal-to-noise ratio. > > The idea was: > > * working on the documentation must not be complicated by having to > consider non-standard exports. > > -> successful compiling/testing an edited documentation document with > pdflatex suffices to ensure it can be commited, not tests with other > exports are required. > > * if failures with other exports due to "half-tested" documentation updates > are a problem for the test maintainer, > the test suite should use copies that are > > * copied to a cache dir (autotests/samples/doc/, say) but not changed. > > * updated regularely (but on a time chosen by the test suite maintainer) > from the originals in lib/doc/ > > This way, > > * no test will fail due to ongoing work on documentation, > * the documentation is still tested in full (with some delay) > * failures with non-default export can be examined and handled accordingly > in one run with the cache update. > * "interesting failures" (like the nested-language+polyglossia problem in > es/Customization can be separated and moved into dedicated test samples. I see. This is a clear and good explanation. Scott
signature.asc
Description: PGP signature
