On Sat, Aug 3, 2013 at 4:50 AM, Kornel Benko <kor...@lyx.org> wrote: > Am Samstag, 3. August 2013 um 04:36:03, schrieb Scott Kostyshak > <skost...@lyx.org> > >> On Sat, Aug 3, 2013 at 4:12 AM, Kornel Benko <kor...@lyx.org> wrote: > >> > Am Samstag, 3. August 2013 um 03:54:46, schrieb Scott Kostyshak > >> > <skost...@lyx.org> > >> > > >> >> On Fri, Aug 2, 2013 at 5:55 AM, Kornel Benko <kor...@lyx.org> wrote: > >> > > >> >> > Am Freitag, 2. August 2013 um 11:10:33, schrieb Vincent van >> >> > Ravesteijn > >> > > >> >> > <v...@lyx.org> > >> > > >> >> >> What about testing everything, and marking these tests as "known to > >> >> >> fail". > >> > > >> >> > > >> > > >> >> >> In that way, if suddenly, LuaTex supports such a language, the tests > >> >> >> will > >> > > >> >> > > >> > > >> >> >> tell us. By hardcoding that these languages are not tested, this > >> > > >> >> >> hardcoding > >> > > >> >> > > >> > > >> >> >> will be there for ever, even if some of the languages might get > >> >> >> supported. > >> > > >> >> > > >> > > >> >> > > >> > > >> >> > > >> > > >> >> > The list could be read in CMakeLists.txt, then we can add a parameter >> >> > to > >> > > >> >> > export.cmake > >> > > >> >> > > >> > > >> >> > to revert the meaning of success. > >> > > >> >> > >> > > >> >> I like these ideas. I created the file > >> > > >> >> development/expectedTestFailures partly for this reason (to know when > >> > > >> >> a test fails because of a regression and when a test file fails > >> > > >> >> because of something we already know), but I didn't think about > >> > > >> >> programmatically inputting such a file. > >> > > >> >> > >> > > >> >> My only hesitation with the inversion is that it is not intuitive: if > >> > > >> >> a Hebrew LuaTeX test passes, that means that it is broken, and if it > >> > > >> >> suddenly fails, that means that the export actually worked. This is > >> > > >> >> fine for you and me because we would know about the inversion. But if > >> > > >> >> anyone else ends up running the tests, how will they know? We could > >> > > >> >> have the LastTest.log explain this but I don't know if they would know > >> > > >> >> to look there. This could also be explained in > >> > > >> >> development/autotests/README, but again I don't know if someone will > >> > > >> >> read that. > >> > > >> >> > >> > > >> >> To be clear, I am in favor of this. But I wanted to explain my >> >> hesitation. > >> > > >> >> > >> > > >> >> Scott > >> > > >> > > >> > > >> > We could mark the test as "shouldFail" or "shouldNotSucceed" (in the > >> > testname). > >> > > >> > > >> > > >> > like > >> > > >> > # ctest -R export/doc/attic/shouldNotSucceed_DocStyle_pdf > >> > >> Do you mean instead of inversion or in addition to inversion? I think > >> you mean instead of inversion. I like your idea of inversion better > >> because it would be nice to have the list of failing tests as clean as > >> possible. Also, this way we would not really be notified that a test > >> went from failing to passing. If a test goes from passing to failing > >> it will catch our attention (and we can fix the test and documentation > >> of the now-working feature, etc). > >> > >> How about we combine the two ideas (perhaps this is what you were > >> suggesting and I misunderstood)? We invert and change the name. > > > > I meant this one. Sorry for being unclear. > > > >> For > >> example: > >> ctest -R export/doc/attic/INVERTED-TEST_FAILURE-IS-GOOD_DocStyle_pdf > >> It's ugly but the name of the test is our only way to pass > >> information. If someone sees this, either they will understand it's > >> not bad or they will be confused and look for more information in > >> README or LastTest.log > > > > I like it. > > > > So now comes to the hard part. Name and content of the exceptions file. > > Should it be for exports only? In that case it may consist of filenames, > > otherwise it should contain testnames. Both are easy to implement.
Either would be fine in my opinion. I can't think of a specific use case now but using test names leaves open the possibility of generalizing this mechanism in the future. Scott
