"Chromatic" <[EMAIL PROTECTED]> wrote: > Advantages of inline tests: > - close to the documentation > - one place to update > - harder for people to update docs without finding code
Plus, it gives us a mechanism to validate example-code within documents > Disadvantages: > - doc tools must skip tests normally > - pure-doc patches are harder > - some tests (bugs, regressions) don't naturally fit in docs > - makes docs much larger > - adds an extra step to extract tests before running them > - adds weight to installed documents (or they must be extracted) These seem to be reasonable arguments: though issues of file-size and the need for extraction seem a bit weak. Oh, and bugs do belong in documents: "erata". The majority of tests do not belong in documents, for the simple reason that they are implementation-centric, not user-centric. But then, in perl, every file can contain POD, so even external-test files can be documents (documents that are not part ofthe perl6 documentation project). > Advantages of external tests: > - tests can be grouped by purpose/thing tested within tests > - test files can be smaller > - individual tests can be run during development > - tests can be grouped by subsystem/purpose All of these seem to be red-herrings: obviously the individual files are smaller if you split them: but the boiler plate probably makes the total size greater then for the integrated form. Similarly, arguments based on structure are probably neutral. All formats will have structure; and all will/should allow us to run individual tests. The important arguments should be based on human psychology: which is easier to comprehend and maintain? > Disadvantages of external tests: > - proper organization is important > - multiplies files > - doc patchers may find test to patch and vice versa These are red-herrings too. Proper organisation is always important; you'll always have multiple files (and if you include tests as part of the document files, you'll probably end up with more doc files). It may be true that doc patchers need to find corresponding tests ... but its quite easy to miss a test even if its in the same file. > On the whole, I prefer external tests. Brent's schema looks good. > > In Perl 5 land, the culture expects documentation and test patches with the > addition of a new feature. I think we can cultivate that for Perl 6. As > Brent also alluded, there will probably be standalone regression tests anyway. Maybe there's a terminology problem: but what is a regression test? In my world, we create a regression by running existing tests: we don't write a special test suite for the regression. There may be a small number of tests that we exclude from the regression, but then the special case is the non-regressed tests. I'm happy pick a format and run with it. When we've a few micro-sections done, then we can review. I see (in another post) that Mike has opted for external, "without objection". I'm abstaining. But I would like to see executable examples as part of the docs. Dave.
