A Proposal for Rewriting the Stylesheets Tests

The current state of the tests

The Stylesheets Test folder contains the current testing mechanism, and I've been investigating it over the last few weeks. The set of tests clearly grew by accretion with not much in the way of an overall plan, and predictably it has become an over-complicated collection of inconsistencies, obsolescence and mystery. It basically works like this:

1. The tests are run using a Makefile (Test/Makefile). 2. The Makefile is divided into a range of targets intended to test different features, such as test-to-html, test-markdown, test-cocoa and so on. 3. Each target contains multiple tests which apply one of the conversions supported by the Stylesheets and then check the result in some way.

The conversions are applied in various different ways:

• By running the symlinked command (e.g. bin/teitohtml) which calls the bin/transformtei script
• By running ant and passing one of the ant build files such as /docx/build-to.xml
• By running Saxon and applying one of the XSLT stylesheets directly (such as fo/to.xsl)

There appears to be no rationale governing which method is used to invoke a process.

Problems with the current tests

As I've worked on figuring out how the tests work, I've raised a number of bugs and fixed a few (see for instance GitHub issues 213 through 218). The problems fall into these main categories:

• There are dozens of input test files, and very few of them have meaningful names or helpful comments to explain what they're testing (for instance, test14.xml, test2.xml, test25.xml, test26.xml, test3.xml, test4.xml, test6.xml, test31.xml, test36.xml, test23.xml, test17.xml, test20.xml, test27.xml, test22.xml, test28.xml, test8.xml, test32.xml, test24.xml, test12.xml, test5.xml).
• Many test files are already invalid or obsolete due to deprecation of various features in recent versions of P5.
• Different ways of calling the tests (see above) mean that we're not always testing the full process pathway, and it's not clear why.
• Different test results are handled in very different ways for no apparent reason. For instance, some test results are formatted with xmllint before being compared with their matching expected results file, and some are not.
• The approach of processing many small files rather than a small number of large ones means that the test process takes longer, because of the repeated necessity to spin up Java VMs.
• The tests can only be run on *NIX systems which have make installed.

Principles for rewriting the tests

I believe it's time to start again with a new approach to testing, and I think we should adopt these principles in doing so:

Use Ant rather than make

Ant is already used in the test process anyway. If we abandon make and write the whole thing in Ant, there will be a number of advantages:

• One less technology to install and understand.
• Greater efficiency and therefore speed when other Java-based tasks are invoked.
• Better documentation and commenting.

I've started rewriting a number of the existing tests in the Stylesheets/Test2 directory, using Ant instead of make, and I've encountered no major obstacles so far.

Cross-platform support

This is difficult but feasible; I have a pilot project In the Stylesheets/Test2 directory I've written an ant buildfile called convert.xml which calls a transformation in two different ways based on the platform it's running on. We know that most people working on the Stylesheets currently are using *NIX platforms, but when you need to be on *NIX before you can even call the command line transformations at all, this is hardly a surprise. Many people use Windows, and we ought to do our best not only to support them but to enable them to contribute to the codebase.

Smaller numbers of test files

Rather than ten small files testing various different things, a better approach would be a single test file which attempts to include a wide range of different features, each documented in-place; adding a new test would then be simpler since it would require only adding a new section to a file rather than constructing a whole new file, which might have features irrelevant to the test which later go stale or obsolete.

Include test files drawn from P5 Exemplars

Despite the fact that they're maintained in a separate repository, the Stylesheets are tightly coupled with P5, and development on the two codebases must proceed largely in lock-step fashion. The Stylesheets tests which relate to ODD processing (creating schemas and documentation in various formats) should be tested using ODD files from the P5 Exemplars folder. This will help to ensure that:

1. We're testing all the features people are likely to use most commonly. 2. All P5 changes (assuming we use tei_all as a test file) will be tested as soon as they're introduced, and it should not be so easy for us to neglect to implement Stylesheets support for a P5 change.

Validate all test input files before using them

We need to know when a test file goes stale, especially a TEI file.

Name test files meaningfully

"cocoatest.txt" is more helpful than "test40.txt", obviously. Where larger files are created to test many things, this is harder, of course.

Output results into a separate folder

Right now, all the results go into the same folder as the test files, which means the directory is even more cluttered with "test25.xyz" files, making it harder to see what's happening. Results should go into a "results" folder, alongside the "expected-results" folder.

Keep linked resources in separate folders

The Test folder currently contains linked images, various schemas used for validation, and other bits and pieces such as PERL scripts used for cleanup. These should be organized properly.

Process all X*ML output in the same way after creation

As mentioned above, some XML files are formatted with xmllint before checking, and some are not; some have comments removed and some don't. A single XSLT transformation (I've started one called cleanForDiff.xsl in Test2) should be applied to all X*ML output files before they're compared with expected results.

Separate out the Oxygen tests

The current test process includes some tests which depend on an installation of Oxygen, and run transformations using Oxygen's Ant, as well as saxon8ee.jar. These tests properly belong in the oxygen-tei repo, where we should be doing a much better job of testing anyway (see https://github.com/TEIC/TEI/issues/464, which should be converted to an issue in the oxygen-tei repo at some point).