A Proposal for Rewriting the Stylesheets Tests

The current state of the tests

The Stylesheets Test folder contains the current testing mechanism, and I've been investigating it over the last few weeks. The set of tests clearly grew by accretion with not much in the way of an overall plan, and predictably it has become an over-complicated collection of inconsistencies, obsolescence and mystery. It basically works like this:

1. The tests are run using a Makefile (Test/Makefile). 2. The Makefile is divided into a range of targets intended to test different features, such as test-to-html, test-markdown, test-cocoa and so on. 3. Each target contains multiple tests which apply one of the conversions supported by the Stylesheets and then check the result in some way.

The conversions are applied in various different ways:

• By running the symlinked command (e.g. bin/teitohtml) which calls the bin/transformtei script
• By running ant and passing one of the ant build files such as /docx/build-to.xml
• By running Saxon and applying one of the XSLT stylesheets directly (such as fo/to.xsl)

There appears to be no rationale governing which method is used to invoke a process.

Problems with the current tests

As I've worked on figuring out how the tests work, I've raised a number of bugs and fixed a few (see for instance GitHub issues 213 through 218). The problems fall into these main categories:

• There are dozens of input test files, and very few of them have meaningful names or helpful comments to explain what they're testing (for instance, test14.xml, test2.xml, test25.xml, test26.xml, test3.xml, test4.xml, test6.xml, test31.xml, test36.xml, test23.xml, test17.xml, test20.xml, test27.xml, test22.xml, test28.xml, test8.xml, test32.xml, test24.xml, test12.xml, test5.xml).
• Many test files are already invalid or obsolete due to deprecation of various features in recent versions of P5.
• Different ways of calling the tests (see above) mean that we're not always testing the full process pathway, and it's not clear why.
• The approach of processing many small files rather than a small number of large ones means that the test process takes longer, because of the repeated necessity to spin up Java VMs.
• The tests can only be run on *NIX systems which have make installed.

Principles for rewriting the tests

I believe it's time to start again with a new approach to testing, and I think we should adopt these principles in doing so:

Use Ant rather than make

Ant is already used in the test process anyway. If we abandon make and write the whole thing in Ant, there will be a number of advantages:

• One less technology to install and understand.
• Greater efficiency and therefore speed when other Java-based tasks are invoked.
• Better documentation and commenting.

I've started rewriting a number of the existing tests in the Stylesheets/Test2 directory, using Ant instead of make, and I've encountered no major obstacles so far.

Cross-platform support

This is difficult but feasible; I have a pilot project I'm working on in the Stylesheets/Test2 directory where I've written an ant buildfile called convert.xml which calls a transformation in two different ways based on the platform it's running on. The