This page follows the model of Mapping ODD Processing in trying to provide a clear general introduction to the way that different language inputs are processed in the building of Guidelines/documentation.

=TEI Guidelines=

This section deals with the Guidelines build process specifically.

==Input parameters==

The Guidelines Makefile provides the following set of parameters related to language processing when building the Guidlines:

- ALLLANGUAGES=en
- LANGUAGE=en
- INPUTLANGUAGE=en
- DOCUMENTATIONLANGUAGE=en

ALLLANGUAGES is a parameter which can take multiple values, space-separated; for each of those values, a separate version of the Guidelines will be built in its own folder by using the ant macro <code>buildweb</code>. This calls the Stylesheets <code>utilties/Guidelines.xsl</code>, passing the current value of the language in all three of the following parameters:

- lang
- doclang
- documentationLanguage

(These will be documented later.) This parameter is automatically set to the following list of languages for a number of Makefile targets, including <code>teiwebsiteguidelines</code>.

LANGUAGE is an apparently unused parameter; it appears in both the P5/Makefile and the P5/Test/Makefile, but is never used in either. _We recommend deleting this confusing parameter._

INPUTLANGUAGE governs which source tree for the Guidelines specifically is used; it governs the name of the source file,[1] setting the variable DRIVER to this file. That makes the root language effectively the input language, and then when other language versions are built in the <code>html-web.stamp</code> target, they are based on the input language rather than the default (en), assuming they are different.

[1]:P5/Source/en/Guidelines-en.xml or P5/Source/fr/Guidelines-fr.xml are the only two available right now, and the French chapter points only to English source files anyway, because the French header chapter is out of date and will not build

Mapping language processing

2020-11-06T17:34:23Z

Mapping ODD processing

2020-02-14T16:55:06Z

Mholmes:

[[Category:Customization]]

This page is intended to help us map out and document the various processing steps that are applied to [[ODD]] files, both in the current Stylesheets and in a future alternative ODD processor. The main goal is to document the steps in generating TEI schemas from a TEI customization ODD file. However, processing P5 itself is part of that process. And although not our primary goal, we hope to also provide documentation for generating custom documentation from a TEI customization ODD file.

An ''italicized'' name in column A (NAME) indicates a step that is performed for ''building P5'', but not necessarily for generating a customization.

The '''bold''' upper-case term in the NAME column is the term we have agreed to use for the process in question in future documentation. There has been a great deal of confusion over nomenclature which impedes clear discussion of these steps, much of it resulting from the varied naming conventions in use in the Stylesheets code, so establishing a standard set of terms is important.

The '''bold''' stylesheet name in the XSLT FILES column is the one that is called; the others are the ones that the called one imports or includes.

{| class="wikitable"
|-
! # || NAME || INPUT || OUTPUT FORM || PROCESSES INVOLVED || PREREQS || XSLT FILES || COMMENTS
|-
| 1 || ''Generate P5'' || guidelines-[lang].xml || p5.xml || Combine all chapter and spec files into a single document; processing instructions such as the generation of a table for chapter ST are handled. || There must be a repodate.xml file containing an XML representation of the git repo state. || '''TEI/P5/Utilities/expand.xsl''' || *Spec elements occur in specGrp and in div.
|-
| 2 || ''Generate P5 Subset'' || p5.xml (generated above) || p5subset.xml; p5subsetDoctored.xml || Create a special cut-down version of P5 which has only divisions and their headers along with all the specs. || Complete guidelines || '''TEI/P5/Utilities/subset.xsl''' || Why are all the div/head elements included? Why is a schemaSpec not created? Is it the case that schemaSpec always and only represents a customization of an existing source, which must eventually chain back to p5subset? If so, ODD is not a generic language; it depends on P5. This also generates the beautiful p5subsetDoctored.xml, which is a hack for DTD production.
|-
| 3 || ''Generate JSON from p5subset'' || p5subset.xml || p5subset.json ||   ||   || '''Stylesheets/odds/odd2json.xsl''', odds/teiodds.xsl, common/common_tagdocs.xsl, common/common_param.xsl, common/functions.xsl, common/i18n.xsl ||  
|-
| 4 || '''COMPILE ODD'''. Merge ODD customization with P5 source (SR calls this oddexpand; aka tangle aka compile aka process aka odd2odd aka flatten) || ODD customization; p5subset || file.processedodd ('''COMPILED ODD''') || The macrodef oddexpan is called, and that calls odd2odd.xsl with params || p5subset.xml; ODD customization || '''Stylesheets/odds/odd2odd.xsl''' ||
|-
| 5 || '''[[ODD CHAINING]]''' || An ODD customization (#2) whose @source points to another customization (#1) || file.processedodd || #1 undergoes merging with p5subset.xml, then #2 undergoes merging with the result of that. || Both ODDs and p5subset.xml || [See above] || Wrinkle: either #1 or #2 might have components that point out to another source, which would then be imported. See Lou's tutorial [https://teic.github.io/TCW/howtoChain.html]
|-
| 6 || '''COMPILED ODD TO RELAXNG''' Generate RELAX NG from processed ODD || file.processedodd (output of rows above) || file.rng || Straight conversion, albeit complicated || || '''Stylesheets/odds/odd2relax.xsl'''; odds/teiodds.xsl; odds/classatts.xsl; common/functions.xsl; common/i18n.xsl; common/common_param.xsl || The output is used to generate rnc and xsd using trang.
|-
| 7 || '''COMPILED ODD TO DTD''' Generate DTD from ODD || file.processedodd || file.[odd.]dtd || Generate a DTD from the processed ODD file. ||   || '''Stylesheets/odds/odd2dtd.xsl''', odds/teiodds.xsl, odds/classatts.xsl, common/i18n.xsl, common/functions.xsl, common/common_param.xsl ||  
|-
| 8 || '''COMPILED ODD TO XSD''' || file.processedodd || file.xsd || Generate RelaxNG from compiled ODD; convert with Trang to XSD; post-process XSD. || file.processedodd; Trang. || Ant task: '''xsd/build-to.xml'''; xsd/postprocess.xsl || G
|-
| 9 || '''COMPILED ODD TO SCHEMATRON''' Extract Schematron || file.processedodd || file.isosch || Generate a Schematron file with all rules correctly contextualized. || || '''Stylesheets/odds/extract-isosch.xsl''' || Obsolete stuff with "sch" exists alongside iso-sch; former should be removed and latter renamed to "sch".
|-
| 10 || '''COMPILED ODD TO TEI LITE''' Generate TEI Lite from ODD || file.processedodd ($tmpodd in teianttasks.xml; output of odds/odd2odd.xsl) || TEI Lite XML document || odd2lite.xsl || || '''Stylesheets/odds/odd2lite.xsl'''; odds/teiodds.xsl; classatts.xsl; common/verbatim.xsl; common/common.xsl; common.tagdocs.xsl; common/common_param.xsl; common/common_core.xsl; common/common_textstructure.xsl; common/common_header.xsl; common/common_linking.xsl; common/common_msdescription.xsl; common/common_figures.xsl; common/common_textcrit.xsl; common/common_gaiji.xsl; common/i18n.xsl; common/functions.xsl|| This is the prerequisite step for generating XHTML5, ePub and PDF documentation.
|}

SIG:Newspapers&Periodicals

2019-09-17T13:29:31Z

Mholmes:

SIG convener: Dario Kampkaspar

Co-convener: Mary Isbell

Publications with any kind af regular publication cycle, from several times a day to cycles of several years, arguably form the majority of (printed) texts nowadays. They span from newspapers via almanacs to comics and many others.

While these different types all have their own needs, their periodicity means that there are special questions to keep in mind when systematically encoding them.

The SIG Periodicals and Newspapers will focus on aspects that are common to all these kinds of periodicals:

* how to encode this periodicity with its aspects in a concise and machine readable way;
* how to deal with parts of periodicals that have their own periodicity (e.g. weekly, monthly etc. additions);
* how to deal with texts continued over several issues of a perdiodical;
* how to encode the title of e.g. a newspaper (which usually do not cover an entire page).

Additionally, the SIG will focus on methods and workflows for preparing full texts of periodicals by means of OCR and related technologies.

[[Category:SIG]]

SIG:Newspapers&Periodicals

2019-09-17T13:29:09Z

Mholmes:

SIG convener: Dario Kampkaspar
Co-convener: Mary Isbell

Pulbications with any kind af regular publication cycle, from several times a day to cycles of several years, arguably form the majority of (printed) texts nowadays. They span from newspapers via almanacs to comics and many others.

While these different types all have their own needs, their periodicity means that there are special questions to keep in mind when systematically encoding them.

The SIG Periodicals and Newspapers will focus on aspects that are common to all these kinds of periodicals:

* how to encode this periodicity with its aspects in a concise and machine readable way;
* how to deal with parts of periodicals that have their own periodicity (e.g. weekly, monthly etc. additions);
* how to deal with texts continued over several issues of a perdiodical;
* how to encode the title of e.g. a newspaper (which usually do not cover an entire page).

Additionally, the SIG will focus on methods and workflows for preparing full texts of periodicals by means of OCR and related technologies.

[[Category:SIG]]

Tei-xsl

2019-09-16T20:44:22Z

Mholmes: Updating to fix obsolete info

[[Category:Tools]]

[[Category:Conversion and preprocessing tools]]
[[Category:Publishing and delivery tools]]
[[Category:XSLT]]

== Synopsis ==
This is a family of XSLT 2.0 stylesheets to transform TEI XML documents to various formats, including XHTML, LaTeX, XSL Formatting Objects, ePub, plain text, RDF, JSON; and to/from Word OOXML (docx) and OpenOfice (odt). They concentrate on the core TEI modules which are used for simple transcription and "born digital" writing. It is important to understand that they do '''not'''
* cover ''all'' TEI elements and possible attribute values
* attempt to define a standard TEI processing or rendering model
They should not be treated as the definitive view of the TEI Consortium about rendering TEI documents.

== Features ==

== User commentary ==
'''Please sign all comments.'''

== System requirements ==
"The XSL FO style sheets were developed for use with PassiveTeX (http://projects.oucs.ox.ac.uk/passivetex/), a system using XSL formatting objects to render XML to PDF via LaTeX. They have not been extensively tested with the other XSL FO implementations."<ref>http://www.tei-c.org/release/doc/tei-xsl/</ref>

Software dependencies:

* Saxon 9.2 or later, which you can get from https://www.saxonica.com/download/java.xml .
* ant and ant-contrib.

== Source code and licensing ==
This software is dual-licensed:

1. Distributed under a Creative Commons Attribution-ShareAlike 3.0
Unported License http://creativecommons.org/licenses/by-sa/3.0/

2. http://www.opensource.org/licenses/BSD-2-Clause

== Support for TEI ==

The stylesheets render TEI XML documents to various formats, with the constraint that they do not offer an implementation of all tags,
and only support a limited range of values for the @rend attribute.

== Language(s) ==
Documentation is in English.

== Documentation ==
http://www.tei-c.org/release/doc/tei-xsl/

Among the components of this package of stylesheets not documented are the following:

* <code>tools/oddbyexample.xsl</code> – generates a list of all the attribute values used in the directory of texts you apply it to, in the form of an [[ODD]] file, from which you can in turn generate documentation and schemas using Roma
* <code>tools/odd2nuodd.xsl</code> – switches from a traditional ODD-by-exclusion to a newer form ODD-by-inclusion

=== Creating a custom profile ===

To create a profile for converting to/from a format and TEI XML, create either or both as needed:

<code>profiles/$PROFILENAME/$FORMAT/to.xsl</code>

<code>profiles/$PROFILENAME/$FORMAT/from.xsl</code>

Start by copying the files in <code>profiles/default/$FORMAT/</code> and then adding your own overrides and/or additional templates. See [http://listserv.brown.edu/archives/cgi-bin/wa?A2=ind1303&L=TEI-L&F=&S=&P=99998 this brief example of additional templates].

Note that if your <code>$FORMAT</code> is docx, this directory must contain a file <code>template.docx</code> which is used to create .docx files from. See the sample in the default profile.

=== Converting from DOCX format ===

"The TEI conversions from docx are better in many ways than the conversions from other the wordprocessing formats. There are also small tricks like having docx styles of 'tei_elementName' to get certain phrase-level elements converted."<ref>https://listserv.brown.edu/archives/cgi-bin/wa?A2=TEI-L;1123776a.1605</ref>

== Tech support ==
Bugs and feature requests should be made in GitHub: https://github.com/TEIC/Stylesheets/issues

== User community ==

== Sample implementations ==

The package includes a number of profiles in the <code>profiles/</code> directory.

== Current version number and date of release ==

7.48.0 (2019-07-16)

== History of versions ==

See detailed changelog at https://github.com/TEIC/Stylesheets and automatically generated commit log at https://github.com/TEIC/Stylesheets/commits/master . Past versions can be downloaded at https://github.com/TEIC/Stylesheets/releases .

== How to download or buy ==

If you use [[Oxygen]], the easiest thing to do might be to use the open-source TEI framework built into <oXygen/>. Otherwise, there are '''three options''':

=== a) Download zip file ===
Download the latest zip file from https://github.com/TEIC/Stylesheets/releases .

=== b) Install the Debian package ===

Install the <code>tei-xsl</code> Debian packages (see https://wiki.tei-c.org/index.php?title=TEIDebian).

=== c) Get from Github ===

<code>
git clone git@github.com:TEIC/Stylesheets.git /path/to/local/directory/Stylesheets
</code>

Then, if you want to add these to <code>/usr/</code> (which you will need to do in order to use the command-line shell scripts), you can do the following:

<code>
cd /path/to/local/directory/Stylesheets/
</code>

<code>
make install
</code>

<strong>but note the following</strong>:

* You will need to run <code>make install</code> as a user with write permission to <code>/usr/</code> . So if using Ubuntu, for example, you will need to instead do <code>sudo make install</code> .

== Additional notes ==
The XSLT 1.0 version of this stylesheet family, which is no longer actively maintained, can be found on Github at https://github.com/sebastianrahtz/TEIXSL-v1.git

== Notes ==

<references/>

P6-dev

2017-11-03T18:03:16Z

Mholmes:

This page is a place to record ideas which people feel are not appropriate for addition to the TEI P5 Guidelines (perhaps because they break backwards compatibility so extremely) but which they think should be recorded for reconsideration during development of the next major revision of P6. Things here should probably have had a FR submitted on GitHub first and been turned down for P5. One when we should be thinking about P6 see the final paragraph of the Birnbaum Doctrine: http://www.tei-c.org/Activities/Council/Working/tcw09.xml

With the new (as of Nov. 2015) GitHub issue tracker, TEI Council collects P6 issues with the label [https://github.com/TEIC/TEI/issues?q=is%3Aopen+is%3Aissue+label%3A%22Status%3A+Reconsider+for+P6%22 Reconsider for P6]

* msDesc and other ms* elements should be renamed, since they now refer to text-bearing objects rather than only manuscripts. Perhaps ''tboDesc'' etc.?

* Module fragmentation, e.g. http://purl.org/tei/fr/3490054 asked for splitting off all dateTime related elements into a small module of its own. Ann Arbor 2012 face to face meeting decided this kind of fragmentation should be postponed until P6.
** Would it be an idea to group all elements that describe ''entities'', if a new element ''object'' would be established? By doing this, the dateTime related elements were splitted "naturally".

* Consider placing a restriction on the simultaneous use of @url and @facs on <graphic> (see Council list discussion 2012-06-20).

NOTE: Some of the work on lists has already been done in release 2.7.0.
* Completely redo handling of lists by:
** dropping @type
** introducing @ordered in case the creator of a TEI document wants to record the semantic order of items in the list (but does anyone want to do this? on other elements as well?)
** using only @rend or @style (pending http://purl.org/TEI/FR/3519866 ) to record bulleted, arabic numbers, roman numerals, etc.
** redefining handling of "gloss lists"

* Consider a proper object-oriented inheritance structure, whereby (for instance) there could be a generic "entity" object, with a core structure, and "person", "place" and "org" objects could inherit this structure from it. Inheritance would allow overriding of various components of the structure, and specialization of them; for instance, the generic entity might have a start date and an end date, and for the descendant person would refine these into birth and death. We'd have to consider whether multiple inheritance would be practical (for instance, how would you merge the content models of two ancestors)?

* [[Ad-hoc_committee_on_encoding_of_bibliographic_citations#Proposed_future_revisions_to_the_Guidelines_.28which_will_break_backwards_compatibility.29|Things related to encoding of bibliographic citations]]

* An object inheritance model

* A firm distinction between elements used in the header and those used in the text. For example:
** There would be one element, with specific attributes and a specific content model, for the title in the metadata and another for any title transcribed in the document)
** att.global might be separated into a class for attributes on header elements and those on text elements.

* A clear distinction between elements used when transcribing a source document (like bibl) and elements used in creating born-digital documents (like biblStruct and biblFull). Need to figure out how msDesc fits into this.

* Expansion of [[ODD#Future_plans:_.22Pure_ODD.22|Pure ODD]] to enable us to define the content model of elements based on context; this would be convertible into models based on complexTypes in XSD, and into Schematron in RelaxNG. This would require that we drop support for DTDs (which most agree would be a good idea anyway).

* Go through a list of recent standards and standards updates and recommend how to use TEI in combination with them. Sample recommendations might be (I'm not asserting the truth of any of these):
** ''If encoders use NISO Z39.84-2005, the DOI may be expected to appear in the ''idno'' element.''
** ''None of the bibliographic elements exactly match any of the bibliographic elements in ANSI/NISO Z39.96-2012''
** ''If reporting on use digital objects using NISO SUSHI, each object with a TEI header should be reported on''
** ''If linking to resources that are available over both http:// and https:// protocols, it is recommended to use the 'procotol-indepedent' uris starting with // ''
** ...

[[Category:Suggestions]]

A Proposal for Rewriting the Stylesheets Tests

2017-01-09T00:06:32Z

Mholmes: /* Keep linked resources in separate folders */

==The current state of the tests==
The Stylesheets Test folder contains the current testing mechanism, and I've been investigating it over the last few weeks. The set of tests clearly grew by accretion with not much in the way of an overall plan, and predictably it has become an over-complicated collection of inconsistencies, obsolescence and mystery. It basically works like this:

# The tests are run using a Makefile (Test/Makefile).
# The Makefile is divided into a range of targets intended to test different features, such as test-to-html, test-markdown, test-cocoa and so on.
# Each target contains multiple tests which apply one of the conversions supported by the Stylesheets and then check the result in some way.

The conversions are applied in various different ways:

* By running the symlinked command (e.g. bin/teitohtml) which calls the bin/transformtei script
* By running ant and passing one of the ant build files such as /docx/build-to.xml
* By running Saxon and applying one of the XSLT stylesheets directly (such as fo/to.xsl)

There appears to be no rationale governing which method is used to invoke a process.

==Problems with the current tests==
As I've worked on figuring out how the tests work, I've raised a number of bugs and fixed a few (see for instance GitHub issues 213 through 218). The problems fall into these main categories:

* There are dozens of input test files, and very few of them have meaningful names or helpful comments to explain what they're testing (for instance, test14.xml, test2.xml, test25.xml, test26.xml, test3.xml, test4.xml, test6.xml, test31.xml, test36.xml, test23.xml, test17.xml, test20.xml, test27.xml, test22.xml, test28.xml, test8.xml, test32.xml, test24.xml, test12.xml, test5.xml).
* Many test files are already invalid or obsolete due to deprecation of various features in recent versions of P5.
* Different ways of calling the tests (see above) mean that we're not always testing the full process pathway, and it's not clear why.
* Different test results are handled in very different ways for no apparent reason. For instance, some test results are formatted with xmllint before being compared with their matching expected results file, and some are not.
* The approach of processing many small files rather than a small number of large ones means that the test process takes longer, because of the repeated necessity to spin up Java VMs.
* The tests can only be run on *NIX systems which have make installed.

==Principles for rewriting the tests==
I believe it's time to start again with a new approach to testing, and I think we should adopt these principles in doing so:

===Use Ant rather than make===
Ant is already used in the test process anyway. If we abandon make and write the whole thing in Ant, there will be a number of advantages:

* One less technology to install and understand.
* Greater efficiency and therefore speed when other Java-based tasks are invoked.
* Better documentation and commenting.

I've started rewriting a number of the existing tests in the Stylesheets/Test2 directory, using Ant instead of make, and I've encountered no major obstacles so far.

===Try to support Windows===
This is difficult but feasible. In the Stylesheets/Test2 directory I've written an ant buildfile called convert.xml which calls a transformation in two different ways based on the platform it's running on. We know that most people working on the Stylesheets currently are using *NIX platforms, but when you need to be on *NIX before you can even call the command line transformations at all, this is hardly a surprise. Many people use Windows, and we ought to do our best not only to support them but to enable them to contribute to the codebase.

===Use smaller numbers of test files===
Rather than ten small files testing various different things, a better approach would be a single test file which attempts to include a wide range of different features, each documented in-place; adding a new test would then be simpler since it would require only adding a new section to a file rather than constructing a whole new file, which might have features irrelevant to the test which later go stale or obsolete.

===Include test files drawn from P5 Exemplars===
Despite the fact that they're maintained in a separate repository, the Stylesheets are tightly coupled with P5, and development on the two codebases must proceed largely in lock-step fashion. The Stylesheets tests which relate to ODD processing (creating schemas and documentation in various formats) should be tested using ODD files from the P5 Exemplars folder. This will help to ensure that:

# We're testing all the features people are likely to use most commonly.
# All P5 changes (assuming we use tei_all as a test file) will be tested as soon as they're introduced, and it should not be so easy for us to neglect to implement Stylesheets support for a P5 change.

===Validate all test input files before using them===

We need to know when a test file goes stale, especially a TEI file. Validation should be part of the test process.

===Name test files meaningfully===

"cocoatest.txt" is more helpful than "test40.txt", obviously. Where larger files are created to test many things, this is harder, of course.

===Output results into a separate folder===

Right now, all the results go into the same folder as the test files, which means the directory is even more cluttered with "test25.xyz" files, making it harder to see what's happening. Results should go into a "results" folder, alongside the "expected-results" folder.

===Keep linked resources and other libs in separate folders===

The Test folder currently contains linked images, various schemas used for validation, and other bits and pieces such as PERL scripts used for cleanup. These should be organized properly. There are also jar files used for general Stylesheets work (in Stylesheets/lib) and others used only for testing (in Stylesheets/Test/lib), with the result that we have multiple versions of Jing, Saxon etc. We should have a single Stylesheets/lib folder containing everything we need for both transformation and testing, and settle on a single version of each jar.

===Process all X*ML output in the same way after creation===

As mentioned above, some XML files are formatted with xmllint before checking, and some are not; some have comments removed and some don't. A single XSLT transformation (I've started one called cleanForDiff.xsl in Test2) should be applied to all X*ML output files before they're compared with expected results.

===Separate out the Oxygen tests===

The current test process includes some tests which depend on an installation of Oxygen, and run transformations using Oxygen's Ant, as well as saxon9ee.jar. These tests properly belong in the oxygen-tei repo, where we should be doing a much better job of testing anyway (see https://github.com/TEIC/TEI/issues/464, which should be converted to an issue in the oxygen-tei repo at some point).

===Try to imagine an alternative to diffing results===

I don't know what this might be, but I do know that the current approach where any change to the Stylesheets or the input resulting in different output fails the build, and the standard way to fix it is to copy the new output over the "expected results" file. It is all too easy to do this without enough care and attention, and to miss unpredicted or unexpected changes hiding alongside expected changes. If we process fewer files in the first place, and strongly encourage the use of good diff tools whenever results change, we can probably do better here, but perhaps there are other approaches to this sort of testing that would help.

A Proposal for Rewriting the Stylesheets Tests

2017-01-09T00:03:44Z

Mholmes: /* Separate out the Oxygen tests */

==The current state of the tests==
The Stylesheets Test folder contains the current testing mechanism, and I've been investigating it over the last few weeks. The set of tests clearly grew by accretion with not much in the way of an overall plan, and predictably it has become an over-complicated collection of inconsistencies, obsolescence and mystery. It basically works like this:

# The tests are run using a Makefile (Test/Makefile).
# The Makefile is divided into a range of targets intended to test different features, such as test-to-html, test-markdown, test-cocoa and so on.
# Each target contains multiple tests which apply one of the conversions supported by the Stylesheets and then check the result in some way.

The conversions are applied in various different ways:

* By running the symlinked command (e.g. bin/teitohtml) which calls the bin/transformtei script
* By running ant and passing one of the ant build files such as /docx/build-to.xml
* By running Saxon and applying one of the XSLT stylesheets directly (such as fo/to.xsl)

There appears to be no rationale governing which method is used to invoke a process.

==Problems with the current tests==
As I've worked on figuring out how the tests work, I've raised a number of bugs and fixed a few (see for instance GitHub issues 213 through 218). The problems fall into these main categories:

* There are dozens of input test files, and very few of them have meaningful names or helpful comments to explain what they're testing (for instance, test14.xml, test2.xml, test25.xml, test26.xml, test3.xml, test4.xml, test6.xml, test31.xml, test36.xml, test23.xml, test17.xml, test20.xml, test27.xml, test22.xml, test28.xml, test8.xml, test32.xml, test24.xml, test12.xml, test5.xml).
* Many test files are already invalid or obsolete due to deprecation of various features in recent versions of P5.
* Different ways of calling the tests (see above) mean that we're not always testing the full process pathway, and it's not clear why.
* Different test results are handled in very different ways for no apparent reason. For instance, some test results are formatted with xmllint before being compared with their matching expected results file, and some are not.
* The approach of processing many small files rather than a small number of large ones means that the test process takes longer, because of the repeated necessity to spin up Java VMs.
* The tests can only be run on *NIX systems which have make installed.

==Principles for rewriting the tests==
I believe it's time to start again with a new approach to testing, and I think we should adopt these principles in doing so:

===Use Ant rather than make===
Ant is already used in the test process anyway. If we abandon make and write the whole thing in Ant, there will be a number of advantages:

* One less technology to install and understand.
* Greater efficiency and therefore speed when other Java-based tasks are invoked.
* Better documentation and commenting.

I've started rewriting a number of the existing tests in the Stylesheets/Test2 directory, using Ant instead of make, and I've encountered no major obstacles so far.

===Try to support Windows===
This is difficult but feasible. In the Stylesheets/Test2 directory I've written an ant buildfile called convert.xml which calls a transformation in two different ways based on the platform it's running on. We know that most people working on the Stylesheets currently are using *NIX platforms, but when you need to be on *NIX before you can even call the command line transformations at all, this is hardly a surprise. Many people use Windows, and we ought to do our best not only to support them but to enable them to contribute to the codebase.

===Use smaller numbers of test files===
Rather than ten small files testing various different things, a better approach would be a single test file which attempts to include a wide range of different features, each documented in-place; adding a new test would then be simpler since it would require only adding a new section to a file rather than constructing a whole new file, which might have features irrelevant to the test which later go stale or obsolete.

===Include test files drawn from P5 Exemplars===
Despite the fact that they're maintained in a separate repository, the Stylesheets are tightly coupled with P5, and development on the two codebases must proceed largely in lock-step fashion. The Stylesheets tests which relate to ODD processing (creating schemas and documentation in various formats) should be tested using ODD files from the P5 Exemplars folder. This will help to ensure that:

# We're testing all the features people are likely to use most commonly.
# All P5 changes (assuming we use tei_all as a test file) will be tested as soon as they're introduced, and it should not be so easy for us to neglect to implement Stylesheets support for a P5 change.

===Validate all test input files before using them===

We need to know when a test file goes stale, especially a TEI file. Validation should be part of the test process.

===Name test files meaningfully===

"cocoatest.txt" is more helpful than "test40.txt", obviously. Where larger files are created to test many things, this is harder, of course.

===Output results into a separate folder===

Right now, all the results go into the same folder as the test files, which means the directory is even more cluttered with "test25.xyz" files, making it harder to see what's happening. Results should go into a "results" folder, alongside the "expected-results" folder.

===Keep linked resources in separate folders===

The Test folder currently contains linked images, various schemas used for validation, and other bits and pieces such as PERL scripts used for cleanup. These should be organized properly.

===Process all X*ML output in the same way after creation===

As mentioned above, some XML files are formatted with xmllint before checking, and some are not; some have comments removed and some don't. A single XSLT transformation (I've started one called cleanForDiff.xsl in Test2) should be applied to all X*ML output files before they're compared with expected results.

===Separate out the Oxygen tests===

The current test process includes some tests which depend on an installation of Oxygen, and run transformations using Oxygen's Ant, as well as saxon9ee.jar. These tests properly belong in the oxygen-tei repo, where we should be doing a much better job of testing anyway (see https://github.com/TEIC/TEI/issues/464, which should be converted to an issue in the oxygen-tei repo at some point).

===Try to imagine an alternative to diffing results===

I don't know what this might be, but I do know that the current approach where any change to the Stylesheets or the input resulting in different output fails the build, and the standard way to fix it is to copy the new output over the "expected results" file. It is all too easy to do this without enough care and attention, and to miss unpredicted or unexpected changes hiding alongside expected changes. If we process fewer files in the first place, and strongly encourage the use of good diff tools whenever results change, we can probably do better here, but perhaps there are other approaches to this sort of testing that would help.

A Proposal for Rewriting the Stylesheets Tests

2017-01-09T00:02:35Z

Mholmes: /* Validate all test input files before using them */

==The current state of the tests==
The Stylesheets Test folder contains the current testing mechanism, and I've been investigating it over the last few weeks. The set of tests clearly grew by accretion with not much in the way of an overall plan, and predictably it has become an over-complicated collection of inconsistencies, obsolescence and mystery. It basically works like this:

# The tests are run using a Makefile (Test/Makefile).
# The Makefile is divided into a range of targets intended to test different features, such as test-to-html, test-markdown, test-cocoa and so on.
# Each target contains multiple tests which apply one of the conversions supported by the Stylesheets and then check the result in some way.

The conversions are applied in various different ways:

* By running the symlinked command (e.g. bin/teitohtml) which calls the bin/transformtei script
* By running ant and passing one of the ant build files such as /docx/build-to.xml
* By running Saxon and applying one of the XSLT stylesheets directly (such as fo/to.xsl)

There appears to be no rationale governing which method is used to invoke a process.

==Problems with the current tests==
As I've worked on figuring out how the tests work, I've raised a number of bugs and fixed a few (see for instance GitHub issues 213 through 218). The problems fall into these main categories:

* There are dozens of input test files, and very few of them have meaningful names or helpful comments to explain what they're testing (for instance, test14.xml, test2.xml, test25.xml, test26.xml, test3.xml, test4.xml, test6.xml, test31.xml, test36.xml, test23.xml, test17.xml, test20.xml, test27.xml, test22.xml, test28.xml, test8.xml, test32.xml, test24.xml, test12.xml, test5.xml).
* Many test files are already invalid or obsolete due to deprecation of various features in recent versions of P5.
* Different ways of calling the tests (see above) mean that we're not always testing the full process pathway, and it's not clear why.
* Different test results are handled in very different ways for no apparent reason. For instance, some test results are formatted with xmllint before being compared with their matching expected results file, and some are not.
* The approach of processing many small files rather than a small number of large ones means that the test process takes longer, because of the repeated necessity to spin up Java VMs.
* The tests can only be run on *NIX systems which have make installed.

==Principles for rewriting the tests==
I believe it's time to start again with a new approach to testing, and I think we should adopt these principles in doing so:

===Use Ant rather than make===
Ant is already used in the test process anyway. If we abandon make and write the whole thing in Ant, there will be a number of advantages:

* One less technology to install and understand.
* Greater efficiency and therefore speed when other Java-based tasks are invoked.
* Better documentation and commenting.

I've started rewriting a number of the existing tests in the Stylesheets/Test2 directory, using Ant instead of make, and I've encountered no major obstacles so far.

===Try to support Windows===
This is difficult but feasible. In the Stylesheets/Test2 directory I've written an ant buildfile called convert.xml which calls a transformation in two different ways based on the platform it's running on. We know that most people working on the Stylesheets currently are using *NIX platforms, but when you need to be on *NIX before you can even call the command line transformations at all, this is hardly a surprise. Many people use Windows, and we ought to do our best not only to support them but to enable them to contribute to the codebase.

===Use smaller numbers of test files===
Rather than ten small files testing various different things, a better approach would be a single test file which attempts to include a wide range of different features, each documented in-place; adding a new test would then be simpler since it would require only adding a new section to a file rather than constructing a whole new file, which might have features irrelevant to the test which later go stale or obsolete.

===Include test files drawn from P5 Exemplars===
Despite the fact that they're maintained in a separate repository, the Stylesheets are tightly coupled with P5, and development on the two codebases must proceed largely in lock-step fashion. The Stylesheets tests which relate to ODD processing (creating schemas and documentation in various formats) should be tested using ODD files from the P5 Exemplars folder. This will help to ensure that:

# We're testing all the features people are likely to use most commonly.
# All P5 changes (assuming we use tei_all as a test file) will be tested as soon as they're introduced, and it should not be so easy for us to neglect to implement Stylesheets support for a P5 change.

===Validate all test input files before using them===

We need to know when a test file goes stale, especially a TEI file. Validation should be part of the test process.

===Name test files meaningfully===

"cocoatest.txt" is more helpful than "test40.txt", obviously. Where larger files are created to test many things, this is harder, of course.

===Output results into a separate folder===

Right now, all the results go into the same folder as the test files, which means the directory is even more cluttered with "test25.xyz" files, making it harder to see what's happening. Results should go into a "results" folder, alongside the "expected-results" folder.

===Keep linked resources in separate folders===

The Test folder currently contains linked images, various schemas used for validation, and other bits and pieces such as PERL scripts used for cleanup. These should be organized properly.

===Process all X*ML output in the same way after creation===

As mentioned above, some XML files are formatted with xmllint before checking, and some are not; some have comments removed and some don't. A single XSLT transformation (I've started one called cleanForDiff.xsl in Test2) should be applied to all X*ML output files before they're compared with expected results.

===Separate out the Oxygen tests===

The current test process includes some tests which depend on an installation of Oxygen, and run transformations using Oxygen's Ant, as well as saxon8ee.jar. These tests properly belong in the oxygen-tei repo, where we should be doing a much better job of testing anyway (see https://github.com/TEIC/TEI/issues/464, which should be converted to an issue in the oxygen-tei repo at some point).

===Try to imagine an alternative to diffing results===

I don't know what this might be, but I do know that the current approach where any change to the Stylesheets or the input resulting in different output fails the build, and the standard way to fix it is to copy the new output over the "expected results" file. It is all too easy to do this without enough care and attention, and to miss unpredicted or unexpected changes hiding alongside expected changes. If we process fewer files in the first place, and strongly encourage the use of good diff tools whenever results change, we can probably do better here, but perhaps there are other approaches to this sort of testing that would help.

A Proposal for Rewriting the Stylesheets Tests

2017-01-09T00:01:48Z

Mholmes: /* Include test files drawn from P5 Exemplars */

==The current state of the tests==
The Stylesheets Test folder contains the current testing mechanism, and I've been investigating it over the last few weeks. The set of tests clearly grew by accretion with not much in the way of an overall plan, and predictably it has become an over-complicated collection of inconsistencies, obsolescence and mystery. It basically works like this:

# The tests are run using a Makefile (Test/Makefile).
# The Makefile is divided into a range of targets intended to test different features, such as test-to-html, test-markdown, test-cocoa and so on.
# Each target contains multiple tests which apply one of the conversions supported by the Stylesheets and then check the result in some way.

The conversions are applied in various different ways:

* By running the symlinked command (e.g. bin/teitohtml) which calls the bin/transformtei script
* By running ant and passing one of the ant build files such as /docx/build-to.xml
* By running Saxon and applying one of the XSLT stylesheets directly (such as fo/to.xsl)

There appears to be no rationale governing which method is used to invoke a process.

==Problems with the current tests==
As I've worked on figuring out how the tests work, I've raised a number of bugs and fixed a few (see for instance GitHub issues 213 through 218). The problems fall into these main categories:

* There are dozens of input test files, and very few of them have meaningful names or helpful comments to explain what they're testing (for instance, test14.xml, test2.xml, test25.xml, test26.xml, test3.xml, test4.xml, test6.xml, test31.xml, test36.xml, test23.xml, test17.xml, test20.xml, test27.xml, test22.xml, test28.xml, test8.xml, test32.xml, test24.xml, test12.xml, test5.xml).
* Many test files are already invalid or obsolete due to deprecation of various features in recent versions of P5.
* Different ways of calling the tests (see above) mean that we're not always testing the full process pathway, and it's not clear why.
* Different test results are handled in very different ways for no apparent reason. For instance, some test results are formatted with xmllint before being compared with their matching expected results file, and some are not.
* The approach of processing many small files rather than a small number of large ones means that the test process takes longer, because of the repeated necessity to spin up Java VMs.
* The tests can only be run on *NIX systems which have make installed.

==Principles for rewriting the tests==
I believe it's time to start again with a new approach to testing, and I think we should adopt these principles in doing so:

===Use Ant rather than make===
Ant is already used in the test process anyway. If we abandon make and write the whole thing in Ant, there will be a number of advantages:

* One less technology to install and understand.
* Greater efficiency and therefore speed when other Java-based tasks are invoked.
* Better documentation and commenting.

I've started rewriting a number of the existing tests in the Stylesheets/Test2 directory, using Ant instead of make, and I've encountered no major obstacles so far.

===Try to support Windows===
This is difficult but feasible. In the Stylesheets/Test2 directory I've written an ant buildfile called convert.xml which calls a transformation in two different ways based on the platform it's running on. We know that most people working on the Stylesheets currently are using *NIX platforms, but when you need to be on *NIX before you can even call the command line transformations at all, this is hardly a surprise. Many people use Windows, and we ought to do our best not only to support them but to enable them to contribute to the codebase.

===Use smaller numbers of test files===
Rather than ten small files testing various different things, a better approach would be a single test file which attempts to include a wide range of different features, each documented in-place; adding a new test would then be simpler since it would require only adding a new section to a file rather than constructing a whole new file, which might have features irrelevant to the test which later go stale or obsolete.

===Include test files drawn from P5 Exemplars===
Despite the fact that they're maintained in a separate repository, the Stylesheets are tightly coupled with P5, and development on the two codebases must proceed largely in lock-step fashion. The Stylesheets tests which relate to ODD processing (creating schemas and documentation in various formats) should be tested using ODD files from the P5 Exemplars folder. This will help to ensure that:

# We're testing all the features people are likely to use most commonly.
# All P5 changes (assuming we use tei_all as a test file) will be tested as soon as they're introduced, and it should not be so easy for us to neglect to implement Stylesheets support for a P5 change.

===Validate all test input files before using them===

We need to know when a test file goes stale, especially a TEI file.

===Name test files meaningfully===

"cocoatest.txt" is more helpful than "test40.txt", obviously. Where larger files are created to test many things, this is harder, of course.

===Output results into a separate folder===

Right now, all the results go into the same folder as the test files, which means the directory is even more cluttered with "test25.xyz" files, making it harder to see what's happening. Results should go into a "results" folder, alongside the "expected-results" folder.

===Keep linked resources in separate folders===

The Test folder currently contains linked images, various schemas used for validation, and other bits and pieces such as PERL scripts used for cleanup. These should be organized properly.

===Process all X*ML output in the same way after creation===

As mentioned above, some XML files are formatted with xmllint before checking, and some are not; some have comments removed and some don't. A single XSLT transformation (I've started one called cleanForDiff.xsl in Test2) should be applied to all X*ML output files before they're compared with expected results.

===Separate out the Oxygen tests===

The current test process includes some tests which depend on an installation of Oxygen, and run transformations using Oxygen's Ant, as well as saxon8ee.jar. These tests properly belong in the oxygen-tei repo, where we should be doing a much better job of testing anyway (see https://github.com/TEIC/TEI/issues/464, which should be converted to an issue in the oxygen-tei repo at some point).

===Try to imagine an alternative to diffing results===

I don't know what this might be, but I do know that the current approach where any change to the Stylesheets or the input resulting in different output fails the build, and the standard way to fix it is to copy the new output over the "expected results" file. It is all too easy to do this without enough care and attention, and to miss unpredicted or unexpected changes hiding alongside expected changes. If we process fewer files in the first place, and strongly encourage the use of good diff tools whenever results change, we can probably do better here, but perhaps there are other approaches to this sort of testing that would help.

A Proposal for Rewriting the Stylesheets Tests

2017-01-09T00:00:55Z

Mholmes: /* Try to support Windows */

==The current state of the tests==
The Stylesheets Test folder contains the current testing mechanism, and I've been investigating it over the last few weeks. The set of tests clearly grew by accretion with not much in the way of an overall plan, and predictably it has become an over-complicated collection of inconsistencies, obsolescence and mystery. It basically works like this:

# The tests are run using a Makefile (Test/Makefile).
# The Makefile is divided into a range of targets intended to test different features, such as test-to-html, test-markdown, test-cocoa and so on.
# Each target contains multiple tests which apply one of the conversions supported by the Stylesheets and then check the result in some way.

The conversions are applied in various different ways:

* By running the symlinked command (e.g. bin/teitohtml) which calls the bin/transformtei script
* By running ant and passing one of the ant build files such as /docx/build-to.xml
* By running Saxon and applying one of the XSLT stylesheets directly (such as fo/to.xsl)

There appears to be no rationale governing which method is used to invoke a process.

==Problems with the current tests==
As I've worked on figuring out how the tests work, I've raised a number of bugs and fixed a few (see for instance GitHub issues 213 through 218). The problems fall into these main categories:

* There are dozens of input test files, and very few of them have meaningful names or helpful comments to explain what they're testing (for instance, test14.xml, test2.xml, test25.xml, test26.xml, test3.xml, test4.xml, test6.xml, test31.xml, test36.xml, test23.xml, test17.xml, test20.xml, test27.xml, test22.xml, test28.xml, test8.xml, test32.xml, test24.xml, test12.xml, test5.xml).
* Many test files are already invalid or obsolete due to deprecation of various features in recent versions of P5.
* Different ways of calling the tests (see above) mean that we're not always testing the full process pathway, and it's not clear why.
* Different test results are handled in very different ways for no apparent reason. For instance, some test results are formatted with xmllint before being compared with their matching expected results file, and some are not.
* The approach of processing many small files rather than a small number of large ones means that the test process takes longer, because of the repeated necessity to spin up Java VMs.
* The tests can only be run on *NIX systems which have make installed.

==Principles for rewriting the tests==
I believe it's time to start again with a new approach to testing, and I think we should adopt these principles in doing so:

===Use Ant rather than make===
Ant is already used in the test process anyway. If we abandon make and write the whole thing in Ant, there will be a number of advantages:

* One less technology to install and understand.
* Greater efficiency and therefore speed when other Java-based tasks are invoked.
* Better documentation and commenting.

I've started rewriting a number of the existing tests in the Stylesheets/Test2 directory, using Ant instead of make, and I've encountered no major obstacles so far.

===Try to support Windows===
This is difficult but feasible. In the Stylesheets/Test2 directory I've written an ant buildfile called convert.xml which calls a transformation in two different ways based on the platform it's running on. We know that most people working on the Stylesheets currently are using *NIX platforms, but when you need to be on *NIX before you can even call the command line transformations at all, this is hardly a surprise. Many people use Windows, and we ought to do our best not only to support them but to enable them to contribute to the codebase.

===Use smaller numbers of test files===
Rather than ten small files testing various different things, a better approach would be a single test file which attempts to include a wide range of different features, each documented in-place; adding a new test would then be simpler since it would require only adding a new section to a file rather than constructing a whole new file, which might have features irrelevant to the test which later go stale or obsolete.

===Include test files drawn from P5 Exemplars===
Despite the fact that they're maintained in a separate repository, the Stylesheets are tightly coupled with P5, and development on the two codebases must proceed largely in lock-step fashion. The Stylesheets tests which relate to ODD processing (creating schemas and documentation in various formats) should be tested using ODD files from the P5 Exemplars folder. This will help to ensure that:

1. We're testing all the features people are likely to use most commonly.
2. All P5 changes (assuming we use tei_all as a test file) will be tested as soon as they're introduced, and it should not be so easy for us to neglect to implement Stylesheets support for a P5 change.

===Validate all test input files before using them===

We need to know when a test file goes stale, especially a TEI file.

===Name test files meaningfully===

"cocoatest.txt" is more helpful than "test40.txt", obviously. Where larger files are created to test many things, this is harder, of course.

===Output results into a separate folder===

Right now, all the results go into the same folder as the test files, which means the directory is even more cluttered with "test25.xyz" files, making it harder to see what's happening. Results should go into a "results" folder, alongside the "expected-results" folder.

===Keep linked resources in separate folders===

The Test folder currently contains linked images, various schemas used for validation, and other bits and pieces such as PERL scripts used for cleanup. These should be organized properly.

===Process all X*ML output in the same way after creation===

As mentioned above, some XML files are formatted with xmllint before checking, and some are not; some have comments removed and some don't. A single XSLT transformation (I've started one called cleanForDiff.xsl in Test2) should be applied to all X*ML output files before they're compared with expected results.

===Separate out the Oxygen tests===

The current test process includes some tests which depend on an installation of Oxygen, and run transformations using Oxygen's Ant, as well as saxon8ee.jar. These tests properly belong in the oxygen-tei repo, where we should be doing a much better job of testing anyway (see https://github.com/TEIC/TEI/issues/464, which should be converted to an issue in the oxygen-tei repo at some point).

===Try to imagine an alternative to diffing results===

I don't know what this might be, but I do know that the current approach where any change to the Stylesheets or the input resulting in different output fails the build, and the standard way to fix it is to copy the new output over the "expected results" file. It is all too easy to do this without enough care and attention, and to miss unpredicted or unexpected changes hiding alongside expected changes. If we process fewer files in the first place, and strongly encourage the use of good diff tools whenever results change, we can probably do better here, but perhaps there are other approaches to this sort of testing that would help.

A Proposal for Rewriting the Stylesheets Tests

2017-01-08T23:58:52Z

Mholmes: /* Separate out the Oxygen tests */

==The current state of the tests==
The Stylesheets Test folder contains the current testing mechanism, and I've been investigating it over the last few weeks. The set of tests clearly grew by accretion with not much in the way of an overall plan, and predictably it has become an over-complicated collection of inconsistencies, obsolescence and mystery. It basically works like this:

# The tests are run using a Makefile (Test/Makefile).
# The Makefile is divided into a range of targets intended to test different features, such as test-to-html, test-markdown, test-cocoa and so on.
# Each target contains multiple tests which apply one of the conversions supported by the Stylesheets and then check the result in some way.

The conversions are applied in various different ways:

* By running the symlinked command (e.g. bin/teitohtml) which calls the bin/transformtei script
* By running ant and passing one of the ant build files such as /docx/build-to.xml
* By running Saxon and applying one of the XSLT stylesheets directly (such as fo/to.xsl)

There appears to be no rationale governing which method is used to invoke a process.

==Problems with the current tests==
As I've worked on figuring out how the tests work, I've raised a number of bugs and fixed a few (see for instance GitHub issues 213 through 218). The problems fall into these main categories:

* There are dozens of input test files, and very few of them have meaningful names or helpful comments to explain what they're testing (for instance, test14.xml, test2.xml, test25.xml, test26.xml, test3.xml, test4.xml, test6.xml, test31.xml, test36.xml, test23.xml, test17.xml, test20.xml, test27.xml, test22.xml, test28.xml, test8.xml, test32.xml, test24.xml, test12.xml, test5.xml).
* Many test files are already invalid or obsolete due to deprecation of various features in recent versions of P5.
* Different ways of calling the tests (see above) mean that we're not always testing the full process pathway, and it's not clear why.
* Different test results are handled in very different ways for no apparent reason. For instance, some test results are formatted with xmllint before being compared with their matching expected results file, and some are not.
* The approach of processing many small files rather than a small number of large ones means that the test process takes longer, because of the repeated necessity to spin up Java VMs.
* The tests can only be run on *NIX systems which have make installed.

==Principles for rewriting the tests==
I believe it's time to start again with a new approach to testing, and I think we should adopt these principles in doing so:

===Use Ant rather than make===
Ant is already used in the test process anyway. If we abandon make and write the whole thing in Ant, there will be a number of advantages:

* One less technology to install and understand.
* Greater efficiency and therefore speed when other Java-based tasks are invoked.
* Better documentation and commenting.

I've started rewriting a number of the existing tests in the Stylesheets/Test2 directory, using Ant instead of make, and I've encountered no major obstacles so far.

===Try to support Windows===
This is difficult but feasible; I have a pilot project In the Stylesheets/Test2 directory I've written an ant buildfile called convert.xml which calls a transformation in two different ways based on the platform it's running on. We know that most people working on the Stylesheets currently are using *NIX platforms, but when you need to be on *NIX before you can even call the command line transformations at all, this is hardly a surprise. Many people use Windows, and we ought to do our best not only to support them but to enable them to contribute to the codebase.

===Use smaller numbers of test files===
Rather than ten small files testing various different things, a better approach would be a single test file which attempts to include a wide range of different features, each documented in-place; adding a new test would then be simpler since it would require only adding a new section to a file rather than constructing a whole new file, which might have features irrelevant to the test which later go stale or obsolete.

===Include test files drawn from P5 Exemplars===
Despite the fact that they're maintained in a separate repository, the Stylesheets are tightly coupled with P5, and development on the two codebases must proceed largely in lock-step fashion. The Stylesheets tests which relate to ODD processing (creating schemas and documentation in various formats) should be tested using ODD files from the P5 Exemplars folder. This will help to ensure that:

1. We're testing all the features people are likely to use most commonly.
2. All P5 changes (assuming we use tei_all as a test file) will be tested as soon as they're introduced, and it should not be so easy for us to neglect to implement Stylesheets support for a P5 change.

===Validate all test input files before using them===

We need to know when a test file goes stale, especially a TEI file.

===Name test files meaningfully===

"cocoatest.txt" is more helpful than "test40.txt", obviously. Where larger files are created to test many things, this is harder, of course.

===Output results into a separate folder===

Right now, all the results go into the same folder as the test files, which means the directory is even more cluttered with "test25.xyz" files, making it harder to see what's happening. Results should go into a "results" folder, alongside the "expected-results" folder.

===Keep linked resources in separate folders===

The Test folder currently contains linked images, various schemas used for validation, and other bits and pieces such as PERL scripts used for cleanup. These should be organized properly.

===Process all X*ML output in the same way after creation===

As mentioned above, some XML files are formatted with xmllint before checking, and some are not; some have comments removed and some don't. A single XSLT transformation (I've started one called cleanForDiff.xsl in Test2) should be applied to all X*ML output files before they're compared with expected results.

===Separate out the Oxygen tests===

The current test process includes some tests which depend on an installation of Oxygen, and run transformations using Oxygen's Ant, as well as saxon8ee.jar. These tests properly belong in the oxygen-tei repo, where we should be doing a much better job of testing anyway (see https://github.com/TEIC/TEI/issues/464, which should be converted to an issue in the oxygen-tei repo at some point).

===Try to imagine an alternative to diffing results===

I don't know what this might be, but I do know that the current approach where any change to the Stylesheets or the input resulting in different output fails the build, and the standard way to fix it is to copy the new output over the "expected results" file. It is all too easy to do this without enough care and attention, and to miss unpredicted or unexpected changes hiding alongside expected changes. If we process fewer files in the first place, and strongly encourage the use of good diff tools whenever results change, we can probably do better here, but perhaps there are other approaches to this sort of testing that would help.

A Proposal for Rewriting the Stylesheets Tests

2017-01-08T23:53:31Z

Mholmes: /* Smaller numbers of test files */

==The current state of the tests==
The Stylesheets Test folder contains the current testing mechanism, and I've been investigating it over the last few weeks. The set of tests clearly grew by accretion with not much in the way of an overall plan, and predictably it has become an over-complicated collection of inconsistencies, obsolescence and mystery. It basically works like this:

# The tests are run using a Makefile (Test/Makefile).
# The Makefile is divided into a range of targets intended to test different features, such as test-to-html, test-markdown, test-cocoa and so on.
# Each target contains multiple tests which apply one of the conversions supported by the Stylesheets and then check the result in some way.

The conversions are applied in various different ways:

* By running the symlinked command (e.g. bin/teitohtml) which calls the bin/transformtei script
* By running ant and passing one of the ant build files such as /docx/build-to.xml
* By running Saxon and applying one of the XSLT stylesheets directly (such as fo/to.xsl)

There appears to be no rationale governing which method is used to invoke a process.

==Problems with the current tests==
As I've worked on figuring out how the tests work, I've raised a number of bugs and fixed a few (see for instance GitHub issues 213 through 218). The problems fall into these main categories:

* There are dozens of input test files, and very few of them have meaningful names or helpful comments to explain what they're testing (for instance, test14.xml, test2.xml, test25.xml, test26.xml, test3.xml, test4.xml, test6.xml, test31.xml, test36.xml, test23.xml, test17.xml, test20.xml, test27.xml, test22.xml, test28.xml, test8.xml, test32.xml, test24.xml, test12.xml, test5.xml).
* Many test files are already invalid or obsolete due to deprecation of various features in recent versions of P5.
* Different ways of calling the tests (see above) mean that we're not always testing the full process pathway, and it's not clear why.
* Different test results are handled in very different ways for no apparent reason. For instance, some test results are formatted with xmllint before being compared with their matching expected results file, and some are not.
* The approach of processing many small files rather than a small number of large ones means that the test process takes longer, because of the repeated necessity to spin up Java VMs.
* The tests can only be run on *NIX systems which have make installed.

==Principles for rewriting the tests==
I believe it's time to start again with a new approach to testing, and I think we should adopt these principles in doing so:

===Use Ant rather than make===
Ant is already used in the test process anyway. If we abandon make and write the whole thing in Ant, there will be a number of advantages:

* One less technology to install and understand.
* Greater efficiency and therefore speed when other Java-based tasks are invoked.
* Better documentation and commenting.

I've started rewriting a number of the existing tests in the Stylesheets/Test2 directory, using Ant instead of make, and I've encountered no major obstacles so far.

===Try to support Windows===
This is difficult but feasible; I have a pilot project In the Stylesheets/Test2 directory I've written an ant buildfile called convert.xml which calls a transformation in two different ways based on the platform it's running on. We know that most people working on the Stylesheets currently are using *NIX platforms, but when you need to be on *NIX before you can even call the command line transformations at all, this is hardly a surprise. Many people use Windows, and we ought to do our best not only to support them but to enable them to contribute to the codebase.

===Use smaller numbers of test files===
Rather than ten small files testing various different things, a better approach would be a single test file which attempts to include a wide range of different features, each documented in-place; adding a new test would then be simpler since it would require only adding a new section to a file rather than constructing a whole new file, which might have features irrelevant to the test which later go stale or obsolete.

===Include test files drawn from P5 Exemplars===
Despite the fact that they're maintained in a separate repository, the Stylesheets are tightly coupled with P5, and development on the two codebases must proceed largely in lock-step fashion. The Stylesheets tests which relate to ODD processing (creating schemas and documentation in various formats) should be tested using ODD files from the P5 Exemplars folder. This will help to ensure that:

1. We're testing all the features people are likely to use most commonly.
2. All P5 changes (assuming we use tei_all as a test file) will be tested as soon as they're introduced, and it should not be so easy for us to neglect to implement Stylesheets support for a P5 change.

===Validate all test input files before using them===

We need to know when a test file goes stale, especially a TEI file.

===Name test files meaningfully===

"cocoatest.txt" is more helpful than "test40.txt", obviously. Where larger files are created to test many things, this is harder, of course.

===Output results into a separate folder===

Right now, all the results go into the same folder as the test files, which means the directory is even more cluttered with "test25.xyz" files, making it harder to see what's happening. Results should go into a "results" folder, alongside the "expected-results" folder.

===Keep linked resources in separate folders===

The Test folder currently contains linked images, various schemas used for validation, and other bits and pieces such as PERL scripts used for cleanup. These should be organized properly.

===Process all X*ML output in the same way after creation===

As mentioned above, some XML files are formatted with xmllint before checking, and some are not; some have comments removed and some don't. A single XSLT transformation (I've started one called cleanForDiff.xsl in Test2) should be applied to all X*ML output files before they're compared with expected results.

===Separate out the Oxygen tests===

The current test process includes some tests which depend on an installation of Oxygen, and run transformations using Oxygen's Ant, as well as saxon8ee.jar. These tests properly belong in the oxygen-tei repo, where we should be doing a much better job of testing anyway (see https://github.com/TEIC/TEI/issues/464, which should be converted to an issue in the oxygen-tei repo at some point).

A Proposal for Rewriting the Stylesheets Tests

2017-01-08T23:53:15Z

Mholmes: /* Cross-platform support */

==The current state of the tests==
The Stylesheets Test folder contains the current testing mechanism, and I've been investigating it over the last few weeks. The set of tests clearly grew by accretion with not much in the way of an overall plan, and predictably it has become an over-complicated collection of inconsistencies, obsolescence and mystery. It basically works like this:

# The tests are run using a Makefile (Test/Makefile).
# The Makefile is divided into a range of targets intended to test different features, such as test-to-html, test-markdown, test-cocoa and so on.
# Each target contains multiple tests which apply one of the conversions supported by the Stylesheets and then check the result in some way.

The conversions are applied in various different ways:

* By running the symlinked command (e.g. bin/teitohtml) which calls the bin/transformtei script
* By running ant and passing one of the ant build files such as /docx/build-to.xml
* By running Saxon and applying one of the XSLT stylesheets directly (such as fo/to.xsl)

There appears to be no rationale governing which method is used to invoke a process.

==Problems with the current tests==
As I've worked on figuring out how the tests work, I've raised a number of bugs and fixed a few (see for instance GitHub issues 213 through 218). The problems fall into these main categories:

* There are dozens of input test files, and very few of them have meaningful names or helpful comments to explain what they're testing (for instance, test14.xml, test2.xml, test25.xml, test26.xml, test3.xml, test4.xml, test6.xml, test31.xml, test36.xml, test23.xml, test17.xml, test20.xml, test27.xml, test22.xml, test28.xml, test8.xml, test32.xml, test24.xml, test12.xml, test5.xml).
* Many test files are already invalid or obsolete due to deprecation of various features in recent versions of P5.
* Different ways of calling the tests (see above) mean that we're not always testing the full process pathway, and it's not clear why.
* Different test results are handled in very different ways for no apparent reason. For instance, some test results are formatted with xmllint before being compared with their matching expected results file, and some are not.
* The approach of processing many small files rather than a small number of large ones means that the test process takes longer, because of the repeated necessity to spin up Java VMs.
* The tests can only be run on *NIX systems which have make installed.

==Principles for rewriting the tests==
I believe it's time to start again with a new approach to testing, and I think we should adopt these principles in doing so:

===Use Ant rather than make===
Ant is already used in the test process anyway. If we abandon make and write the whole thing in Ant, there will be a number of advantages:

* One less technology to install and understand.
* Greater efficiency and therefore speed when other Java-based tasks are invoked.
* Better documentation and commenting.

I've started rewriting a number of the existing tests in the Stylesheets/Test2 directory, using Ant instead of make, and I've encountered no major obstacles so far.

===Try to support Windows===
This is difficult but feasible; I have a pilot project In the Stylesheets/Test2 directory I've written an ant buildfile called convert.xml which calls a transformation in two different ways based on the platform it's running on. We know that most people working on the Stylesheets currently are using *NIX platforms, but when you need to be on *NIX before you can even call the command line transformations at all, this is hardly a surprise. Many people use Windows, and we ought to do our best not only to support them but to enable them to contribute to the codebase.

===Smaller numbers of test files===
Rather than ten small files testing various different things, a better approach would be a single test file which attempts to include a wide range of different features, each documented in-place; adding a new test would then be simpler since it would require only adding a new section to a file rather than constructing a whole new file, which might have features irrelevant to the test which later go stale or obsolete.

===Include test files drawn from P5 Exemplars===
Despite the fact that they're maintained in a separate repository, the Stylesheets are tightly coupled with P5, and development on the two codebases must proceed largely in lock-step fashion. The Stylesheets tests which relate to ODD processing (creating schemas and documentation in various formats) should be tested using ODD files from the P5 Exemplars folder. This will help to ensure that:

1. We're testing all the features people are likely to use most commonly.
2. All P5 changes (assuming we use tei_all as a test file) will be tested as soon as they're introduced, and it should not be so easy for us to neglect to implement Stylesheets support for a P5 change.

===Validate all test input files before using them===

We need to know when a test file goes stale, especially a TEI file.

===Name test files meaningfully===

"cocoatest.txt" is more helpful than "test40.txt", obviously. Where larger files are created to test many things, this is harder, of course.

===Output results into a separate folder===

Right now, all the results go into the same folder as the test files, which means the directory is even more cluttered with "test25.xyz" files, making it harder to see what's happening. Results should go into a "results" folder, alongside the "expected-results" folder.

===Keep linked resources in separate folders===

The Test folder currently contains linked images, various schemas used for validation, and other bits and pieces such as PERL scripts used for cleanup. These should be organized properly.

===Process all X*ML output in the same way after creation===

As mentioned above, some XML files are formatted with xmllint before checking, and some are not; some have comments removed and some don't. A single XSLT transformation (I've started one called cleanForDiff.xsl in Test2) should be applied to all X*ML output files before they're compared with expected results.

===Separate out the Oxygen tests===

The current test process includes some tests which depend on an installation of Oxygen, and run transformations using Oxygen's Ant, as well as saxon8ee.jar. These tests properly belong in the oxygen-tei repo, where we should be doing a much better job of testing anyway (see https://github.com/TEIC/TEI/issues/464, which should be converted to an issue in the oxygen-tei repo at some point).

A Proposal for Rewriting the Stylesheets Tests

2017-01-08T23:52:02Z

Mholmes: /* The current state of the tests */

==The current state of the tests==
The Stylesheets Test folder contains the current testing mechanism, and I've been investigating it over the last few weeks. The set of tests clearly grew by accretion with not much in the way of an overall plan, and predictably it has become an over-complicated collection of inconsistencies, obsolescence and mystery. It basically works like this:

# The tests are run using a Makefile (Test/Makefile).
# The Makefile is divided into a range of targets intended to test different features, such as test-to-html, test-markdown, test-cocoa and so on.
# Each target contains multiple tests which apply one of the conversions supported by the Stylesheets and then check the result in some way.

The conversions are applied in various different ways:

* By running the symlinked command (e.g. bin/teitohtml) which calls the bin/transformtei script
* By running ant and passing one of the ant build files such as /docx/build-to.xml
* By running Saxon and applying one of the XSLT stylesheets directly (such as fo/to.xsl)

There appears to be no rationale governing which method is used to invoke a process.

==Problems with the current tests==
As I've worked on figuring out how the tests work, I've raised a number of bugs and fixed a few (see for instance GitHub issues 213 through 218). The problems fall into these main categories:

* There are dozens of input test files, and very few of them have meaningful names or helpful comments to explain what they're testing (for instance, test14.xml, test2.xml, test25.xml, test26.xml, test3.xml, test4.xml, test6.xml, test31.xml, test36.xml, test23.xml, test17.xml, test20.xml, test27.xml, test22.xml, test28.xml, test8.xml, test32.xml, test24.xml, test12.xml, test5.xml).
* Many test files are already invalid or obsolete due to deprecation of various features in recent versions of P5.
* Different ways of calling the tests (see above) mean that we're not always testing the full process pathway, and it's not clear why.
* Different test results are handled in very different ways for no apparent reason. For instance, some test results are formatted with xmllint before being compared with their matching expected results file, and some are not.
* The approach of processing many small files rather than a small number of large ones means that the test process takes longer, because of the repeated necessity to spin up Java VMs.
* The tests can only be run on *NIX systems which have make installed.

==Principles for rewriting the tests==
I believe it's time to start again with a new approach to testing, and I think we should adopt these principles in doing so:

===Use Ant rather than make===
Ant is already used in the test process anyway. If we abandon make and write the whole thing in Ant, there will be a number of advantages:

* One less technology to install and understand.
* Greater efficiency and therefore speed when other Java-based tasks are invoked.
* Better documentation and commenting.

I've started rewriting a number of the existing tests in the Stylesheets/Test2 directory, using Ant instead of make, and I've encountered no major obstacles so far.

===Cross-platform support===
This is difficult but feasible; I have a pilot project In the Stylesheets/Test2 directory I've written an ant buildfile called convert.xml which calls a transformation in two different ways based on the platform it's running on. We know that most people working on the Stylesheets currently are using *NIX platforms, but when you need to be on *NIX before you can even call the command line transformations at all, this is hardly a surprise. Many people use Windows, and we ought to do our best not only to support them but to enable them to contribute to the codebase.

===Smaller numbers of test files===
Rather than ten small files testing various different things, a better approach would be a single test file which attempts to include a wide range of different features, each documented in-place; adding a new test would then be simpler since it would require only adding a new section to a file rather than constructing a whole new file, which might have features irrelevant to the test which later go stale or obsolete.

===Include test files drawn from P5 Exemplars===
Despite the fact that they're maintained in a separate repository, the Stylesheets are tightly coupled with P5, and development on the two codebases must proceed largely in lock-step fashion. The Stylesheets tests which relate to ODD processing (creating schemas and documentation in various formats) should be tested using ODD files from the P5 Exemplars folder. This will help to ensure that:

1. We're testing all the features people are likely to use most commonly.
2. All P5 changes (assuming we use tei_all as a test file) will be tested as soon as they're introduced, and it should not be so easy for us to neglect to implement Stylesheets support for a P5 change.

===Validate all test input files before using them===

We need to know when a test file goes stale, especially a TEI file.

===Name test files meaningfully===

"cocoatest.txt" is more helpful than "test40.txt", obviously. Where larger files are created to test many things, this is harder, of course.

===Output results into a separate folder===

Right now, all the results go into the same folder as the test files, which means the directory is even more cluttered with "test25.xyz" files, making it harder to see what's happening. Results should go into a "results" folder, alongside the "expected-results" folder.

===Keep linked resources in separate folders===

The Test folder currently contains linked images, various schemas used for validation, and other bits and pieces such as PERL scripts used for cleanup. These should be organized properly.

===Process all X*ML output in the same way after creation===

As mentioned above, some XML files are formatted with xmllint before checking, and some are not; some have comments removed and some don't. A single XSLT transformation (I've started one called cleanForDiff.xsl in Test2) should be applied to all X*ML output files before they're compared with expected results.

===Separate out the Oxygen tests===

The current test process includes some tests which depend on an installation of Oxygen, and run transformations using Oxygen's Ant, as well as saxon8ee.jar. These tests properly belong in the oxygen-tei repo, where we should be doing a much better job of testing anyway (see https://github.com/TEIC/TEI/issues/464, which should be converted to an issue in the oxygen-tei repo at some point).

A Proposal for Rewriting the Stylesheets Tests

2017-01-08T23:51:25Z

Mholmes: /* The current state of the tests */

==The current state of the tests==
The Stylesheets Test folder contains the current testing mechanism, and I've been investigating it over the last few weeks. The set of tests clearly grew by accretion with not much in the way of an overall plan, and predictably it has become an over-complicated collection of inconsistencies, obsolescence and mystery. It basically works like this:

1. The tests are run using a Makefile (Test/Makefile).

2. The Makefile is divided into a range of targets intended to test different features, such as test-to-html, test-markdown, test-cocoa and so on.

3. Each target contains multiple tests which apply one of the conversions supported by the Stylesheets and then check the result in some way.

The conversions are applied in various different ways:

* By running the symlinked command (e.g. bin/teitohtml) which calls the bin/transformtei script
* By running ant and passing one of the ant build files such as /docx/build-to.xml
* By running Saxon and applying one of the XSLT stylesheets directly (such as fo/to.xsl)

There appears to be no rationale governing which method is used to invoke a process.

==Problems with the current tests==
As I've worked on figuring out how the tests work, I've raised a number of bugs and fixed a few (see for instance GitHub issues 213 through 218). The problems fall into these main categories:

* There are dozens of input test files, and very few of them have meaningful names or helpful comments to explain what they're testing (for instance, test14.xml, test2.xml, test25.xml, test26.xml, test3.xml, test4.xml, test6.xml, test31.xml, test36.xml, test23.xml, test17.xml, test20.xml, test27.xml, test22.xml, test28.xml, test8.xml, test32.xml, test24.xml, test12.xml, test5.xml).
* Many test files are already invalid or obsolete due to deprecation of various features in recent versions of P5.
* Different ways of calling the tests (see above) mean that we're not always testing the full process pathway, and it's not clear why.
* Different test results are handled in very different ways for no apparent reason. For instance, some test results are formatted with xmllint before being compared with their matching expected results file, and some are not.
* The approach of processing many small files rather than a small number of large ones means that the test process takes longer, because of the repeated necessity to spin up Java VMs.
* The tests can only be run on *NIX systems which have make installed.

==Principles for rewriting the tests==
I believe it's time to start again with a new approach to testing, and I think we should adopt these principles in doing so:

===Use Ant rather than make===
Ant is already used in the test process anyway. If we abandon make and write the whole thing in Ant, there will be a number of advantages:

* One less technology to install and understand.
* Greater efficiency and therefore speed when other Java-based tasks are invoked.
* Better documentation and commenting.

I've started rewriting a number of the existing tests in the Stylesheets/Test2 directory, using Ant instead of make, and I've encountered no major obstacles so far.

===Cross-platform support===
This is difficult but feasible; I have a pilot project In the Stylesheets/Test2 directory I've written an ant buildfile called convert.xml which calls a transformation in two different ways based on the platform it's running on. We know that most people working on the Stylesheets currently are using *NIX platforms, but when you need to be on *NIX before you can even call the command line transformations at all, this is hardly a surprise. Many people use Windows, and we ought to do our best not only to support them but to enable them to contribute to the codebase.

===Smaller numbers of test files===
Rather than ten small files testing various different things, a better approach would be a single test file which attempts to include a wide range of different features, each documented in-place; adding a new test would then be simpler since it would require only adding a new section to a file rather than constructing a whole new file, which might have features irrelevant to the test which later go stale or obsolete.

===Include test files drawn from P5 Exemplars===
Despite the fact that they're maintained in a separate repository, the Stylesheets are tightly coupled with P5, and development on the two codebases must proceed largely in lock-step fashion. The Stylesheets tests which relate to ODD processing (creating schemas and documentation in various formats) should be tested using ODD files from the P5 Exemplars folder. This will help to ensure that:

1. We're testing all the features people are likely to use most commonly.
2. All P5 changes (assuming we use tei_all as a test file) will be tested as soon as they're introduced, and it should not be so easy for us to neglect to implement Stylesheets support for a P5 change.

===Validate all test input files before using them===

We need to know when a test file goes stale, especially a TEI file.

===Name test files meaningfully===

"cocoatest.txt" is more helpful than "test40.txt", obviously. Where larger files are created to test many things, this is harder, of course.

===Output results into a separate folder===

Right now, all the results go into the same folder as the test files, which means the directory is even more cluttered with "test25.xyz" files, making it harder to see what's happening. Results should go into a "results" folder, alongside the "expected-results" folder.

===Keep linked resources in separate folders===

The Test folder currently contains linked images, various schemas used for validation, and other bits and pieces such as PERL scripts used for cleanup. These should be organized properly.

===Process all X*ML output in the same way after creation===

As mentioned above, some XML files are formatted with xmllint before checking, and some are not; some have comments removed and some don't. A single XSLT transformation (I've started one called cleanForDiff.xsl in Test2) should be applied to all X*ML output files before they're compared with expected results.

===Separate out the Oxygen tests===

The current test process includes some tests which depend on an installation of Oxygen, and run transformations using Oxygen's Ant, as well as saxon8ee.jar. These tests properly belong in the oxygen-tei repo, where we should be doing a much better job of testing anyway (see https://github.com/TEIC/TEI/issues/464, which should be converted to an issue in the oxygen-tei repo at some point).

A Proposal for Rewriting the Stylesheets Tests

2017-01-08T23:50:18Z

Mholmes: /* Process all X*ML output in the same way after creation */

==The current state of the tests==
The Stylesheets Test folder contains the current testing mechanism, and I've been investigating it over the last few weeks. The set of tests clearly grew by accretion with not much in the way of an overall plan, and predictably it has become an over-complicated collection of inconsistencies, obsolescence and mystery. It basically works like this:

1. The tests are run using a Makefile (Test/Makefile).
2. The Makefile is divided into a range of targets intended to test different features, such as test-to-html, test-markdown, test-cocoa and so on.
3. Each target contains multiple tests which apply one of the conversions supported by the Stylesheets and then check the result in some way.

The conversions are applied in various different ways:

* By running the symlinked command (e.g. bin/teitohtml) which calls the bin/transformtei script
* By running ant and passing one of the ant build files such as /docx/build-to.xml
* By running Saxon and applying one of the XSLT stylesheets directly (such as fo/to.xsl)

There appears to be no rationale governing which method is used to invoke a process.

==Problems with the current tests==
As I've worked on figuring out how the tests work, I've raised a number of bugs and fixed a few (see for instance GitHub issues 213 through 218). The problems fall into these main categories:

* There are dozens of input test files, and very few of them have meaningful names or helpful comments to explain what they're testing (for instance, test14.xml, test2.xml, test25.xml, test26.xml, test3.xml, test4.xml, test6.xml, test31.xml, test36.xml, test23.xml, test17.xml, test20.xml, test27.xml, test22.xml, test28.xml, test8.xml, test32.xml, test24.xml, test12.xml, test5.xml).
* Many test files are already invalid or obsolete due to deprecation of various features in recent versions of P5.
* Different ways of calling the tests (see above) mean that we're not always testing the full process pathway, and it's not clear why.
* Different test results are handled in very different ways for no apparent reason. For instance, some test results are formatted with xmllint before being compared with their matching expected results file, and some are not.
* The approach of processing many small files rather than a small number of large ones means that the test process takes longer, because of the repeated necessity to spin up Java VMs.
* The tests can only be run on *NIX systems which have make installed.

==Principles for rewriting the tests==
I believe it's time to start again with a new approach to testing, and I think we should adopt these principles in doing so:

===Use Ant rather than make===
Ant is already used in the test process anyway. If we abandon make and write the whole thing in Ant, there will be a number of advantages:

* One less technology to install and understand.
* Greater efficiency and therefore speed when other Java-based tasks are invoked.
* Better documentation and commenting.

I've started rewriting a number of the existing tests in the Stylesheets/Test2 directory, using Ant instead of make, and I've encountered no major obstacles so far.

===Cross-platform support===
This is difficult but feasible; I have a pilot project In the Stylesheets/Test2 directory I've written an ant buildfile called convert.xml which calls a transformation in two different ways based on the platform it's running on. We know that most people working on the Stylesheets currently are using *NIX platforms, but when you need to be on *NIX before you can even call the command line transformations at all, this is hardly a surprise. Many people use Windows, and we ought to do our best not only to support them but to enable them to contribute to the codebase.

===Smaller numbers of test files===
Rather than ten small files testing various different things, a better approach would be a single test file which attempts to include a wide range of different features, each documented in-place; adding a new test would then be simpler since it would require only adding a new section to a file rather than constructing a whole new file, which might have features irrelevant to the test which later go stale or obsolete.

===Include test files drawn from P5 Exemplars===
Despite the fact that they're maintained in a separate repository, the Stylesheets are tightly coupled with P5, and development on the two codebases must proceed largely in lock-step fashion. The Stylesheets tests which relate to ODD processing (creating schemas and documentation in various formats) should be tested using ODD files from the P5 Exemplars folder. This will help to ensure that:

1. We're testing all the features people are likely to use most commonly.
2. All P5 changes (assuming we use tei_all as a test file) will be tested as soon as they're introduced, and it should not be so easy for us to neglect to implement Stylesheets support for a P5 change.

===Validate all test input files before using them===

We need to know when a test file goes stale, especially a TEI file.

===Name test files meaningfully===

"cocoatest.txt" is more helpful than "test40.txt", obviously. Where larger files are created to test many things, this is harder, of course.

===Output results into a separate folder===

Right now, all the results go into the same folder as the test files, which means the directory is even more cluttered with "test25.xyz" files, making it harder to see what's happening. Results should go into a "results" folder, alongside the "expected-results" folder.

===Keep linked resources in separate folders===

The Test folder currently contains linked images, various schemas used for validation, and other bits and pieces such as PERL scripts used for cleanup. These should be organized properly.

===Process all X*ML output in the same way after creation===

As mentioned above, some XML files are formatted with xmllint before checking, and some are not; some have comments removed and some don't. A single XSLT transformation (I've started one called cleanForDiff.xsl in Test2) should be applied to all X*ML output files before they're compared with expected results.

===Separate out the Oxygen tests===

The current test process includes some tests which depend on an installation of Oxygen, and run transformations using Oxygen's Ant, as well as saxon8ee.jar. These tests properly belong in the oxygen-tei repo, where we should be doing a much better job of testing anyway (see https://github.com/TEIC/TEI/issues/464, which should be converted to an issue in the oxygen-tei repo at some point).

A Proposal for Rewriting the Stylesheets Tests

2017-01-08T23:46:42Z

Mholmes: /* Test file drawn from P5 Exemplars */

==The current state of the tests==
The Stylesheets Test folder contains the current testing mechanism, and I've been investigating it over the last few weeks. The set of tests clearly grew by accretion with not much in the way of an overall plan, and predictably it has become an over-complicated collection of inconsistencies, obsolescence and mystery. It basically works like this:

1. The tests are run using a Makefile (Test/Makefile).
2. The Makefile is divided into a range of targets intended to test different features, such as test-to-html, test-markdown, test-cocoa and so on.
3. Each target contains multiple tests which apply one of the conversions supported by the Stylesheets and then check the result in some way.

The conversions are applied in various different ways:

* By running the symlinked command (e.g. bin/teitohtml) which calls the bin/transformtei script
* By running ant and passing one of the ant build files such as /docx/build-to.xml
* By running Saxon and applying one of the XSLT stylesheets directly (such as fo/to.xsl)

There appears to be no rationale governing which method is used to invoke a process.

==Problems with the current tests==
As I've worked on figuring out how the tests work, I've raised a number of bugs and fixed a few (see for instance GitHub issues 213 through 218). The problems fall into these main categories:

* There are dozens of input test files, and very few of them have meaningful names or helpful comments to explain what they're testing (for instance, test14.xml, test2.xml, test25.xml, test26.xml, test3.xml, test4.xml, test6.xml, test31.xml, test36.xml, test23.xml, test17.xml, test20.xml, test27.xml, test22.xml, test28.xml, test8.xml, test32.xml, test24.xml, test12.xml, test5.xml).
* Many test files are already invalid or obsolete due to deprecation of various features in recent versions of P5.
* Different ways of calling the tests (see above) mean that we're not always testing the full process pathway, and it's not clear why.
* Different test results are handled in very different ways for no apparent reason. For instance, some test results are formatted with xmllint before being compared with their matching expected results file, and some are not.
* The approach of processing many small files rather than a small number of large ones means that the test process takes longer, because of the repeated necessity to spin up Java VMs.
* The tests can only be run on *NIX systems which have make installed.

==Principles for rewriting the tests==
I believe it's time to start again with a new approach to testing, and I think we should adopt these principles in doing so:

===Use Ant rather than make===
Ant is already used in the test process anyway. If we abandon make and write the whole thing in Ant, there will be a number of advantages:

* One less technology to install and understand.
* Greater efficiency and therefore speed when other Java-based tasks are invoked.
* Better documentation and commenting.

I've started rewriting a number of the existing tests in the Stylesheets/Test2 directory, using Ant instead of make, and I've encountered no major obstacles so far.

===Cross-platform support===
This is difficult but feasible; I have a pilot project In the Stylesheets/Test2 directory I've written an ant buildfile called convert.xml which calls a transformation in two different ways based on the platform it's running on. We know that most people working on the Stylesheets currently are using *NIX platforms, but when you need to be on *NIX before you can even call the command line transformations at all, this is hardly a surprise. Many people use Windows, and we ought to do our best not only to support them but to enable them to contribute to the codebase.

===Smaller numbers of test files===
Rather than ten small files testing various different things, a better approach would be a single test file which attempts to include a wide range of different features, each documented in-place; adding a new test would then be simpler since it would require only adding a new section to a file rather than constructing a whole new file, which might have features irrelevant to the test which later go stale or obsolete.

===Include test files drawn from P5 Exemplars===
Despite the fact that they're maintained in a separate repository, the Stylesheets are tightly coupled with P5, and development on the two codebases must proceed largely in lock-step fashion. The Stylesheets tests which relate to ODD processing (creating schemas and documentation in various formats) should be tested using ODD files from the P5 Exemplars folder. This will help to ensure that:

1. We're testing all the features people are likely to use most commonly.
2. All P5 changes (assuming we use tei_all as a test file) will be tested as soon as they're introduced, and it should not be so easy for us to neglect to implement Stylesheets support for a P5 change.

===Validate all test input files before using them===

We need to know when a test file goes stale, especially a TEI file.

===Name test files meaningfully===

"cocoatest.txt" is more helpful than "test40.txt", obviously. Where larger files are created to test many things, this is harder, of course.

===Output results into a separate folder===

Right now, all the results go into the same folder as the test files, which means the directory is even more cluttered with "test25.xyz" files, making it harder to see what's happening. Results should go into a "results" folder, alongside the "expected-results" folder.

===Keep linked resources in separate folders===

The Test folder currently contains linked images, various schemas used for validation, and other bits and pieces such as PERL scripts used for cleanup. These should be organized properly.

===Process all X*ML output in the same way after creation===

As mentioned above, some XML files are formatted with xmllint before checking, and some are not; some have comments removed and some don't. A single XSLT transformation (I've started one called cleanForDiff.xsl in Test2) should be applied to all X*ML output files before they're compared with expected results.

A Proposal for Rewriting the Stylesheets Tests

2017-01-08T23:34:41Z

Mholmes: /* Cross-platform support */

A Proposal for Rewriting the Stylesheets Tests

2017-01-08T23:23:53Z

Mholmes: /* Problems with the current tests */

A Proposal for Rewriting the Stylesheets Tests

2017-01-08T23:22:22Z

Mholmes: /* Principles for rewriting the tests */

A Proposal for Rewriting the Stylesheets Tests

2017-01-08T23:15:51Z

Mholmes: /* The current state of the tests */