CorpusReader

From TEIWiki
Revision as of 19:56, 26 August 2006 by Sloiseau (talk | contribs)
Jump to navigation Jump to search

Summary

CorpusReader (CR) is a tool for extracting subcorpora and quantitative information from arbitrarly large corpora in the TEI vocabulary. It intends to provide ways for processing corpora containing milestoned annotation. It provides mechanism for merging several XML documents together.

Background

CorpusReader was developped for quantitative corpus linguistics. Its original goal was to provide a way for extracting quantitative information (such as cooccurrency matrix) using all information expressed in the XML infoset.

As a quantitative corpus linguistic tool, CorpusReader do not have any linguistic nor statistical skill... but provides help for:

  • importing outputs of existing linguistic tools into a corpus (tagger, parser, etc.),
  • exporting quantitative data in statistical tool formats (toward Matlab, R and DTM in particulary).

The rationale is that existing linguistic and statistical tools should be reused and made easy to use in the context of a TEI corpus. Thus CorpusReader is a bridge between linguistic and statistical tools for creating and exploring empirically complex data.

Technical features

CR is written in java, and works with Java 1.4 and further. It relies on numerous external high quality open source libraries.

It is run only at the command line.

It is released under the BSD licence.

There is a web site : http://panini.u-paris10.fr/~sloiseau/CR/

Properties

Functions are filters

CR relies heavily on the SAX API. The "functions" of the program are implemented as SAX filters. The program is mainly a collection of SAX filters and a mechanism for pluging the filters into a pipeline.

Each filter is specialized into a precise task and takes few argument. This allows modularity and reusability. While each filter performs a simple task, the pipeline of filters may achieve complexe tasks.

The names of the filters (and their arguments for some of them), defining the pipeline, are given to the program through a file (an XML document of course), called the "query document". The URL of this document is given as an argument to the program, at the command line.

A query document looks like: 

<query>
  <header>
    <name></name>
    <date></date>
    <desc></desc>
  </header>
 
  <corpus inURI="corpus.xml"
          outURI="sample-manuel-1.out"
          />
  <filterList>
    <filter name="myFilterName" javaClass="java.class.qualified.Name">
      <args>
        <!--  Argument subtree, passed to the filter if any, after
              validation if a schema is know for this filter.
        -->
      </args>
    </filter>
    <!--  etc.: as many filter as needed  -->
  </filterList>
</query>

In some cases, filters can communicate directly with each other (in addition to communicating through the stream of SAX events).

Dealing with intersecting hierarchies

[TODO]

Using high level languages on large documents (XPath, XSLT, XQuery)

High level languages (XSLT and XQuery) are made available: the stream of XML events is bufferised, transformed, and throw back to the next filter in the pipeline as a stream of XML events.

When the corpus does not fit in memory, a mechanism allows to address the subtrees to be bufferised and transformed successively, one by one, as separate document. It is the "split" element in the query document. Each "filter" into a "split" element see the corpus as several documents rooted at the split/@localName elements. 

<query>

  <header>
    <name></name>
    <date></date>
    <desc></desc>
  </header>

  <corpus inURI="path/to/corpus" outURI="path/to/output"></corpus>

  <filterList>
    <split localName="TEI">
      <filterList>
        <filter name="transform_my_div" javaClass="tei.cr.filters.XSLT">
          <args>
            <stylesheet URI="path/to/stylesheet"></stylesheet>
          </args>
        </filter>
      </filterList>
    </split>
  </filterList>
</query>


There is also a way for addressing elements in the stream through XPath expression evaluated against each element, one at one, as if they were a stand-alone document. For instance, the query document above could be written as: 

<query>

  <header>
    <name></name>
    <date></date>
    <desc></desc>
  </header>

  <corpus inURI="path/to/corpus" outURI="path/to/output"></corpus>

  <filterList>
    <split elxpath="*[namespace-uri()='http://www.tei-c.org/ns/1.0' 
                      and local-name()='div']">
      <filterList>
        <filter name="transform_my_div" javaClass="tei.cr.filters.XSLT">
          <args>
            <stylesheet URI="path/to/stylesheet"></stylesheet>
          </args>
        </filter>
      </filterList>
    </split>
  </filterList>
</query>

Merging several documents

CR contains a mechanism for merging an external document into an already annotated corpus without breaking well-formedness. It is useful for reuse the output of existing linguistic annotation tool.

Using a low level API

I tried to make the program stand between a "tool" and an "API". (a framework for facilitating the use of a low level API). Thus, there may be different ways of using it:

  • provide directly usable functions for creating kwic, extracting subcorpus, computing cooccurrency matrix, merging markup, etc.
  • may be used as a way of applying XSLT / XQuery on big corpora
  • may be used for plugging any SAX filters; and reduce the complexity of SAX by allowing to write only a SAX handler code, while the program manage the parser, the pipeline, and the serialisation of the output of the pipeline back to disk. Any class implementing the XMLFilter interface may be plugged in the filter by providing the qualified name of the java class in the query document (this class should be found in the CLASSPATH variable).
  • may be used for prototyping java code by embedding in the pipeline java code defining a SAX filter, see http://panini.u-paris10.fr/~sloiseau/CR/filtres/Script.html or http://panini.u-paris10.fr/~sloiseau/CR/exemples.html#embeddedJava

The goal was to make easy the use of a low-level API. "CorpusReader" is nammed after "XMLReader", the name of the class parsing a document in the java SAX API. It is intended to be a "layer" onto SAX for the TEI vocabulary.

Fitted for documents in the TEI scheme

The program tries to rely on the TEI scheme: some structural properties of TEI documents are sometime needed, but I try not to make it relying on a specific TEI customisation. (I would like to develop a mechanism for using the "TEI customization" document produced by Roma for overriding the default vocabulary).

External Links

Retrieved from "https://wiki.tei-c.org/index.php?title=CorpusReader&oldid=2735"