Lobbying for features

We are a large and active community, a well-mixed blend of mostly Digital Humanities and (generally speaking) Computer Science professionals, with ties both to academia and software industry, and as such, we are potentially quite a force, a pressure group, if only our needs as TEI-ers can be identified and our efforts towards reaching them coordinated. And the power is not only in the numbers, but also, fundamentally, in the achievements of the TEI: from the widespread use of the standard in digitalizing manuscripts and literary works, through corpus/dictionary encoding, to the TEI's strong connection to Web/XML standards, such as XLink or ISO 24610-1:2006 (feature structure encoding).

Complain all you want, but then ACT

From time to time, we complain on the TEI-L about the insufficient software support for various devices/techniques suggested in the Guidelines (mostly, this pertains to XML manipulation or visualization). Complaining in a closed group is rarely successful, however, the outcome being usually of two kinds: either disappointment followed by surrender, or investment of one's own time and effort, possibly duplicating the work of others. The third way, however, is for the closed group to channel its pressure towards external, specialized groups, e.g. software developers or standards bodies.

Why should these external groups care about the needs of TEI-ers? As usual, mostly because they can benefit from satisfying these needs, but they need to learn about this opportunity first. Here's a list of the relevant factors:

• if there is an articulated need for feature X in our community, many of us are going to use it
• because we are a strong community, the number of users of feature X is potentially large
• software developers and standards need users to survive: users test the development versions, create feedback, suggest ideas, and spread the word
• if, therefore, the particular software package supports the features we need, a lot of mouths are going to praise it and make others use it
• and because of the above-mentioned past achievements of the TEI, it is quite possible that the currently undersupported features (think e.g. of advanced XPointer), will turn out to be indispensable in the near future, if their usefulness can be demonstrated not only in theory, but also in actual use; the one who supports it early reaps most of the harvest (think Saxon, always on the front line with respect to XSLT standards support).

Act how?

How can our community exert influence on outside groups? Several ways come to mind, and the list below won't certainly be exhaustive, feel welcome to add to it.

Firstly, the perhaps most trivial way to get things done is to talk about them to people who can help. Whether we are teachers/researchers, librarians or software architects -- all of us have contacts or ways of getting in contact with people in charge of software packages or participants in standards organizations (and the TEI has had a few, still does). Let's not underestimate the power of a friendly conversation over lunch or during a conference coffee break ;-)

Secondly, sometimes we happen to have a brilliant student who nevertheless needs a suggestion for his or her diploma work. If they happen to be CS students, why not have them work on an extra module to (or a modification of) an open-source package that will support the functionality the TEI needs. A job like that may actually be much more satisfying for the student than building some obscure program/package merely in order to demonstrate that they can.

Finally, and this is what I want to focus on, there is something all of us can do as users of various kinds of software: exert pressure on the developers by indicating to them how desperately a given feature or bug-fix is needed. Let me call this lobbying for features.

Lobbying for features

Most projects have components designed for gathering user feedback. At this point, many readers are going to think of tedious writing of bug reports or feature requests, and they will partially be right -- after all, if you want something done, then at least make sure to signal it to the developers in such a way as to make the report easy to find and the work on it easy to track and test. However, there is also a related activity that costs much less effort, and is more in line with the tile of the section: you can vote.

The immediate impulse for writing up this article came from two sources: Michael Sperberg-McQueen's blog entry and the recent thread on TEI-L on "certainty, @match, and XPaths redux", with a subtopic on XPointer that reminded me of my own earlier lobbying efforts in this area.

Firefox (rendering, client-side on-the-fly transformation)

In his recent blog entry, "Firefox and namespace nodes: an open plea", Michael Sperberg-McQueen asks readers for help in feature-lobbying, by voting on bug94270 in mozilla.org's Bugzilla -- a well-known bug-tracking system. I now recall how, after Firefox 3 got released, I would bring an installation package of Firefox 2 to several conferences and either make the organizers install it or complain if they refused, because XPointer-based XML links in the demo version of a certain TEI-encoded dictionary I presented worked well in FF 2.x only. Now I think I should have acted back then, at least by reporting this in the Bugzilla or by finding a suitable report and voting on it. Silly me.

libxml2 (validation, XInclude, XPointer, transformation)

Libxml2 is the only XML toolkit that has rudimentary support for XPointer's xpointer() schema. In fact, the support is rather bad... but at least it is there and can be fixed and extended if the need for it can be demonstrated. (The trick with diploma works may also work here -- there is a separate XPointer module that the student can concentrate on.)

In fact, the TEI doesn't need the xpointer() scheme to be supported in xmllint (libxml2's parser) -- it just needs the general XPointer Framework to work (see the Addendum below for explanation).

I have reported or commented on several XPointer/XInclude-related bugs in libxml2, but without votes, testing and comments from others, fixing them may take a while (insert link to Daniel Veillard's mail here). I have even to talk a colleague into providing a few patches, with some success (link to Jakub Wilk's report of the bug persisting).

This section is messy and will remain so for at least several hours, because I need some sleep :-)

Have to extract some bits from this fragment still. I searched for freely-available free-standing XPointer-aware tools and found out that only libxml2 (with xmllint) comes reasonably close, but its XPointer support is incomplete and buggy. I reported some of that on TEI-L some time ago. Since then, libxml2 has seen two bugfix releases, but the crucial functionality is still missing.

We have a colleague, Jakub Wilk, who did some bug-hunting and submitted a few patches to libxml2 in his free time, but I guess both his free time and patience have run out now (which I find perfectly understandable).

In case you were interested in pursuing this further, let me give you some links as starters:

"internal error, xpointer.c:2409" when using string-range() https://bugzilla.gnome.org/show_bug.cgi?id=562541

Xpointer range-to function loses the end-point children https://bugzilla.gnome.org/show_bug.cgi?id=306081

buggy range() XPointer function https://bugzilla.gnome.org/show_bug.cgi?id=584219

buggy string-range() XPointer function https://bugzilla.gnome.org/show_bug.cgi?id=583442

I tried to use the xpointer-schema string-range() function instead of the TEI-defined string-range schema, but that was impossible for a while, until this bug got fixed:

unrecognized XPointer schemes are not skipped silently https://bugzilla.gnome.org/show_bug.cgi?id=563562

(so there is a light...)

But that would require a few complications in the markup, to provide a cascade of XPointer schemas, with the W3C schema as fallback until the TEI-defined schemas are supported by some tool.

List of projects and places where lobbying is needed

This is to be the climax of this article: I would like to put here a list of places where members of the TEI community can go and e.g. cast their votes or help in bug triaging, etc. More tomorrow.

Temporary Addendum: XPointer issues

This section belongs elsewhere in this wiki, but because it addresses an issue recently brought up on TEI-L, and provides context for my suggestion above, let it stay here for a while.

Let me briefly explain what the place of XPointer is with respect to the TEI machinery, and address two issues that may be easy to misunderstand:

1. XPointer IS supported by tools, just not in its entirety
2. TEI's string-range() is not W3C's string-range()

The XPointer Framework is a collection of schemes (note: schemes not schemas) that specify the method for addressing into the XML tree or help in this task. One end of its functionality overlaps with XPath, but the other allows one to address points and ranges inside elements, and this is often precious to us, especially in some stand-off markup systems or for working with ontologies (RDF may use it).

Some of the schemes, e.g. the simplest element() scheme, which uses simple tree-traversal syntax, are supported by any decent XML tool nowadays. In fact, any tool that claims to support XInclude, must support the XPointer element() scheme.

Apart from element(), there is also the xmlns() scheme that does namespace binding, and finally the xpointer() scheme that does an incredible lot of useful things in a very clever way, except it's not supported in full anywhere, and that should change (see above).

The three schemes mentioned above have been defined or, in the case of xpointer(), drafted, by the W3C. The XPointer Framework, however, allows other parties to define their own schemes and get them registered in a special corner of the W3C called the XPointer Registry. And this is the point where we, as the TEI community, may want to focus some of our attention. Thanks to Syd Bauman, a number of schemes (described in the Linking and Alignment chapter of the Guidelines) have been registered with the W3C. One of them, xpath1(), is actually shared with other Internet communities, and implemented in Firefox (as far as I know). Another is the string-range() scheme, useful in stand-off applications that address into the content of elements.

The structure of the XPointer Framework is illustrated below:

                     XPointer Framework
                      .               .
                     /                 \
                    /                   \
                   .                     .
            scheme syntax           scheme repository
                                     .            .
                                    /              \
                                   /                \
                                  .                  .
                        W3C schemes            external-party (e.g. TEI) schemes
                      .   .       .                     .      .
                     /    |        \                   /        \
                    /     |         \                 /          \
                   .      .          .               .            .
              element() xmlns() xpointer()        range() string-range() ...
                                .       .
                               /         \
                              /           \
                             .             .
                     XPath functions    XPointer functions
                                           .   .       .
                                          /    |        \
                                         /     |         \
                                        .      .          .
                                 range-to()  end-point() string-range() ...

Notice that string-range() is mentioned twice in the diagram above. This is because of an unfortunate homonymy that has generated some confusion in the past but, hopefully, will not do so any longer.

The XPointer xpointer() scheme uses XPath functions and adds to them several others, which, by entering the inter-character space, are able to cleverly address what no XPath has been able to address before. Among these functions is one called string-range().

Thus the difference between the W3C's location-set string-range(location-set, string, number?, number?) function and the TEI's string string-range(pointer, offset, length?) scheme is not merely a difference in the definition, but also a difference in the status. There is at least one important consequence of this, worth bearing in mind: the status and implementation of TEI's schemes does not directly depend on the status and implementation of the xpointer() scheme. Of course, one can hope that the former may piggyback on the latter, from the perspective of software developers. And that takes us back to the previous section.

---

If for some reason you do not want to use the discussion page, I welcome comments by e-mail. Piotr 23:35, 30 October 2009 (EDT)