Lobbying for features

We are a large and active community, a well-mixed blend of mostly Digital Humanities and (generally speaking) Computer Science speci-alists (needed to spoil the preceding word b/c of the spam filter...), with ties both to academia and software industry, and as such, we are potentially quite a force, a pressure group, if only our needs as TEI-ers can be identified and our efforts towards reaching them coordinated. And the power is not only in the numbers, but also, fundamentally, in the achievements of the TEI: from the widespread use of the standard in digitalizing manuscripts and literary works, through corpus/dictionary encoding, to the TEI's strong connection to Web/XML standards, such as XLink or ISO 24610-1:2006 (feature structure encoding).

Complain all you want, but then ACT

From time to time, we complain on the TEI-L about the insufficient software support for various devices suggested in the Guidelines (which mostly means XML manipulation or visualization). Complaining in a closed group is rarely successful, however, the outcome being usually of two kinds: either disappointment followed by surrender, or investment of one's own time and effort, possibly duplicating the work of others. The third way, however, is for the closed group to channel its pressure towards external, specialized groups, e.g. software developers or standards bodies.

Why should these external groups care about the needs of TEI-ers? As usual, mostly because they can benefit from satisfying these needs, but they need to learn about this opportunity first. Here's a list of the relevant factors:

• if there is an articulated need for feature X in our community, many of us are going to use it
• because we are a strong community, the number of users of feature X is potentially large
• software developers and standards need users to survive: users test the development versions, create feedback, suggest ideas, and spread the word
• if, therefore, the particular software package supports the features we need, a lot of mouths are going to praise it and make others use it
• and because of the above-mentioned past achievements of the TEI, it is quite possible that the currently undersupported features (think e.g. of advanced XPointer), will turn out to be indispensable in the near future, if their usefulness can be demonstrated not only in theory, but also in actual use.

Act how?

How can our community exert influence on outside groups? Several ways come to mind, and the list below won't certainly be exhaustive, feel welcome to add to it.

Firstly, the perhaps most trivial way to get things done is to talk about them to people who can help. Whether we are academics, librarians or software architects -- all of us have contacts or ways of getting in contact with people in charge of software packages or participants in standards organizations (and the TEI has had a few, still does). Let's not underestimate the power of a friendly conversation over lunch or during a conference coffee break ;-)

Secondly, sometimes we happen to have a brilliant student who nevertheless needs a suggestion for his or her diploma work. If they happen to be CS students, why not have them work on an extra module to (or a modification of) an open-source package that will support the functionality the TEI needs. A job like that may actually be much more satisfying for the student than building some obscure program/package merely in order to demonstrate that they can.

Finally, and this is where I want to stop for today, there is something all of us can do as users of various kinds of software: exert pressure on the developers by indicating to them how desperately a given feature or bug-fix is needed. Let's call this feature-lobbying, because I need a title for the next subsection.

Feature-lobbying

Most projects have components designed for gathering user feedback. At this point, many readers are going to think of tedious writing of bug reports or feature requests, and they will partially be right -- after all, if you want something done, then at least make sure to signal it to the developers in such a way as to make the report easy to find and the work on it easy to track and test. However, there is also a related activity that costs much less effort, and is more in line with the tile of the section: you can vote.

The immediate impulse for writing up this article came from two sources: Michael Sperberg-McQueen's blog entry and the recent thread on TEI-L on "certainty, @match, and XPaths redux", with a subtopic on XPointer that reminded me of my own earlier feature-lobbying efforts in this area.

Firefox (rendering)

In his recent blog entry, "Firefox and namespace nodes: an open plea", Michael Sperberg-McQueen asks readers for help in feature-lobbying, by voting on bug94270 in mozilla.org's Bugzilla -- a well-known bug-tracking system. I now recall how, after Firefox 3 got released, I would bring an installation package of Firefox 2 to several conferences and either make the organizers install it or complain if they refused, because the XML links in the demo version of a certain TEI-encoded dictionary I worked well in FF 2.x only. Now I think I should have acted back then, at least by reporting this or finding a suitable report and voting on it. Silly me.

libxml2

This is the only XML suite that has reasonably good support for XPointer's xpointer() schema. In fact, the support is rather bad... but at least it is there and can be fixed and extended if the need for it can be demonstrated.

I have to be running, so for now, let me merely paste a fragment of my recent e-mail, and finish the article later:

Hello Laurent,

Laurent Romary writes:
> > Thanks a lot. Do you have an available implementation for string-range
> > somewhere?

Nope. I searched for freely-available free-standing XPointer-aware tools
and found out that only libxml2 (with xmllint) comes reasonably close,
but its XPointer support is incomplete and buggy. I reported some of
that on TEI-L some time ago. Since then, libxml2 has seen two bugfix
releases, but the crucial functionality is still missing.

We have a colleague, Jakub Wilk, who did some bug-hunting and submitted
a few patches to libxml2 in his free time, but I guess both his free
time and patience have run out now (which I find perfectly understandable).

In case you were interested in pursuing this further, let me give you
some links as starters:

"internal error, xpointer.c:2409" when using string-range()
https://bugzilla.gnome.org/show_bug.cgi?id=562541

Xpointer range-to function loses the end-point children
https://bugzilla.gnome.org/show_bug.cgi?id=306081

buggy range() XPointer function
https://bugzilla.gnome.org/show_bug.cgi?id=584219

buggy string-range() XPointer function
https://bugzilla.gnome.org/show_bug.cgi?id=583442


I tried to use the xpointer-schema string-range() function instead of
the TEI-defined string-range schema, but that was impossible for a
while, until this bug got fixed:

unrecognized XPointer schemes are not skipped silently
https://bugzilla.gnome.org/show_bug.cgi?id=563562

(so there is a light...)

But that would require a few complications in the markup, to provide a
cascade of XPointer schemas, with the W3C schema as fallback until the
TEI-defined schemas are supported by some tool.

Obviously, xmllint would not reach inside @corresp attributes in Adam's
example below -- some preprocessing is needed for them, and my guess is
that when the NCP actually needs string-range() or its kin to function,
the way out will be via dedicated Java-based XML-processing, similar to
what the ANC does when merging its annotations.

And having written all of the above, I realised that I could give you a
much shorter answer. Your question about an available implementation for
string-range may actually be restated more generally: "do you know of
any tool that supports any of the TEI-defined XPointer schemas?". To
that, my answer is: sadly, I know of NO such tool. xmllint is (say) half
way towards supporting the W3C-defined xpointer() scheme. Implementation
of the TEI specs would have to be a next step.

Best,

  Piotr

> > 
> > Le 21 sept. 09 à 19:52, Adam Przepiorkowski a écrit :
> > 
>> >> Dear Laurent,
>> >>
>>> >>> I am trying to get an overview of what syntax projects are currently
>>> >>> using to refer to point in a textual or audio source. Could those of
>>> >>> you for which it is relevant give me an idea of the syntax you use
>>> >>> (e.g. string-range()), which normative reference you had in mind when
>>> >>> you did so, and maybe a tiny example of implementation (one two line
>>> >>> of XML context).
>> >>
>> >> In the National Corpus of Polish, we are planning to use string-range
>> >> in the following way:
>> >>
>> >>    <p corresp="text.xml#txt_1-div" xml:id="segm_1-p">
>> >>     <s xml:id="segm_1.20-s">
>> >>      <!-- Ważnym -->
>> >>      <seg corresp="text_structure.xml#string-range(txt_1.1-ab,0,6)"
>> >> xml:id="segm_1.1-seg"/>
>> >>      <!-- momentem -->
>> >>      <seg corresp="text_structure.xml#string-range(txt_1.1-ab,7,8)"
>> >> xml:id="segm_1.2-seg"/>
>> >>      <!-- w -->
>> >>      <seg corresp="text_structure.xml#string-range(txt_1.1-ab,16,1)"
>> >> xml:id="segm_1.3-seg"/>
>> >>      ...
>> >>     </s>
>> >>     ...
>> >>    </p>
>> >>
>> >> We are trying to follow TEI (hence the @corresp attribute).
>> >>
>> >> Best,
>> >>
>> >> Adam