Checking Pointers Using your TEI ODD Customization

Syd Bauman Customizing TEI to Check Pointers Northeastern University Women Writers Project 2022

Acknowledgments

The author is indebted to Sarah Connell for editing the prose, and to the following people for testing the Schematron. Elisa Beshero-Bondar Lou Burnard Ash Clark Martin Holmes David Maus Helena Bermúdez Sabel Martina Scholger Raffaele Viglianti Without their help this document might not exist, and if it did, it would be harder to read and would have a lot more mistakes.

Introduction

As with any other part of document production, when using URIs (web pointers) it is advantageous to check that they work early in the production process. The aspects of a pointer we would want to test, in increasing order of difficulty, are that the URI: points to something (i.e., has proper syntax) points to something that can be retrieved (i.e., refers to an object that exists and to which the user has access) points to the right kind of thing (e.g., the ref of persName should point to a person, not a fileDesc nor an SVG chart of COVID-19 cases) points to the right thing (e.g., the #MLK of <persName ref="#MLK">Martin Luther King, Jr.</persName> should point to an entry for Martin Luther King, Jr., not an entry for Lois Lane).

Most of the main closed schema languages, including PureODD and RELAX NG (but excluding DTDs), have the capability to test that an information item (in our case, and thus hereafter, an attribute value) meets the syntactic constraints of a URI. That is, we can test (1) above just by using our schemas. But none of those languages (including PureODD without use of constraintSpec) have the capability to test (2)–(4).

The open schema language used by TEI (ISO Schematron) does allow for (2) and (3). However, a) it is difficult to associate the needed tests with all teidata.pointer attributes, and b) it requires special-case coding for (3), depending on what is considered right in the current circumstances. In any case, (4) is much more difficult, and usually requires a human, if not a human with domain-specific knowledge.

Thus the TEI Technical Council does not plan to include any general tests for (2)–(4) directly in TEI P5, at least not in the near future. However, recognizing that at least (2) & (3) represent an important subset of validation tests that projects want to perform, we exemplify herein some mechanisms for doing so. The intent is that projects can copy-and-paste ODD fragments from this file into their own, and modify as desired to suit their local needs.

Note, however, that this document only discusses testing of URIs that are intended to return an XML document or fragment thereof. It is possible to use URIs to point to and retrieve other sorts of information objects including JSON, images, HTML (other than XHTML), audio files, word processor files, etc., but these cases are not considered herein.

Also note that the examples in this document often use an attribute as the context node of an ISO Schematron rule. Some Schematron processors fail to process this correctly. (The oXygen XML Editor handles these perfectly well, as does David Maus’s SchXslt.)

There are over 100 attributes in TEI defined as teidata.pointer.And another 4 or so defined as teidata.namespace, whose values have the same syntax as a URI, but are not necessarily intended to point at anything. Roughly 60 of those are restricted to having only 1 URI as a value, but the other nearly 50 permit multiple URIs in the value. Processing multiple pointers is much more difficult than handling a single URI, and thus the singleton cases are considered separately from the cases in which multiple URI values of a single pointer attribute need to be tested.

Cases where a single item is being pointed to directly are considered first, then cases where multiple items may be pointed at directly by multiple URIs expressed as the value of a single attribute, and then then cases where an item is being referred to indirectly either via an intermediate link or alt element, or via a prefix defined in the prefixDef. Lastly, a method of ensuring that validation is being performed after XInclude processing is demonstrated.

Direct reference by single value

Projects that intend to always use one and only one URI as the value of an attribute which by default in TEI may take multiple URIs will probably find it best to constrain said attribute to only 1 URI. For example, if the (fictional) project The Papers of Dr. Virgil Swann were to provide correspondence links between Dr. Swann’s English translation of and his transcription of an intercepted Kryptonian message, each s element with an xml:lang of en would bear a corresp attribute that pointed to the corresponding s element with an xml:lang of x-kr. The TEI corresp attribute allows one or more URIs. But for this project, the corresp of s should be limited to only 1 URI. This can be accomplished as shown in .

shorthand pointer fragment identifier

The fragment identifier portion of a URI is that which follows the first # (read left-to-right). The portion of the URI before the first # locates the document of interest; the portion after the first # locates the element of interest within said document. The portion after the first # can take many forms, only one of which is considered in this document: the shorthand pointer fragment identifier. It is probably by far the most common form of fragment identifier; it refers to an element in the document in question by referring to its ID. (In TEI the ID is always indicated by the xml:id attribute.) This typically looks like either https://www.w3.org/TR/xptr-framework/#shorthand or #DeRoseetal1990 If there is no document mentioned to the left of the first #, then the document being referred to is the base document, which is typically the current document in which the URI appears (but which can be modified by the xml:base attribute).

First, just to demonstrate how this works in general, example shows how to test that the ref attribute of g has the correct syntax to point to something in the same file. However, as this check can be expressed in PureODD without resorting to Schematron, a PureODD-only mechanism is exemplified in ; it has the sizable advantage that the testing is performed by the closed schema language. Note, however, that although this use of the restriction attribute is supported by the TEI’s current ODD processor, other uses may not be; in particular, restriction is currently only usable with the name attribute, not the key attribute of dataRef.

To check that a shorthand pointer fragment identifier URI points to something that can be retrieved (i.e., to test a pointer that looks like #duck for (2), above) you can take advantage of the XPath id() function. This technique is used in to ensure that the ref of a g actually points to something in the current document. A call to id('tho') returns the element from the same document, if there is one, that bears an xml:id of tho. There is supposed to be at most one such element; if there were two or more, only the first is returned. Thus id('tho') may be thought of as //*[@xml:id eq 'tho'][1]Except there are differences in whitespace normalization..

To check that a shorthand pointer fragment identifier URI specifically points to a particular element type (in this case char or glyph), you can simply append a node test, as seen in . Note that references to TEI elements in the XPath expressions in the Schematron inside constraint need to be explicitly bound to the TEI namespace. Although an ODD author may define any namespace prefix for the purpose (using the Schematron ns element), TEI ODD software will automatically insert a definition that binds the prefix tei: to the TEI namespace.

local filesystem

A URI may refer to a file on the local filesystem using either an absolute-path reference or a relative-path reference.A URI may also refer to a file on the local filesystem using the file: URI scheme. This document does not consider these URIs, except in a demonstration that testing that a URI uses the file: scheme is possible (in ) and the following comment on the general syntax of a file: scheme URI — Said format is file://host/path, where the //host portion is optional (defaulting to localhost), or may be expressed as just // (again defaulting to localhost). Thus file:/path and file:///path are both perfectly acceptable ways to refer to the file found at path on the local filesystem. An absolute-path reference starts with a slash; a relative-path reference does not. A relative-path reference may start with a dot segment (./), unless the first segment of its path contains a colon, in which case it must start with a dot segment.

The URI that refers to a file on the local filesystem may have a # followed by a fragment identifier. The scenarios covered here are, in order, testing that a URI: points to a local file, points to a local file with a particular file extension, points to a local file with a particular root element, and last that a URI points to a particular element type in a local file.

entire local file

A moduleRef element is typically used to refer to one of the TEI modules (for example, core, gaiji, or namesdates) from a customization ODD file using its key attribute. But moduleRef can also refer to a non-TEI module using its url attribute. The TEI schema ensures that the value of the url attribute is, in fact, a URI (that is, it performs test (1)). It would be quite reasonable for a project to want to check that the value of url was a URI that referred to an existing, readable, local XML file.

It is possible to write generic constraints for this purpose that would allow any valid URI that referred to a local file (whether an absolute path reference, a relative path reference, or a file URL scheme; see ) or that a value uses a private URI scheme prefix defined by a prefixDef (see ). However, in most cases projects would probably want to constrain the value to a particular method for referring to an existing, readable, local XML file (if not a particular file, for which see ). For example, demonstrates a method for requiring that the url attribute of moduleRef refers to a local file that is in the same directory as the instance ODD and whose filename ends in .rng (and thus is probably a RELAX NG file in the XML syntax).

There is often good reason to ensure that the file referred to (whether local or remote) is readable, well-formed XML. Luckily, XPath provides the doc-available() function for this very purpose. The constraints demonstrated in first ensure that the information item referred to by the url attribute is a file, not an element, and then require that the file be readable, well-formed XML. The example at duplicates these constraints, and also tests that the outermost element of the retrieved file is an rng:grammar element.

element of a local file

The ref attribute of persName should typically point to a person element, which will often be in a separate personography file. Presuming that file is in a known location in the local filesystem, can be used to test that the ref attribute refers to a person in that file.

remote pointers

Checking remote pointers is in principle very similar to checking local pointers, but with different tests on the syntax of the URI. (They are also, of course, a bit harder to test because you must have a working internet connection (or alternatively use an XML Catalog), and have to worry about firewalls, proxies, same-source problems, cached files, etc.) For example, ensures that any equiv element has a uri attribute, and further that said attribute refers to the (fictional) markup_taxonomy page of the WWP website, allowing a reference to that page on either the production or test version of the website, and allowing access with or without specifying a secure connection. Note that this example only checks the syntax of the uri attribute, and does not ensure that there actually is such a page or specific element on that page.

The example, on the other hand, ensures that the filter attribute of equiv points to an XSLT program. It does this by testing that either the namespace of the outermost element of the retrieved file is the XSLT namespace (because the outermost element of a normal XSLT program could be either xsl:stylesheet or xsl:transform), or that the outermost element has a xsl:version attribute (because a simplified stylesheet must have such an attribute). Thus, in order for a filter attribute to pass this test, it not only needs the right syntax, but it must also point to an XSLT program that exists and is accessible via the web.

Direct access by multiple values

If a pointer attribute may have multiple values, testing is mildly more difficult because the attribute needs to be parsed first. In the general case, delivering a precise error message is quite difficult, as the entire process needs to be handled in XPath because Schematron does not have an iteration construct. However, specific cases may be reasonably easy to handle. tests that a rendition attribute refers to at most 2 rendition elements in the same file. This is not particularly difficult because of the restriction that there are at most 2 pointers.

But in many, if not most, cases there is no such restriction. For example, there may be dozens of witnesses to a particular manuscript manifestation. tests that each pointer in the value of a wit refers to a witness element in the same file. In addition to tests similar to the previous example, this example reports to the user what each pointer in a failed value is, in fact, pointing to.

Possibly indirect reference

demonstrates a methodology for ensuring that the ref of persName points to a person. One recommended method for encoding an ambiguous reference to a person is to use the alt element — the encoding of the ambiguous reference itself points to an alt element which, in turn, points to each of the possible persons. (I.e., as if the value of evaluate were one. See for a sample encoding.) Similarly, a reference to multiple individuals could be encoded as multiple pointers on a single ref, or could be encoded as a single pointer on the ref that points to a link element which, in turn, points to each person referred to. One advantage to this latter method is that you can restrict the ref of persName to one and only one pointer, and use similar constraints for both ambiguous and multiple references. demonstrates such constraints.

A similar set of constraints can be expressed in a simpler, perhaps easier to follow, way by splitting them up as constraints on the ref of persName and the target of link and alt separately, as demonstrated in . Because the code for alt and link is somewhat complicated, this technique expresses that code only once as an abstract Schematron pattern, which is instantiated separately, once for alt and once for link.

Reference via prefixDef

The TEI provides a method of indirection for both shortening URLs and having a single place to change a set of URIs. This mechanism makes use of a local URL prefix and a definition of how that prefix is mapped to a full URI which is expressed in a prefixDef element. Creating pointers using this method is simpler, shorter, easier to read (and thus proofread), and reduces the chance for errors in the first place. Checking pointers that use this method, however, is significantly more difficult. As with several other types of pointer checking, it is the general case that is most difficult; particular cases may be reasonably easy.

The scenario considered here is the simple (and to my knowledge far most common) case in which the matchPattern follows the shorthand pointer syntax or a subset thereof, and the replacementPattern follows the syntax of a URL except it ends with #$1. I.e., the case in which the matched bit is the shorthand pointer, as seen in an example from the Guidelines. While example limits itself to prefixDefs of this sort, it does not limit itself to any particular prefix. The document is searched for possible prefix values.

XInclude, yet?

It is often the case that it is known in advance that certain tests will fail before XInclude processing, whereas we hope they would succeed after XInclude processing.In the OXygen XML editor, you can toggle whether XInclude processing is performed before validation using the Enable XInclude Processing checkbox in the Options > Preferences > XML > XML Parser pane. For example, imagine that at our The Papers of Dr. Virgil Swann project the extant documents to be encoded (which include letters, scientific notebooks, satellite schematics, a dictionary, and translations of intercepted radio transmissions) have been categorized using a project-specific taxonomy. This taxonomy is encoded as a taxonomy element. Since each TEI document refers to this taxonomy (from its /TEI/teiHeader/profileDesc/textClass/catRef/@target), the project has chosen to have a copy of the entire project taxonomy in each TEI document’s header (in its /TEI/teiHeader/encodingDesc/classDecl). In order to avoid multiple copies of the same information, the project has chosen to store the taxonomy in a separate document and use XInclude to insert it into each TEI header.Why they would choose to do this rather than use a URI that points to an external document is an interesting, if irrelevant, question. They may well have been influenced by the fact that every example of catRef in the Guidelines uses shorthand pointers.

Given this situation, imagine now that the project wishes to check that the target attributes actually point to one or more category elements. This is problematic, because the category elements are not actually in the file as it sits unprocessed. There are a variety of ways this could be handled, probably the easiest of which is simply to skip the test (and warn the user it is being skipped) if XInclude processing has not yet taken place.

There are two straightforward methods of asking the question has XInclude processing taken place yet?. The first relies on the fact that after XInclude processing, no elements from the XInclude namespace should remain: they should have become the file to be included or, in case of error, the contents of the xi:fallback. This method is exemplified in .

The second relies on the fact that before XInclude processing the file does not have a taxonomy, and after XInclude processing it does. This method is exemplified in .

Examples

PureODD to limit attribute to one URI

In this example the corresp of s is altered so that its value must be one and only one URI.

Ensure ref of g is a shorthand pointer Somewhat useless example that just tests that the syntax of the ref attribute of g is nothing but a shorthand pointer, e.g. #duck. The value of the ref= attribute of 'g' ("") is supposed to be a shorthand pointer; i.e., should look like '#duck'.

Snippet of ODD code that uses Schematron to enforce that the value of ref of g is a shorthand pointer. The regular expression \i\c* matches an XML Name, and thus #\i\c* matches a shorthand pointer to an XML element with an ID attribute.

Ensure ref of g is a shorthand pointer, PureODD

This attribute must be a single shorthand pointer to an element in the same XML file, i.e. must look like #duck.

defines the range of attribute values used to provide a single shorthand pointer URI

Formerly referred to as a barename identifier.

Two snippets of ODD code which together use a facet to enforce that the value of ref of g is a shorthand pointer.

While somewhat more complicated, this method has the advantage that it is both faster and generally easier on the encoder, in that the constraint will be expressed in RELAX NG (or the W3C Schema Language), and modern XML editors (like oXygen) can enforce such constraints immediately, as you type.

Check that ref of g points to something Example of how to test that the ref of g points to an element in the same document. The ref= of this 'g' element ("") does not point to an element in this document.

This example presumes that the ref of g is a shorthand pointer. Thus this constraint should be used in conjunction with something like or .

Check that ref of g points to a char or a glyph Example of how to test that the ref of g points to a local char or glyph. The ref= of 'g' is supposed to point to a 'char' or 'glyph'; this one ("") points to a ''.

This example presumes that the ref of g is a shorthand pointer that actually points to something in the current document. Thus this constraint should be used in conjunction with a constraint like (which itself should be used in conjunction with a constraint like or ).

There are several other ways to express the XPath in the test of the sch:assert, of course. E.g., many would consider id( $ref )[ self::tei:char | self::tei:glyph ] better because it is more compact. In either of these example XPaths the union operator (|, which may also be expressed union) could be replaced with or to indicate an OrExpr.

Note: use self::tei:char, not self::char — the use of the namespace prefix in the XPaths within the Schematron is currently required.

Require the url of moduleRef to refer to a particular file Our special module for Kryptonian constructs

ODD fragment which requires that the value of url of moduleRef be ./kryptonian.rng.

Check that url of moduleRef refers to an RNG file in the same directory

The url attribute must refer to a file in the same directory as the ODD customization file, and must have the file extension .rng.

Ensure file is readable XML Module is not readable, well-formed XML.

The url attribute must refer to a file, not an element.

The constraints placed on the url attribute of moduleRef by the restriction attribute ensure both that the value is a single reference (by disallowing whitespace), and also is not a reference to a particular element within a file (by disallowing number sign (#, U+0023)).

The constraintSpec then tests that said file is readable, well-formed XML. The use of resolve-uri() assures that if a relative URI is used, the document requested is the one at that URI relative to the input document, not the Schematron schema.

Ensure file is readable RELAX NG grammar Module is not a readable, well-formed RELAX NG grammar in the XML syntax.

The url attribute must refer to a RELAX NG grammar in the XML syntax.

See comments for .

The prefix rng: is automatically bound to the RELAX NG namespace by the TEI ODD processor.

Note that the test for doc-available() is not strictly necessary; the same pointers will succeed or fail with or without that test. But with the doc-available() test the user gets the custom error message when the file is not readable well-formed XML; without it, the user gets a generic error message, e.g. No such file or directory.

Require persName to refer to a person in the local personography persName should refer to a person; this one refers to .

The ref attribute must refer to a person element in the persons.xml file.

This constraint takes advantage of Schematron’s facility for storing values in variables.

Ensure uri of equiv is present and refers to an item on a particular page

The url attribute is required, and must refer to an ID on our markup_taxonomy page.

Note that this constraint ensures that the uri attribute has the correct syntax, but does not actually try to retrieve the document it refers to.

The our markup_taxonomy web page is accessible at eight different possible URLs; the restriction allows only these eight URLs.

Require a filter on equiv that points to an XSLT program The filter must be an XSLT program!

Note that this constraint actually tries to retrieve the document filter refers to. This results in more thorough testing than , but requires a working internet connection or re-direction via an XML Catalog.

The use of resolve-uri() assures that if a relative URI is used, the document requested is the one at that URI relative to the input document, not the Schematron schema.

Note that the TEI processors do not automatically bind the xsl: prefix to the XSLT namespace. It is common to put all of your sch:ns elements in a constraintSpec that is a direct child of the schemaSpec of your customization ODD.

rendition points to 1 or 2 renditions Ensure that each pointer in a rendition points to a rendition. First rendition reference does not start with a number sign. Second rendition reference does not start with a number sign. The first rendition reference () does not point to a local 'rendition' element. The second rendition reference () does not point to a local 'rendition' element.

Here at the The Papers of Dr. Virgil Swann project the rendition attribute is limited to 1 or 2 values. If a more complex description of rendition is required, the individual CSS declarations (property:value pairs) should be combined into a new rendition, and the rendition should point at that.

This constraint on rendition is global, effecting all such attributes, whether defined is this class or not. However, since we do not declare any other rendition attributes, this is not a concern.

Note that the comparison $VAR eq '' could have been used instead of the function empty( $VAR ).

Check that each pointer in wit points to witness One (or more) of the pointers in this @wit attribute does (do) not start with a '#'. One (or more) of the pointers of this @wit does (do) not point to a 'witness' element. The pointers of this @wit point to the following items, in the order specified: .

Note that the wit is constrained here, not in the definition of att.witnessed, because the only other element that is a member of att.witnessed is lem, and at our project lem never uses the wit attribute.

Required ref of persName eventually refers to person A transcribed personal name (as opposed to a personal name in the metadata) must have a @ref At least one of the pointers in "" does not end up pointing to a <person>, even via a <link type="person"> or an <alt type="person">.

This set of constraints requires that persName have a ref unless it is part of the document metadata, and also requires that the pointers on the ref attribute point to person elements, even if through intermediate alt or link.

Required ref of persName eventually refers to person using abstract patterns One or more of the items pointed to by @target of this (of type "person") does not point to a 'person' element. provides a definition for the entity being named or referred to via a single URI pointer

If the word or phrase being encoded directly refers to multiple entities, this attribute should point to a link which in turn should point to the definitions.

If the word or phrase being encoded is ambiguous, this attribute should point to a alt which in turn should point to the various possible definitions.

A transcribed personal name (as opposed to a personal name in the metadata) must have a @ref The @ref of a personal name should refer to a 'person' element, or either a 'link' or 'alt' element with type="person"; but this one points to the '' with ID "".

This set of constraints makes use of the abstract pattern facility of ISO Schematron.

Resolve prefixDef for references to people Empty @ref of element '' Each pointer in the @ref attribute of a '' element should point to a 'person' element, but the pointers in this one ("") point to the following in the order specified: .

The constraint above checks only ref attributes that occur on persName, author, and rs type="person".

Note that the constraints above only check the prefixDefs that are in the teiHeader child of the outermost element; prefixDefs inside nested TEI or teiCorpus elements are ignored. This can easily be changed by either removing just the asterisk (*, U+002A) from the XPath, or replacing the entire precise XPath with //tei:prefixDef. These paths may be slower, however.

Earlier examples have avoided use of the less-than sign (<, U+003C) and greater-than sign (>, U+003E) inside messages to indicate this is an element. The only reason for this is that some Schematron processors produce ugly output when these characters occur in a message.

XIncluded yet? — one size fits all Error! XInclude processing has not been performed yet. Many rule-based tests will fail because of this.

This is a very generic test which is easy to write and very fast, but does not tell the user which particular test will fail.

Note that the TEI processors do not automatically bind the xi: prefix to the XInclude namespace. It is common to put all of your sch:ns elements in a constraintSpec that is a direct child of the schemaSpec of your customization ODD.

XIncluded yet? — per-test reporting There is no <taxonomy> in the header, so no <catRef> pointer can point into the taxonomy. (Have you performed XInclude processing yet?) This <catRef> does not point to a <category>!

This example presumes that the target of catRef is a single shorthand pointer that actually points to something in the current document. Thus this constraint should be used in conjunction with constraints like those in and (which itself should be used in conjunction with a constraint like or ).

Test for any local URI

This example is different from the others in that it is not intended to be copied, pasted, modified, and used. (And thus has not been tested as well as the others.) It is really here only to demonstrate that a) testing for the syntax of a general URI can be done, and b) it is very complex. I suspect that the vast majority of TEI projects use only a very specific subset of local URI formats — e.g., only relative filepaths with or without a fragment identifier (e.g. KRdict.xml#e05 or just KRdict.xml), or even only relative filepaths that start with an explicit dot-segment (e.g. ./KRdict.xml) — and thus would either test only for the syntax of that particular format, or simply test to see if the desired target is retrievable, which is demonstrated in .

Value of @target should be a local URI (a relative URI reference or a 'file:' scheme URI).

Note that this constraint only tests that the value of target is either a relative URI reference (e.g., kryptonian_charDecl.xml) or uses the file: URI scheme. It does not test that the file referred to with the file: scheme, if any, is actually local. This would require testing that the host specified is, in fact, the local machine. If the host is not specified or is localhost, it definitionally refers to the local machine. But for any other value of host the address would need to be resolved, the local address would need to be ascertained, and the two would need to be compared.

Note that the test in this example does not permit zone identifiers, which were added to IPv6 addresses by RFC 6874.

ref uses defined prefix

This example tests that each separate pointer in a ref attribute on a geogName, name, persName, placeName, pubPlace, rs, or title element begins with a prefix that is defined in the listPrefixDef.

One or more of the references in the @ref of this does not start with a defined prefix. The complete set of pointers on this @ref is "". The complete set of defined prefixes is .

Note that the above code requires an XSLT3 queryBinding, as it uses tokenize(). In order to convert this to a constraint that will work with an XSLT2 queryBinding, change the tokenize() to tokenize( normalize-space(.), ' ').

This constraint could be quite a bit simpler if there were never a need to test more than 1 value in a given ref. In which case, one would only need test that substring-before( normalize-space( @ref ),':') = /tei:TEI/tei:teiHeader/tei:encodingDesc/tei:listPrefixDef//tei:prefixDef/@ident.