Zebra - User's Guide and Reference
Prev |  | Next

# Chapter 8. The Record Model

The Zebra system is designed to support a wide range of data management applications. The system can be configured to handle virtually any kind of structured data. Each record in the system is associated with a record schema which lends context to the data elements of the record. Any number of record schemas can coexist in the system. Although it may be wise to use only a single schema within one database, the system poses no such restrictions.

The record model described in this chapter applies to the fundamental, structured record type `grs`, introduced in the Section called Record Types in Chapter 5.

Records pass through three different states during processing in the system.

- When records are accessed by the system, they are represented in their local, or native format. This might be SGML or HTML files, News or Mail archives, MARC records. If the system doesn't already know how to read the type of data you need to store, you can set up an input filter by preparing conversion rules based on regular expressions and possibly augmented by a flexible scripting language (Tcl). The input filter produces as output an internal representation, a tree structure.

- When records are processed by the system, they are represented in a tree-structure, constructed by tagged data elements hanging off a root node. The tagged elements may contain data or yet more tagged elements in a recursive structure. The system performs various actions on this tree structure (indexing, element selection, schema mapping, etc.),

- Before transmitting records to the client, they are first converted from the internal structure to a form suitable for exchange over the network - according to the Z39.50 standard.

# Local Representation

As mentioned earlier, Zebra places few restrictions on the type of data that you can index and manage. Generally, whatever the form of the data, it is parsed by an input filter specific to that format, and turned into an internal structure that Zebra knows how to handle. This process takes place whenever the record is accessed - for indexing and retrieval.

The RecordType parameter in the `zebra.cfg` file, or the `-t` option to the indexer tells Zebra how to process input records. Two basic types of processing are available - raw text and structured data. Raw text is just that, and it is selected by providing the argument text to Zebra. Structured records are all handled internally using the basic mechanisms described in the subsequent sections. Zebra can read structured records in many different formats. How this is done is governed by additional parameters after the "grs" keyword, separated by "." characters.

Four basic subtypes to the grs type are currently available:

grs.sgml

  This is the canonical input format — described below. It is a simple SGML-like syntax.

grs.regx.filter

  This enables a user-supplied input filter. The mechanisms of these filters are described below.

grs.tcl.filter

  Similar to grs.regx but using Tcl for rules.

grs.marc.abstract syntax

  This allows Zebra to read records in the ISO2709 (MARC) encoding standard. In this case, the last parameter abstract syntax names the `.abs` file (see below) which describes the specific MARC structure of the input record as well as the indexing rules.

grs.xml

  This filter reads XML records. Only one record per file is supported. The filter is only available if Zebra/YAZ is compiled with EXPAT support.

## Canonical Input Format

Although input data can take any form, it is sometimes useful to describe the record processing capabilities of the system in terms of a single, canonical input format that gives access to the full spectrum of structure and flexibility in the system. In Zebra, this canonical format is an "SGML-like" syntax.

To use the canonical format specify `grs.sgml` as the record type.

Consider a record describing an information resource (such a record is sometimes known as a locator record). It might contain a field describing the distributor of the information resource, which might in turn be partitioned into various fields providing details about the distributor, like this:

          <Distributor>
            <Name> USGS/WRD </Name>
            <Organization> USGS/WRD </Organization>
            <Street-Address>
              U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
            </Street-Address>
            <City> ALBUQUERQUE </City>
            <State> NM </State>
            <Zip-Code> 87102 </Zip-Code>
            <Country> USA </Country>
            <Telephone> (505) 766-5560 </Telephone>
          </Distributor>

The keywords surrounded by <...> are tags, while the sections of text in between are the data elements. A data element is characterized by its location in the tree that is made up by the nested elements. Each element is terminated by a closing tag - beginning with `<`/, and containing the same symbolic tag-name as the corresponding opening tag. The general closing tag - `</>` - terminates the element started by the last opening tag. The structuring of elements is significant. The element Telephone, for instance, may be indexed and presented to the client differently, depending on whether it appears inside the Distributor element, or some other, structured data element such a Supplier element.

### Record Root

The first tag in a record describes the root node of the tree that makes up the total record. In the canonical input format, the root tag should contain the name of the schema that lends context to the elements of the record (see the Section called Internal Representation). The following is a GILS record that contains only a single element (strictly speaking, that makes it an illegal GILS record, since the GILS profile includes several mandatory elements - Zebra does not validate the contents of a record against the Z39.50 profile, however - it merely attempts to match up elements of a local representation with the given schema):

           <gils>
              <title>Zen and the Art of Motorcycle Maintenance</title>
           </gils>

### Variants

Zebra allows you to provide individual data elements in a number of variant forms. Examples of variant forms are textual data elements which might appear in different languages, and images which may appear in different formats or layouts. The variant system in Zebra is essentially a representation of the variant mechanism of Z39.50-1995.

The following is an example of a title element which occurs in two different languages.

           <title>
           <var lang lang "eng">
           Zen and the Art of Motorcycle Maintenance</>
           <var lang lang "dan">
           Zen og Kunsten at Vedligeholde en Motorcykel</>
           </title>

The syntax of the variant element is `<var class type value>`. The available values for the class and type fields are given by the variant set that is associated with the current schema (see the Section called The Variant Set (.var) Files).

Variant elements are terminated by the general end-tag </>, by the variant end-tag </var>, by the appearance of another variant tag with the same class and value settings, or by the appearance of another, normal tag. In other words, the end-tags for the variants used in the example above could have been omitted.

Variant elements can be nested. The element

           <title>
           <var lang lang "eng"><var body iana "text/plain">
           Zen and the Art of Motorcycle Maintenance
           </title>

Associates two variant components to the variant list for the title element.

Given the nesting rules described above, we could write

           <title>
           <var body iana "text/plain>
           <var lang lang "eng">
           Zen and the Art of Motorcycle Maintenance
           <var lang lang "dan">
           Zen og Kunsten at Vedligeholde en Motorcykel
           </title>

The title element above comes in two variants. Both have the IANA body type "text/plain", but one is in English, and the other in Danish. The client, using the element selection mechanism of Z39.50, can retrieve information about the available variant forms of data elements, or it can select specific variants based on the requirements of the end-user.

## Input Filters

In order to handle general input formats, Zebra allows the operator to define filters which read individual records in their native format and produce an internal representation that the system can work with.

Input filters are ASCII files, generally with the suffix `.flt`. The system looks for the files in the directories given in the profilePath setting in the `zebra.cfg` files. The record type for the filter is `grs.regx.`filter-filename (fundamental type `grs`, file read type `regx`, argument filter-filename).

Generally, an input filter consists of a sequence of rules, where each rule consists of a sequence of expressions, followed by an action. The expressions are evaluated against the contents of the input record, and the actions normally contribute to the generation of an internal representation of the record.

An expression can be either of the following:

INIT

  The action associated with this expression is evaluated exactly once in the lifetime of the application, before any records are read. It can be used in conjunction with an action that initializes tables or other resources that are used in the processing of input records.

BEGIN

  Matches the beginning of the record. It can be used to initialize variables, etc. Typically, the BEGIN rule is also used to establish the root node of the record.

END

  Matches the end of the record - when all of the contents of the record has been processed.

/pattern/

  Matches a string of characters from the input record.

BODY

  This keyword may only be used between two patterns. It matches everything between (not including) those patterns.

FINISH

  The expression associated with this pattern is evaluated once, before the application terminates. It can be used to release system resources - typically ones allocated in the INIT step.

An action is surrounded by curly braces ({...}), and consists of a sequence of statements. Statements may be separated by newlines or semicolons (;). Within actions, the strings that matched the expressions immediately preceding the action can be referred to as $0, $1, $2, etc.

The available statements are:

begin `type [parameter ... ]`

  Begin a new data element. The `type` is one of the following:

  record

    Begin a new record. The following parameter should be the name of the schema that describes the structure of the record, eg. `gils` or `wais` (see below). The `begin record` call should precede any other use of the `begin` statement.

  element

    Begin a new tagged element. The parameter is the name of the tag. If the tag is not matched anywhere in the tagsets referenced by the current schema, it is treated as a local string tag.

  variant

    Begin a new node in a variant tree. The parameters are `class type value`.

data `parameter`

  Create a data element. The concatenated arguments make up the value of the data element. The option `-text` signals that the layout (whitespace) of the data should be retained for transmission. The option `-element` `tag` wraps the data up in the `tag`. The use of the `-element` option is equivalent to preceding the command with a `begin element` command, and following it with the `end` command.

end `[type]`

  Close a tagged element. If no parameter is given, the last element on the stack is terminated. The first parameter, if any, is a type name, similar to the `begin` statement. For the `element` type, a tag name can be provided to terminate a specific tag.

unread `no`

  Move the input pointer to the offset of first character that match rule given by `no`. The first rule from left-to-right is numbered zero, the second rule is named 1 and so on.

The following input filter reads a Usenet news file, producing a record in the WAIS schema. Note that the body of a news posting is separated from the list of headers by a blank line (or rather a sequence of two newline characters.

          BEGIN                { begin record wais }

          /^From:/ BODY /$/    { data -element name $1 }
          /^Subject:/ BODY /$/ { data -element title $1 }
          /^Date:/ BODY /$/    { data -element lastModified $1 }
          /\n\n/ BODY END      {
             begin element bodyOfDisplay
             begin variant body iana "text/plain"
             data -text $1
             end record
          }

If Zebra is compiled with support for Tcl enabled, the statements described above are supplemented with a complete scripting environment, including control structures (conditional expressions and loop constructs), and powerful string manipulation mechanisms for modifying the elements of a record.

Prev | Home | Next
Z39.50 Protocol Support and Behavior |  | Internal Representation