# DataparkSearch HTML parser

## Tag parser

Tag parser understands the following tag notation:

- < ... parameter=value ... >

- < ... parameter="value" ... >

- < ... parameter='value' ... >

## Special characters

indexer understands the following special HTML characters:

- &lt; &gt; &amp; &nbsp; &quot;

- All HTML-4 character entities: &auml; &uuml; and other.

- Characters in their Unicode code notation: &#234;

## META tags

Indexer's HTML parser currently understands the following META tags. Note that `"HTTP-EQUIV"` may be used instead of `"NAME"` in all entries.

- `<META NAME="Content-Type" Content="text/html; charset=xxxx">` This is used to detect document character set if it is not specified in `Content-type` HTTP header.

- `<META NAME="REFRESH" Content="5; URL=http://www.somewhere.com">` URL value will be inserted in database.

- `<META NAME="Keywords" Content="xxx">`

- `<META NAME="Description" Content="xxx">`

- `<META NAME="Robots" Content="xxx">` with content value `ALL, NONE, INDEX, NOINDEX, FOLLOW, NOFOLLOW, NOARCHIVE`.

- `<META NAME="DP.PopRank" Content="xxx">` with Content value as a real number. This is used to assign the initial value of PopularityRank for the page.

## Links

HTML parser understand the following links:

- <A HREF="xxx">

- <IMG SRC="xxx">

- <LINK HREF="xxx">

- <FRAME SRC="xxx">

- <AREA HREF="xxx">

- <BASE HREF="xxx">

  If BASE HREF value has incorrectly formed URL, current one will be used instead to compose relative links.

However, you can specify the list of HTML which would be omitted in new href lookup with `SkipHrefIn` command.

    HrefSkipIn "img, link, script"

## Comments

- Text inside the <!-- .... --> tag is recognized as HTML comment.

- You may use special <!--UdmComment--> .... <!--/UdmComment--> comment tags to exclude the text between from indexing. This may be useful to hide such things like menus and others from indexing.

- You may also use <NOINDEX> ... </NOINDEX> as a synonyms to <!--UdmComment--> and <!--/UdmComment-->

- For compatibility with ASPSeek, <!--noindex--> ... <!--/noindex--> are also equivalent to <!--UdmComment--> and <!--/UdmComment-->

- Google's special comments <!-- google_ad_section_start -->, <!-- google_ad_section_start(weight=ignore) --> and <!-- google_ad_section_end --> consider as tags to include/exclude content for indexing.

## Body patterns

If you need index not whole page, for example, to exclude navigation, ads, etc., you may use `BodyPattern` command to specify a pattern to extract content of a page for indexing. For example:

    BodyPattern "<!--content-->(.*)<!--/content-->" "$1"

this pattern will extract content between special comments and only that content will be indexed for this page.

You may specify several `BodyPattern` commands, but only the first match will be applied to a page. These patterns are trying to apply to all pages indexed. Beware, huge number of such body patterns may hurt indexing speed.

## Sub-documents

The sub-documents are: frames, iframes and embedded objects (flash tubes in general); temporary redirects (often used to place cookies or redirect to a page, depending on the language preferences of the user); versions of the same page in different languages obtained with Content negotiation.

The indexing of sub-documents is controlled by two commands: the `SubDocLevel` command sets the maximum nesting level of a sub-document to be indexed. The default value is 0, which prohibits the sub-document indexing. The `SubDocCnt` command sets the maximum number of sub-documents to be indexed at all nesting levels (this command is mainly to prevent endless cycles of pages nested into each others). The default value is 5.