# Stopwords

`Stopwords` - are the most frequently used words, i.e. words which appear in almost every document searched. Stopwords are filtered out prior to index construction, what is allow to reduce the total size of the index without any significant loss in quality of search.

## `StopwordFile` command

Load stop words from the given text file. You may specify either absolute file name or a name relative to DataparkSearch `/etc` directory. You may use several `StopwordFile` commands.

    StopwordFile stopwords/en.sl

You must use the same set of `StopwordFile` commands in `indexer.conf` and `search.htm` (`searchd.conf` if `searchd` is used).

## Format of stopword file

You may create your own stopword lists. As an example you may take the English stopword file `etc/stopwords/en.sl`. In the beginning of the list please specify the following two commands:

    Language: en
    Charset:  us-ascii

- `Language` - standard (ISO 639) two-letter language abbreviation.

- `Charset` - any charset supported by DataparkSearch (see ).

Then the list of stopwords is follow, one word per line. Each word is written in character set specified above by `Charset:` command.

You may use optional `Match:` command to specify a pattern to treat any word match it as a stopword. E.g.:

    Match: regex ^\$##

According to this command, any word begins with `$##` will be considered as a stopword.

Options of `Match:` command are the same as for `Allow` (see ). Arguments are in character set specified by `Charset:` command. Regular expressions are limited at the moment (e.g. intervals aren't supported).

# Clones

`Clones` -- are documents having equal values of Hash32 on all document sections. Indentical copies of the same document always have equal values of Hash32. This allow to eliminate duplicate documents in a collection. However, if only `title` section is defined in `sections.conf`, all documents with different bodies but with identical titles will be considered as clones.

## `DetectClones` command

    DetectClones yes/no

Allow/disallow clone detection and eliminating. If allowed, indexer will detect the same documents under different location, such as mirrors, and will index only one document from the group of such equal documents. "DetectClones yes" also allows to reduce space usage. Default value is "yes".

    DetectClones no