# Storing compressed document copies

In DataparkSearch it is possible to store compressed copies of indexed documents. Copies are stored and retrieved by the new daemon - `stored`, that is installed into `sbin` directory of DataparkSearch installation (default: `/usr/local/dpsearch/sbin`).

`stored` is also used to make documents excerpts for search results.
You may use `ExcerptSize` command in `search.htm` template to specify average excerpt size in bytes. Value by default: 256. You may switch off document excerpts (but retain ability to show stored copies) using `DoExcerpt no` command in your search template.

To enable documents copies archiving without `stored` usage, place `DoStore yes` command in your `indexer.conf` file instead of `stored` daemon configuration.

Stored document copies are retrieved by means of `storedoc.cgi` CGI script. It requests a saved copy of a documents from `stored`, then a copy is displayed with user's web browser with search keywords highlighted.

To enable `stored` support, compile DataparkSearch with `zlib` support:

    ./configure --with-zlib <other arguments>

You may use the `Store` and `NoStore` commands to allow or disallow storing several files by pattern. For arguments of those commands are exactly the same as for the `Allow` command (see ). All documents are stores by defaults, if support for stored is enabled.

## Configure stored

To start using `stored`, please do the following:

- Copy `/usr/local/dpsearch/etc/stored.conf-dist` to `/usr/local/dpsearch/etc/stored.conf`.

  Edit `/usr/local/dpsearch/etc/stored.conf`

  There are several commands available for using with `stored`:

  - `
    Listen` tells `stored` which address and/or port to bind to. By default `stored` listens to port 7004 and any address. It is possible to specify port only:

    `Listen 7004`

    Or address only:

    `Listen 127.0.0.2`

    Or both address and port:

    `Listen 127.0.0.2:7004`

  - `
    VarDir` command specifies an alternate `var/` working directory, e.g.

    `VarDir /mnt/d/dpsearch/var/`

  - `
    StoredFiles` command specifies number of stored data files created in `var/stored/` directory, e.g.

    `StoredFiles 256`

  - `
    OptimizeInterval` command specifies interval in seconds between attempts to optimize a stored datafile, e.g.

    `OptimizeInterval 300`

  - `
    OptimizeRatio` command specifies defragmentation threshold for a stored datafile optimization.

    `OptimizeRatio 3`

- Run `stored`:

  `/usr/local/dpsearch/sbin/stored &`

- Configure `indexer.conf` and `search.htm` (or `searchd.conf`, if `searchd` is used). Specify address and port that `indexer` will use to connect to `stored`. Use `stored` parameter for `DBAddr` command, e.g.:

  `DBAddr mysql://localhost/search/?dbmode=cache&stored=localhost:7004`

## How stored works

After you have successfully configured `stored`, the `indexer` pass downloaded documents to `stored` daemon. After that, `stored` will compress the received documents and save them.

## Using stored during search

To enable displaying stored documents during search, do the following:

- Configure `storedoc.htm` (`storedoc.cgi` template) if necessary.

- Add a `storedoc.cgi` link to `search.htm` `<!--res-->` section: e.g. `<A HREF="$(stored_href)">Cached copy</A>`

- Specify `storedoc.cgi` CGI script URL in `search.htm` (by default `$(stored_href)` will return `/cgi-bin/storedoc.cgi`). In case you have to specify other URL, add to `search.htm` `variables` section the following line:

  `StoredocURL /path/to/storedoc.cgi`

  Or an absolute path:

  `StoredocURL http://servername/path/to/storedoc.cgi`

This is how `stored` works during search, if everything configured correctly:

1. `search.htm` displays a link to `storedoc.cgi`;

2. When user clicks the link, `storedoc.cgi` sends a query to `stored` daemon to the address, specified in `storedoc.htm` with the `Stored` parameter of `DBAddr` command;

3. After the query, `stored` will decompress the requested saved copy and send it to `storedoc.cgi`;

4. `storedoc.cgi` parses the received document and apply highlighting of search keywords. Highlighting method is specified with `storedoc.htm` `
   HlBeg` and `
   HlEnd` commands;