# Harvest: A Distributed Search System

Home | Sites using Harvest | Download | Contributed Code | Todo List | Links Contributors | User's Manual | FAQ | Installation | ChangeLog | NEWS

## About

Harvest is a system to collect information and make them searchable using a web interface. Harvest can collect information on inter- and intranet using http, ftp, nntp as well as local files like data on harddisk, CDROM and file servers. Current list of supported formats in addition to HTML include TeX, DVI, PS, full text, mail, man pages, news, troff, WordPerfect, RTF, Microsoft Word/Excel, SGML, C sources and many more. Stubs for PDF support is included in Harvest and will use Xpdf or Acroread to process PDF files. Adding support for new format is easy due to Harvest's modular design.

## Features

Harvest is a modular, distributed search system framework with a working set components to make it a complete search system. The default setup is to be a web search engine, but it is also much more and provides following features:

- Harvest is designed to work as distributed system. It can distribute the load among different machines. It is possible to use a number of machines to gather data. The fulltext indexer doesn't have to run on the same machine as broker or web server.
- Harvest is designed to be modular. Every single step during collecting data, and answering search requests are implemented as single programs. This makes it easy to modify or replace parts of Harvest to customize its behaviour.
- Harvest allows complete control over the content of data in the search database. It is possible to customize the summarizer to create desired summaries which will be used for searching. The filtering mechanism of Harvest allows to make modifications to the summary created by summarizers. Manually created summaries can be inserted to the search database.
- The Search interface is written in Perl to make customization easy, if desired.

## Example Usage

- Web Search Engine

  Harvest can be used to provide search systen to websites. This is most commonly used setup for Harvest. While this works well for many sites, you may also want to take a look at more webcentric systems listed on the links on the Harvest homepage.

- Specialized Search System

  Harvest provides a wide range of methods to process (transform, delete) the gathered objects depending on file name, file content, and retrieving protocol. Using these features, you can build a search system for special purposes. Harvest distribution includes some examples to build a RFC search system, as well as Mail, News, and some more search systems.

- Building a Distributed Search System

  Harvest is designed to be a distributed search system where machines work together to handle the load which a single machine could not handle. Harvest also can be used to save bandwidth by deploying gatherers near the data source and exchanging the summarized data which usually is much smaller than the original data.

- Testbed for Search related Components

  Harvest can be used for experiments with search components. Harvest's modularity allows to replace a component. If you are interested in for example, crawlers, you can use the rest of Harvest to build a test system without having to build a complete search system. If you are developing fulltext engines, you can use harvest to gather and prepare data.

## Copyright

The core of Harvest is licensed under GPL. The components distributed with Harvest are also under GPL or similar license.

## Contents of this Site

- Sites using Harvest

  Link to query page of Harvest demo site. There are also query pages of some sites using Harvest. You can test Harvest before downloading and installing it on your site.

- Download

  Download information and links to release notes.

- Contributed Code

  Codes and documents provided by Harvest users.

- Todo List

  List of open tasks.

- Links

  Collection of Harvest and search related links.

- Contributors

  List of people who contributed to Harvest.

- User's Manual - PDF - PostScript - Plain Text - SGML

  Harvest User's Manual provides information about Harvest.

- FAQ - PDF - PostScript - Plain Text - SGML

  A list of Frequently Asked Questions with answers.

- Installation

  Harvest quick installation instructions.

- ChangeLog

  List of changes in Harvest sources.

- NEWS

  List of news in Harvest sources.

Home | Sites using Harvest | Download | Contributed Code | Todo List | Links Contributors | User's Manual | FAQ | Installation | ChangeLog | NEWS