# Todo-List for Harvest

Home | Sites using Harvest | Download | Contributed Code | Todo List | Links Contributors | User's Manual | FAQ | Installation | ChangeLog | NEWS

## Short Term Goals

- zquery.pl
  - Evaluate ranking.
  - Display search time in result page, this needs sub second resolution timer.
  - Implement proximity search.
  - Implement attribute (element) search.
  - Implement load limiter/query cost guesser.
  - Implement load balancing with replicas.
  - Consider how to deal with very expensive queries like searching for stopwords (the, and, etc.).
  - Remove unused functions read_hash_zoomsh() and remove_quirks().
  - Create internationalization framework.
- soif2xml.pl
  - Remove obvious junk like "-----" from summaries?
  - Reformat summaries to have approximately 80 characters per line? (`fmt -w 80` or use Perl's `format` and `write`) This should make parsing results easier and faster, but we have to evaluate if it's worth the additional work to reformat every object in the gatherer if it turns out, that the number of hits and displayed summaries involves just a small fraction of the numbers of objects in broker.
  - Truncate length of element to a certain maximum of for example 20KB? This will affect ranking.
  - Normalize the element set?
  - Add missing title in some summaries? This will affect ranking. Currently, missing titles are added by zquery.pl.
  - Add additional element, for example <result-display> for displaying results? This should minimize work for zquery.pl.
  - title is repeated at the beginning of full-text sometimes. Consider how to deal with it. Currently, zquery.pl removes the title from full-text. Removing title in full-text field will affect ranking. If we decide to remove it from summary, this should be best done in summarizer instead of by soif2xml.pl.
- Tighten permission for files and directories in broker directory.
- Make `$HARVEST_HOME/RunHarvest`, `$HARVEST_HOME/bin/CreateBroker` and `$HARVEST_HOME/bin/Harvest` obsolete. Creating broker should be a matter of copying template files and editing a configuration file.
- Simplify the management of multiple gather daemons (gatherd). Currently you have to assign a different port for each gatherds and have to add them as collection points in brokers.
- Automate zebrasrv restarts and index updates.
- Do some research on encoding and language recognition, evaluate the possibility to use iconv to build a collection of UTF-8 or other suitably encoded full-text material.
- Evaluate the impact of filtering out stopwords from summaries, do some semantic check to filter out search irrelevant data like words longer than a certain number of letters, repeating letters more than n times, mixed letter and digits, words containing only digits, words containing arbitrary mixed case letters, etc.
- Create/evaluate XSL for presentation of search results, which are in XML.
- Reorganize installation process so that zebra and curl files fit better in the current installation tree of Harvest.
- Clean up QDBM integration.
- Add support for embed tags in gatherer, so that we can summarize Macromedia Flash files.
- Drop support for Glimpse in 1.9.
  - Remove URI element from summaries.
  - comment out glimpseindex.bin in glimpseindex.
- Port fixes to 1.8 series.
  - xlhtml fix.
  - quicksum.cf fix.
  - summarize.c fix.
  - Remove md5 based duplicate elimination.
  - Merge Russian translation of Harvest User's Manual.
  - broker.conf.in typo fix.
  - Port changes around brokerAdmin.cgi.

## Gatherer

- Add hooks to plug in a customized candidate selection filter. This would make it possible to use for example some statistic based methods to distinguish between useful pages and junk.
- Implement candidate selection filter for httpenum.
- Modify httpenum to trust mime type sent by http servers.
- Fix crashing httpenum when essence dies.
- Fix crashing httpenum when database is not in sane state.
- Fix essence not to crash when receiving a signal.
- Unify breadth and depth first enumerator.
- Emulate breadth first enumeration for HTTP-If-Modified-Since gathering for httpenum-breadth.
- Make Delay=x work when local disc cache is disabled. Currently only global "Access-Delay" works. Possibly a bug in enum.c not to call set_envs() for child process.
- Check performance with local disc cache switched off and disable it, if it doesn't slow down gathering.
- Modify Gatherer to spawn multiple enumerators where possible, e.g. the hostname is uniq, hostmax = 1, and the number of running enumerators is smaller than a defined limit. This will be a prototype only, because the http enumerator is too heavyweight 25MB+, to make it really usable.
- Instead of creating temporary filename with tempnam() and keep track of temporary files to delete after processing, use a single name for temporary files and let them be overwritten by the next item. (Wait with this until the implementation of spawning multiple gatherer, since it might be useful when multiple gatherer creates temporary files in the working directory.)
- Spawn multiple enumerators to multiple hosts when possible.
- Add support for importing data from htdig 3.2.
- Add support for importing data from larbin, webbase, aspseek, mnogosearch.
- Add HTTPS support.

## Broker

- Store objects as QDBM BLOBS.
- Store book-keeping information apart from fulltext data.

## Documentation

- Switch from linuxdoc to docbook.
- Document environment variables.
- Document how to plug in other gatherers like htdig.

## General Issues

- Make fulltext objects shareable between gatherer and broker when they are running on same machine.
- Get rid of LeafNode option?

21 November 2003 Kang-Jin Lee

Home | Sites using Harvest | Download | Contributed Code | Todo List | Links Contributors | User's Manual | FAQ | Installation | ChangeLog | NEWS