# News extensions

By Heiko Stoermer heiko.stoermer@innominate.de

mnoGoSearch comes with an integrated extension to archive news servers. (currently MySQL only! see ) This means that you can now download all messages from a news server an save them completely in a database.

## Benefits

- you can expire the messages on the news server to keep it slim and fast

- you can search the complete message base with all the features that regular mnoGoSearch offers

- you can still browse discussion threads over the complete archive

## Restrictions

- currently mysql only (I would have really liked to do this for postgresql, but some really annoying restrictions concerning query size and field size in postgresql finally made me switch to mysql.)

- perl front-end only

- single dict only (because mysql-perl front-end does not support multi-dict)

## To be implemented

No new features are planned for this thing. It works the way it is (at least as far as I can see) and does everything I wanted it to do. What I will do is to make the code a bit more portable to other databases and fix the few very tiny bugs in the front-end. Of course newly discovered bugs will be fixed. I'm maintaining it as good as I can.

## Performance

Of course, important questions always are: how fast.../how big.../how long....

- Our local intranet installation of mnoGoSearch says the following:

                mnoGoSearch statistics

          Status    Expired      Total
         -----------------------------
             200      76132      76132 OK
             404        119        119 Not found
             503         17         17 Service Unavailable
             504        802        802 Gateway Timeout
         -----------------------------
           Total      77070      77070

  which means that roughly 77.000 messages are archived in the database

- Current database size is: 423 Megabytes

- The dict table has 6.076.462 entries

- It's run on an AMD K6 400 with 64 MBs of RAM (very tiny thing)

- typical queries take between 2 and 10 seconds.

## Installation

1. Compile:

   Unpack the mnoGoSearch distribution archive. Start the configure script with the option `--with-mysql`. `make` and `make install` as described in the regular install instructions

2. Create Database:

   The news extension uses a slightly different database layout. The create files can be found in `frontends/mysql-perl-news/create/` (Of course you have to do `mysqladmin create mnoGoSearch` first and set permissions to the account the web front-end and indexer are run as)

3. Install `indexer.conf`:

   an `indexer.conf` for incremental news archiving (messages hardly ever change...) can be found in `frontends/mysql-perl-news/etc/` together with a sample cron shell script that can be run once a day or so. Please see `indexer.conf` for detailed description of the indexing process.

4. Install perl front-end:

   copy `frontends/mysql-perl-news/*.pl` and `frontends/mysql-perl-news/*.htm*` to your cgi-bin directory.

   copy `frontends/mysql-perl-news/*.pm` to your site's perl library directory (`site_perl` or so) where the modules can be found by the perl scripts.

   edit `search.htm` and change the included database login information. The Perl front-end has additional features that allow you to browse message threads. You will see.

5. Now you are set and can run indexer for the first time according to the instructions you can find in `indexer.conf`.

I hope this is a nice feature for you. If anyone is interested in porting this to other databases/multidict mode/the PHP front-end, PLEASE DO SO! I would be pleased and will assist you.