
		    README FILE FOR DIABLO TRANSIT FEEDER

				    ---

    DIABLO is a news transit and reader system.  This readme file describes
    the transit side of things.  You can operate the transit and reader
    sides separately or together, or just operate one or the other depending
    on your needs.

    The transit side of Diablo is designed to transit news between one or
    more incoming feeds and one or more outgoing feeds.  The transit side
    of Diablo is not designed to operate as a reader.  However, it may be 
    used to back a reader's spool for readers that fetch articles by message-id
    (which the reader side of Diablo does).  

    Since the transit portion of Diablo is not designed to support readers,
    an active file is not usually used with it ('active off' in diablo.config).
    If you want the transit portion of Diablo to act as the master article 
    number assignment point which downstream sites are slaved off of, you turn
    ON feeder-side active file support in diablo.config and
    run the reader side, dreaderd, with the -x option to slave it's article
    numbering to the master's.  When turned on, the active file only effects
    how the Xref: header is generated by the transit side of Diablo.  The
    transit side of Diablo will still transit articles even if none of the
    newsgroups are listed in the active file.

    The feeder side of diablo does not process control messages, so if you 
    are mastering article numbers on the feeder, you either have to run the
    reader on the same box in order to process control messages and keep the
    active file up to date, or you must periodically synchronize the active
    file from a remote source using dsyncgroups (taking care to use the proper
    options so you do not overwrite the begin/end/other article numbering
    parameters that Diablo uses to master XRef: lines).

    The transit side of Diablo maintains a history file (the reader side does
    not).  This means that the transit side is able to take multiple feeds
    in parallel without transiting duplicate articles.    The reader side's
    ability to handle duplicate articles is limited only to the case where
    the -x option is used on dreaderd.  If the option is not used dreaderd
    CANNOT handle duplicate articles being fed to it.  It is often beneficial
    to run the feeder and reader on the same machine, with the feeder feeding
    the reader internally, even if the feeder is not going to be used as
    a (major) spool cache.  It is more common to run the feeder and the
    reader on the same machine if the feeder is given a large enough spool
    to act as the backend cache for the reader.  Or you may just want to 
    run a feeder-only machine with no reader elements on it at all.

    The transit side of Diablo (the 'feeder') is strictly for news transit
    and does not understand reader-related NNTP commands.

    The Diablo spool, usually /news/spool/news, is maintained by the transit
    system.   You must size the spool and run dexpire as appropriate to your
    needs.  Diablo stores multiple articles per file in the spool in a 
    two-tiered directory structure.  Article files & directories should
    never be directly edited or removed or you risk corrupting the reference
    data Diablo stores in the history file.  You must use dexpire to free
    space in the spool.  Since Diablo does not write out article files singly
    it tends to be much more efficient then INN without CNFS.   It is
    roughly on-par with INN + CNFS, though I personally believe it is better.

    Typically, anyone taking a full feed these days must dedicate a
    machine to it that is separate from the newsreader machine that your 
    users use to read news.  

    The DIABLO transit system is designed to replace the dedicated newsfeeds
    machine and is designed to be a mostly hands-off affair once you get past
    configuring and stabiliing it.  See TUNING_NOTES for machine configuration
    suggestions.

			      OS REQUIREMENTS

    See TUNING_NOTES.

			    WHO SHOULD RUN DIABLO

    If you need to run a USENET news feeding/transit system and/or a
    USENET newsreading system, Diablo may be for you.

			    WHERE TO GET DIABLO

    http://www.openusenet.org/diablo/

			    REPORTING BUGS

    send the bug to:	   diablo-bugs@openusenet.org
    send non-bug stuff to: diablo-users@openusenet.org
    NEWSGROUPS:		   news.software.nntp
    MAILING LISTS:	   http://www.plig.net/mailman/listinfo/diablo-users


			USE OF REALTIME FEEDS AND FEED DELAYS

	If you have several outgoing feeds, you should consider using the
	realtime, queueskip and startdelay options in dnewsfeeds.  All of your
	local and internal feeds should be realtime.  Cheap external paths
	to the internet can also be realtime.  To reduce the cost of running
	outgoing feeds over your internet transit, you may wish to weight
	the feeds according to cost.  For example, our MAE-WEST connection
	is a lot cheaper then our MCI T3, so I run outgoing feeds with 
	MAE-WEST destinations in realtime and run outgoing feeds which go
	via MCI in batch mode with a 10 second delay.  This way the articles
	may actually propogate to the more expensive destinations via other
	means prior to my actually attempting to send them direct.

	Likewise, if you have T1 and frame customrs, it is usually cheaper
	to supply them with a newsfeed yourself rather then force them to
	go to someone over the internet.  This way they are not eating your
	transit bandwidth on newsfeeds.  A realtime feed to those people is
	best.

			    CATCHING UP AFTER BEING DOWN

	The key item to monitor when catching up on incoming feeds after
	being down for a while is the incoming article rate.  Diablo will
	generate a log line for every 1024 articles received that looks like
	this:

    Jun 24 11:03:59 news1 diablo[18153]: DIABLO uptime=7:46 arts=241.000K tested=0 bytes=1.842G fed=12.613M

	You can calculate the article rate by looking at the delta activity
	from two log lines that are around an hour apart from each other.
	If the article rate is above 9 articles/sec, diablo is catching up
	reasonably well.. as of today, a full feed is around 5 articles/sec.

	With a moderate number of incoming feeds, diablo can do around 30
	articles/sec.  If you have a huge number of incoming feeds that are
	all in catchup, in-kernel filesystem locking will begin to interfere
	with the history file lookups and updates.  Diablo will be able to
	maintain a reasonable history file write transaction rate, but the
	lookup rate will suffer.

	This causes diablo to catch up on articles first without appreciably 
	reducing the backlog at remote sites due to slow check-responses. 
	Once it passes a certain threshold, however, and the load on the
	history file turns to mostly-read rather then read/write, the 
	transaction rate will increase dramatically and diablo will generally 
	be able to cleanup the backlogs very quickly after that.



		SPAMALIAS OPTION IN DNEWSFEEDS

	Submitted by: uhclem @ nemesis.lonestar.org (Frank Durda IV)

	Spamalias
	---------

	Related to an "alias" command, any originating server (not
	transit servers) or username entry in the Path: header that
	match a "spamalias" wildcard declaration will cause that
	message to not be propagated to any neighbors.  Diablo
	behaves as though some "alias" command matched that Path:
	header on all outbound feeds.

	Intended for use on transit-only servers (since in this
	implementation, "spamalias" checks are too late to prevent
	the message from being viewable from a local reader),  the
	"spamalias" parameter acts globally and must appear in the
	GLOBAL section.

	For example,

	groupdef	GLOBAL
		spamalias	badpornsite
		spamalias	xxxnilla
		spamalias	*.unresponsive.net
		spamalias	abuseisokay*
		spamalias	annoyizer
	end

	These entries would block messages with these Path: headers:

	Path: someplace!elsewhere!okaysite!badpornsite!not-for-mail
	Path: someplace!elsewhere!okaysite!notspammershonest!annoyizer
	Path: someplace!okaysite!xxxnilla!seemelive
	Path: someplace!elsewhere!okaysite!west-coast.unresponsive.net!me
	Path: someplace!elsewhere!okaysite!east-coast.unresponsive.net!them
	Path: someplace!goodplace!elsewhere!abuseisokay
	Path: someplace!goodplace!abuseisokay.com!nobody

	However, these messages would not be blocked:

	Path: someplace!xxxnilla!elsewhere!west.unresponsive.net!goodplace!xyz
	Path: someplace!elsewhere!badpornsite!goodplace!gooduser
	Path: someplace!elsewhere!abuseisokay.com!elsewhere!innocentuser

	Only the last and next-to-last elements of the Path: header are checked
	by the "spamalias" command.

	All too frequently there are sites who originate spam or
	abuse that need to be blocked (mainly to punish a lack of
	response to abuse/spam problems), but you don't want to
	lose articles that are just passing through that site and
	didn't originate there, since your history database is now
	poisoned, so you can't get the article via some other route
	just to make sure it didn't really originate there.  (If
	you have limited redundant feeds, hoping an article will
	show up via a less-tainted route may not be an option.)
	Using "spamalias" provides reasonable blocking of specific
	sites or spam/abuse signatures without blacking-out unintended
	chunks of the USENET network.

	When a "spamalias" match is made, an entry is logged to
	news.debug, indicating if it was a position A (originating
	site) or B (user) match.   The entry in incoming.log for
	such an article will show no sites to be fed.


	Odd Things and Cautions about Spamalias:

	If "spamalias" entries are used outside of the GLOBAL
	section, it may either sly slow Diablo because of the amount
	of redundant work it ends up doing and may still act globally
	or act on some but not all feeds.  The "spamalias" command
	is not designed to be used to filter subsets of outbound
	feeds.

	There is a potential to defeat "spamalias" if the blocked
	site allows Path:  preloading, but in three years of use,
	the incidents of any abuser bothering to work around such
	filtering have been virtually non-existent, and the number
	of big sites that still allow preloading and don't take
	their own action against spam/abuse is getting quite small.

	When writing "spamalias" expressions that are intended to
	match usernames, be cautious since some "dummy" usernames
	appear in posts originating from many sites, likely in
	addition to whatever it is you are trying to block.  For
	example, never block "not-for-mail" or "news", since these
	are commonly used in place of actual user names.

	Since the "spamalias" command tests both originating site
	and originating user fields, make sure an entry written
	for one field isn't busily zapping non-abusive posts that
	happen to match on the other field.


		WHEREIS COMMAND

	Diablo allows an optional NNTP command 'WHEREIS' which returns
	the location of an article on the local spool (filename, offset
	and size). This option is useful for a reader accessing the
	article from an NFS mounted spool. The use of NFS in Diablo
	is strongly discouraged, but some people are forced to use
	diablo in an NFS-only environement.

	  200 news.example.com NNTP Service Ready
	  WHEREIS <3ba5b2d9.440788@news.tel.hr>
	  223 0 whereis <3ba5b2d9.440788@news.tel.hr> in \
		/news/spool/news/D.00fe7ee6/B.036a offset 83825 length 1018

	  (line wrap added for this document)

