			Peer - a perl news sucker


   Peer is a perl script and two modules that allow news articles to be
   fed from a remote server to another server. It requires only the perl
   module Net::NNTP, which is part of the libnet package available for
   your favourite CPAN mirror, and needs no special permission or access
   to innd's history file to run.

   One module (NNTPPeer.pm) implements a subset of the commands given in
   the draft-barber-nntp-imp-07 which defines an extension of rfc 977.
   Specifically it implements the TAKETHIS and CHECK commands of
   streaming mode. This module may be of use in other applications.
     
   The other module (PeerTransfer.pm) is the meat of the program;
   essentially only two routines are called in the actual script which
   respectively initiate and actually do the transfer of articles.


Features

     * Totally independent of the news server. It needs to know nothing
       about the article spool area, active file or history; in fact it
       doesn't need to even to run on the same machine as the news
       server.

     * Needs no special privileges to run.

     * Multiple article streams can be used for a very fast (for
       something written perl) transfer rate. It should certainly be
       faster than suck when using kill files.

     * Kill files may be specified as a series of call backs, one for
       each newsgroup to which the article was posted, to perl functions
       which return 1 if the article is killed; 0 otherwise.

     * Standard newsrc file (called status.conf in the default
       implementation) compatible with suck's.

     * Logging is possible via syslog.

     * Written in perl so it should be eminently hackable.

     * News may be collected via newnews or article by article

 
Install	
	
	Place all the modules (the .pm files) and the sample script (peer.pl)
	in a directory. You now need to edit the sample script to give details
	of your local configuration.

	At the very least you need to define the main contructor for the
	transfer so it knows where to get articles from and where to send them.

	Something like:

$link = PeerTransfer->new (
	'from' => "news.at.your.isp.com", 
	'to' => "localhost",
	'statfile' => '/var/lib/news/peer/status.conf',
	'again' => '/var/lib/news/peer/barf.log',
	'Debug' => '0',
	'syslog' => '1',
	'kill' => $kill,
	'con' => '5',
	'newnews' => '1'
);

'from' 		is the hostname of the server to get news from

'to' 		is the hostname of the computer to send it to

'statfile' 	is a random file name to store status information in
           	between runs. If you use newnews the modification time of
		this file is used to work out the last time you transfered
		news. IN THIS CASE MAKE SURE THE FILE EXISTS BEFORE
		RUNNING THE SCRIPT. You might want to use touch to set the
		mtime back 24 hours to get 24 hours worth of news.
			
		If you don't use newnews it's used like a newsrc file and
		should be compatible with suck's newsrc file. If it
		doesn't exist the script will attempt to get the last 100
		articles in each group.
			
'again'		is another random file name. If the transfer gets
		interupted for some reason this is where article id's
		will get dumped. If it exists when the script starts up
		it'll add the article id's in here to the list of
		articles to get so you shouldn't lose anything.

'Debug' 	Set it to '1' if you want _very_ verbose debugging.
		Probably not useful and can be omitted.

'syslog'	Set to '1' if you want logging done to syslog, otherwise
		set to '0' (or omit it) and logging will be done to stdout
		instead. Until you're happy with the operation it's
		probably best to run the script by hand and have logging
		to stdout.

'kill'		This is a reference to a hash of kill callbacks - see
		below. You can omit this if you don't want to use kill
		files.
			
'con'		The number of article streams to use. I've used 5 on a
		machine with 40MB of memory without problems

'newnews'	Set to '1' to use newnews to get the article id's. This is
		the default if the variable is omitted. Not all
		newsservers support newnews so you may have to set it to
		'0'.
			
	You might want to check $link is defined ok after calling the
	constructor. Something like the following will check this:

$link || die "Unable to connect. $!\n";

	Now you need to tell it to actually do the transfer and which
	groups you don't want transfered. By default it will attempt to
	transfer new articles for every newsgroup in your local active
	file.

$res = $link->DoTransfer('to','test','demon.test','junk');

	Here we're transfering everything except the newsgroups "to",
	"test", "demon.test" and "junk".

	The function will return '1' if there's a problem. You can test for
	this and print out the error message with the following:

if($res==1) { print $link->error(); }


Kill Files
							
	If you want to use kill files you'll have to define your kill
	reference before you call the above constructor with your kill
	reference. First define the callbacks - these should be references
	to perl functions. Each function is called with a hash reference
	whose keys are the various headers on the articles and values are
	the values of the respective headers.

	If the function returns 0 or the empty string then the article is
	accepted. If it returns a string then the article is rejected and
	the string logged as the reason for rejecting the article when the
	article's header is saved in the kill logfile.
	
	Here are three example kill functions:
	
$size_check = sub {
	my $header = shift;
	my($num,@groups);
	
	@groups = split /,/, $header->{'newsgroups:'};
	if(@groups > 5) { return("Max Groups Exceeded"); }
	if($header->{'lines:'} > 667) { return("Max Lines Exceeded"); }
	return(0);
};

$kill_nose = sub {
	my $header = shift;
	
	if($header->{'newsgroups:'} =~ /alt\.fan\.karl-malden\.nose/) {
    	return("Crosspost to flame group");
    }
    return(0);
};

$chk_control = sub {
	my $header = shift;
	unless((defined $header->{'control:'})||
	$header->{'subject:'} =~ /^.?cmsg/)) { return("No control"); }
	return(0);
};

	
	The first checks the number of groups in the newsgroup line is less
	than 6; if it's greater then the article is discarded with reason
	`Max Groups Exceeded'. Then it checks the article isn't over 667
	lines in length, otherwise it's killed. The second checks that it
	isn't crossposted to alt.fan.karl-malden.nose. The third checks that
	the article is a control message.

	Try to code these callbacks as efficiently as possible since it is
	possible many will be called to check each article.

	Having defined the callbacks you need to put them into the kill
	hash, along with a suitable file to log the headers and reasons for
	killing articles. Here is an example for the constructor above:
	
$kill = {
	'log' => '/var/log/kill.log',
	'Master' => $size_check,
	'control' => $chk_control,
	'demon.local' => $kill_nose
};
	
	'log' is simply a file to log all headers to.
	
	'Master' is a special newsgroup. The kill function assigned to
		master is applied to all groups.
		
	The other keys are just names of newsgroups. For each article the
	newsgroup line is checked and if there is a function associated to
	any newsgroup then that function is called to decide whether to
	accept or reject the article.	


Running the script

	You might want to run the script by hand the first few times to make
	sure the download script is correct. Once you're happy with it you
	can put it in /etc/ppp/ip-up (or whichever file gets run when the
	link comes up).
	
	
Problems, Suggestions

	If you think there is a bug or you have a feature request, feel free
	to email me (smd@hopf.demon.co.uk).
	
	If you want to be kept informed when new releases are made you can
	subscribe to the peer list. To subscribe send mail to
	majordomo@hopf.demon.co.uk with body "subscribe peer".
	
		
