SnarfNews - USENET Transport Toolkit
by Alec Muffett

E-mail: webmaster

Introduction

SnarfNews was initially written (back in 1990) in order to provide a compressed, ASCII-fied USENET feed of a selection of newsgroups to a home machine (an Amiga A500); the original project included a minimal NNTP server and article-handling system, but that has since been dropped in favour of INN on Linux, although it may be reborn one day if I have enough time.

Things you can do with SnarfNews, include:

• Fetch news articles (by newsgroup) from multiple NNTP servers and inject them into a local news-spool via "rnews", "inews", POST or IHAVE.

• Top-up your main newsfeed with articles from groups that you normally wouldn't get, by snarfing them from another server. This can be useful for receiving (eg:) local or national groups with poor propagation.

• Put articles from a local NNTP server to a remote one, via NNTP's IHAVE or POST commands.

• Gate e-mail lists into newsgroups. (preserves MIME attachments)

• Gate newsgroups into e-mail lists. (preserves MIME attachments)

• Automatically e-mail to yourself, articles that appear in specific newsgroups on an NNTP server.

• Automatically save to disk any articles that are posted to a list of newsgroups on an NNTP server, for archiving or off-line news reading.

• Transparently repost articles from one newsgroup on one NNTP server to another newsgroup on another NNTP server.

• Collect articles from a NNTP server, batch and compress them, and e-mail them to a remote server for unbatching and injection into another newsfeed. (requires gzip)

• Implement encrypted newsfeeds via batching/e-mail. (requires PGP 2.6.2 or above)

• Sanitise batched newsfeeds by removing excessively cross-posted articles, excessively large articles, "MAKE MONEY FAST", and other spams.

• Inject traditional "rnews" batches into NNTP-only News servers, by translating them into individual IHAVE or POST commands.

• Pull/push news through a SOCKS firewall. (requires "runsocks")

• Implement "streaming" retreival of USENET articles, maximising utilisation of your network bandwidth and reducing connection time.

• Enjoy automatic fall-back into RFC-977 compatability; ie: the software attempts to use common but non-standard NNTP etentions for speed, and falls back to stock NNTP commands if they fail.

Requirements

• Access to a NNTP News server, preferably running INN.
• perl5 (mandatory)
• gzip (for compressed news batches)
• pgp 2.6.2 or above (for encrypted news batches)
• SOCKS software (for transport through SOCKS firewalls)

Overall design philosophy

Of course, this modularity means there is some loss in efficiency, but the benefits for the user in being able to trivially extend the software, and in the flexibility that this approach provides, far outweigh any overhead.

You will find that some functionality is duplicated, for instance much functionality in "runfeeds" could be easily reimplemented using "ifbytes". That's OK - as with most toolkits you'll soon find out that there is more than one way to do things, to use the Perl motto.

SnarfNews deals in four basic types: the USENET article, the "rnews" batch, the encoded "rnews" batch, and the encrypted "rnews" batch.

Hopefully you'll start to see a a common thread about this point.

SnarfNews' interpretation of an "rnews" batch is currently rather strict; programs which expect to receive rnews batches cannot cope with being fed bare articles, and vice versa. For those unfamiliar with rnews batches: snarfnews slings groups of USENET articles around bundled together in a format referred to as an "rnews batch".

The format is:

    #! rnews nnn
    [article headers]
    [article headers]
    [article headers]

    [article body]
    #! rnews nnn
    [headers of next article]
    ...
    ...

...where nnn is the size of the entire article (in bytes) measured from where it starts on the next line. This is the standard format for grouping articles together for transmission to other sites, and for a more detailed reference the user is referred to the man page for "rnews".

Encoded (compressed and ASCII-fied) and encrypted batches are discussed below.

Definition of the word "gronk"

To get around this, the author defined the word "gronk", thusly:

Gronk: to gather together a bunch of USENET articles collected from disparate sources, and present them as a "rnews" batch.

Following from this, we have programs named "gronknntp", which gathers news articles via NNTP and creates an "rnews" batch from then, and "gronkfiles", which performs a similar task, reading its articles from files on disk.

Configuration

• Unpack the software into a directory, eg: /usr/local/snarf

• Hand edit the "snarf" script to reflect the directory name that the software is installed into, by setting the value of "SNARFHOME". Also take time to set the values of:
  • SNARFHOST
  • SNARFDOMAIN
  • SNARFADMIN
  • SNARFMAIL
  • SNARFXPOSTMAX
  • SOCKS_SERVER (if used)

  ...to reflect your local site requirements.

• Run the "Configure" script; this will fix the paths to Perl5 in the appropriate scripts.

• Run "make" to cleanup.

• Finish reading the manual.

• Edit the "conf/feeds" and "conf/filter/default" files appropriately.

When installing SnarfNews, check your permissions and consider who'll be running it; if you intend to wire it into /etc/aliases for mail->news gatewaying, the "$SNARFRUN" directory may have to be readable or writable by "mail", "daemon" or similar.

Similarly, if you're heavily integrating it with "INN" and have not set up a "news" group with group-execute bits, you may have to set permissions likewise for the user that "innd" runs as.

To get around this in the first instance, I ship with the permissions 01777 on the "run" directory. If this is not suitable for you, change it, but be aware of what may have to chenge to get it working in your environment.

Top Level Commands and Files

Makefile

This is the makefile for "snarfnews" which does little other than help keep things tidy and set sensible file permissions, etc. Typing:

% make clean

or

% make spotless

- occasionally, is probably a good thing.

README.html

...is this file.

snarf program [arguments...]

Snarf is the frontend for everything. Users are expected to invoke all the programs in the snarf suite as above, where the "snarf" script sets up essential environment variables, etc, which would otherwise clutter the environment unduly. Although this might seem awkward, it prevents "snarf" programs from leaking into the environment yet simplifies installation as there is only one script that needs any amount of serious work to configure.

If a user-written script needs to use several tools from the snarf toolkit, it is probably simplest to write the script as you would normally, and then invoke:

% snarf ~/bin/scriptname

- or whatever, in the appropriate place.

Frontends

snarf-inbound: "| /usr/local/snarf/snarf cbatch2rnews"

...to /etc/aliases, though care should be taken (especially by people writing their own frontends) to ensure that there is no way that this feature may be used by unauthorised people to meddle with your news traffic or gain unauthorised entry to your system.

The supplied frontends are:

art2mail address [file]

This command reads an article from "file" or from "stdin" and e-mails it to "address", eg:

% snarf art2mail user@foo.co.uk < article

art2post servername [file]

This command reads an article from "file" or from "stdin" and posts it in the NNTP server "servername" via NNTP's POST method.

batch2ihave servername [file]
batch2post server [file]

These commands read an rnews-style batches of articles from "file" or from "stdin" and post them (individually) on the NNTP server "servername" via NNTP's IHAVE or POST methods, respectively.

batch2mail to-address [file]
batch2mailfrom to-address from-address [file]
batch2mailrepl to-address reply-address [file]

These three commands read rnews-style batches of articles from "file" or from "stdin" and convert them to mail messages, which are then mailed to "to-address" by invoking "sendmail -t".

The latter two commands permit the user to specify From: and Reply-To: addresses for the messages, respectively.

cbatch2mail to-address [file]
cbatch2rnews [file]

These commands read a compressed and ASCII-encoded "rnews" batch from "file" or from "stdin" (see "batchencode", below) and either mail each article separately to "to-address", or pipe them into "rnews", which will load the articles into the local server.

% cat batchfile | snarf cbatch2rnews

ebatch2mail keyfile to-address [file]
ebatch2rnews keyfile [file]

These are analogues of "cbatch2mail" and "cbatch2rnews", but using PGP's IDEA encryption as implemented in "batchencrypt", below.

The "keyfile argument is the name of a file to be found (only) in the "$SNARFSECRET" directory, containing a long text string which should be kept as secret as possible. The contents of this file will be used by PGP as a passphrase to conventionally encrypt the batch of articles before transmission to the remote site.

The contents of the keyfile should be transferred securely to the other site (via PGP or floppy) and be installed there, where "ebatch2rnews" can be used to unpack it, and the result fed into "rnews" or "runbatch".

mail2post servername newsgroup [file]

This command reads a e-mail message from "file" or "stdin", reformats it to be a USENET article, and posts it to "newsgroup" on "servername" using NNTP's POST method.

One or more comma-separated newsgroups may be specified in the "newsgroup" argument, but there should be no spaces in the list.

Backends

b64decode [-debug] [file ...]

This command reads base-64 encoded data from "stdin" or from filenames specified on the command line, and prints the decoded data on "stdout". Lines of input which do not look like plausible base-64 input are ignored, and can be viewed on "stderr" by specifying the "-debug" switch.

b64encode

This command reads from "stdin" and reproduces its input on "stdout", using base-64 MIME encoding.

batchencode [file ...]
batchdecode [file ...]

These commands read an "rnews" batch (or indeed any arbitrary data, but within "SnarfNews" they are used only for sending "rnews" batches) from "stdin" or filenames specified on the command line, and produce compressed, base-64 encoded output, bracketed by _begin and _end_ statements.

The batchencode process requires use of "gzip" specifically because of two important features of that program:

1. Adaptive compression: "gzip" can cope sensibly with partially-compressed inputs, which is something that the standard "compress" program does not do well.

2. Catenative input: "gzip" has a feature not present in standard "compress", that several files which have been independently compressed using "gzip" may be concatenated, and when gzcat'ed, the contents of all files will be correctly decompressed to "stdout".

batchencrypt keyfile [file]
batchdecrypt keyfile [file]

These commands are analogues of "batchencode" and "batchdecode", above, but utilising PGP to compress, encrypt and ASCII-fy the input.

The argument "keyfile" specifies the name of a file which will be searched for (only) in the "$SNARFSECRET" directory, and which contains a long line of arbirtary text which will be used as the passphrase for the encryption. (see "secret/example", below).

Since PGP also uses adaptive compression and base-64 encoding, it is approximately equivalent to - sometimes slightly better than - "batchencode" in terms of the size of its output; however it has not been investigated whether PGP shares gzip's ability to cope with catenative input (see above), so for the moment it is recommended that the user does not build a system which would pipe more than one encrypted batch at-a-time, into a single instance of "batchdecrypt".

conva2a [-debug] [-skip-HEADER] [-die-HEADER] [-x-HEADER] ['Header: Value' ...]
conva2m [-debug] [-skip-HEADER] [-die-HEADER] ['Header: Value' ...]
convm2a [-debug] [-skip-HEADER] [-die-HEADER] ['Header: Value' ...]

These programs provide essential filtering functionality for "SnarfNews" when gating USENET and E-mail together; the knowledgable user will already know that there are important differences between USENET postings and E-mail messages, specifically in what header fields are required for traffic to be propagated by either of those two means.

These programs provide the ability to add, remove, and change arbitrary headers in a USENET article or E-mail message, and reformat the headers appropriately for re-transmission. Enough defaults are built into the filters to avoid most of the work in converting messages of one type to another, but certain data must be supplied on the command line. See (or use) the frontend scripts for examples.

• "conva2m" expects to read a USENET article from "stdin" and reformats it as a E-mail message on "stdout". The user is expected to supply a recipient address on the command line, thusly:

  conva2m -skip-to "To: address"

• "convm2a" expects a E-mail message and produces a USENET article. Notably, it checks the contents of the "From:" field; if the address in the "From:" field looks like a fully-qualified Internet mail address, it is left alone.

  If the address looks like:

  From: username@hostname

  ...it is modified to:

  From: username@hostname.$SNARFDOMAIN

  Otherwise, if it looks like a bare username, it is modified to:

  From: username@$SNARFHOST

  ...however, the user is free to override this by specifying:

  convm2a -no-from "From: user@host.domain.com"

  If these rules are not sufficent to provide the "From:" address rewriting functionality that you require at your site, users are recommended to write a wrapper to invoke "convm2a" with the "-skip-from" argument, and to insert their own "From:" header. Several pieces of USENET software (notably INN) will ot accept articles for posting which do not contain fully-qualified "From:" addresses.

• "conva2a" expects a USENET article and modifies certain headers so that it can be re-posted on another NNTP server via NNTP's POST method; this should not be necessary if NNTP's IHAVE method is used instead.

In all cases, a filter can be set to remove certain headers from the input via the "-skip-HEADER", eg: "-skip-subject" will remove the "Subject:" header. (header matching is case-insensitive)

Similarly messages which contain "HEADER" will be silently discarded when "-die-HEADER" is specified; as a security mechanism this is permanently enabled for the header "X-Reflected-By:" to prevent messages from passing through more than one instance of convX2Y, which could otherwise lead to a disasterous USENET/E-mail loop.

It is also for this reason that the "postart" and "ihaveart" scripts are designed to dilently ignore empty inputs, since they are likely to be the next in the pipeline after a convX2Y filter.

"Conva2a" also has the ability to "comment" out any specified header using the "-x-HEADER" mechanism, which prepends "X-" to any given header; typically however, use of this will not be required, as USENET articles require very little modification for reposting beyond what "conva2a" provides by default.

All filters can also add headers to the articles/messages that are produced, simply by specifying them on the command line, eg:

conva2m 'X-My-Favourite-Cookie: Chocolate Chip'

- will add the above header to the message generated by "convm2a".

Important Note - users who are interested in making changes to E-Mail and USENET headers (other than those already available via the SnarfNews frontends) are strongly advised to read the reference material first (RFC822, RFC1036) to see what is/is not permitted.

Never, under any circumstances, modify, add or remove a USENET article's "Message-Id:" header, if it already has one.

filterart [-debug] [configfile]
filterbatch [-debug] [configfile]

These programs read a single article or an "rnews" batch from "stdin" (respectively) and only print articles to "stdout" (in their respective formats) if the article headers do not match any of the patters cited in the configuration file, the format of which is described below.

If a configuration file is named on the command line (eg" "fred") it will be searched for in $SNARFCONF/filter ("$SNARFCONF/filter/fred") and then as an explicit path; if no such file is specified, the default ("$SNARFCONF/filter/default") is used.

This permits the removal of MAKE MONEY FAST spams, excessively crossposted articles, traffic sourced from particular users/sites, and oversize articles before the traffic is forwarded to another process.

forprep

This command copies from "stdin" to "stdout", indenting the output with a space character and a ">" sign.

gronkfiles [-debug]

This program reads a list of filenames, one per line, from "stdin", and presents their contents as a "rnews" batch on stdout. Files specified as relative pathnames are taken from "$SNARFSPOOL".

gronknntp [-debug] [-save [-tree]] hostname[:port] [config]

This program expects to find a list of newsgroups in the file "config" (default: $SNARFCONF/sites/hostname[:port]) and connects to the NNTP server on "hostname", where it tries to read new articles from each newsgroup, and present them as a rnews batch on "stdout".

If the "-save" option is specified, articles are saved to individual files under the directory:

$SNARFSAVE/the.group.name

If "-tree" is also specified, articles are saved under:

$SNARFSAVE/the/group/name

ifbytes command [arg ...]

This command reads "stdin" up to an EOF and stores the input in a file; if the file's size is non-zero, then "command" is executed reading its "stdin" from the file, and the file is then removed.

This command is chiefly used where "sendmail" (or similar) is at the end of a pipeline where it may (or may not) receive an empty input from "stdin"; when "ifbytes" is utilised in this case, "sendmail" will never actually be invoked, thereby preventing it from being confused.

ihaveart [-debug] [-genid] hostname[:port] [file]
postart [-debug] hostname[:port] [file]

These command read a USENET article from "file" or "stdin" and post it on "hostname" using NNTP's IHAVE or POST methods, respectively. Alternative NNTP port numbers or service names can be specified by appending ":port" to the hostname.

If "-genid" is specified with "ihaveart", and there is no "Message-Id:" header in the article, then a Message-Id: header will be created using the value of the "$SNARFHOST" variable as a hostname.

Important: unless the "-debug" switch is specified, these two programs will accept and silently discard empty articles, so that no messages are generated on "stderr" when articles are dropped by "filterart" or the "convX2Y" scripts (below), thereby presenting an empty input to these programs.

innbatch [innfeedname ...]

For interfacing with "INN"; assuming that the user has read/write permissions on the INN "out.going" directory and can invoke "ctlinnd", this script flushes the INN feed named "innfeedname", and invokes "gronkfiles" on the spooled articles to produce an rnews batch on "stdout".

For reference, the INN "newsfeeds" entry for such a feed would look like:

innfeedname[/exclude,exclude,...]::Tf:

- see the INN "newsfeeds" man page for details.

lock [cookie]
unlock [cookie]

These commands provide basic advisory locking functionality, if required. They are used by "runfeed" to prevent running of multiple instances of the same feed at the same time.

 
nntpmsgids [-debug] [-gronk] hostname[:port] [config]
gronkmsgids [-debug] hostname[:port] [pfcnt]

These two commands are complementary and when combined by the user (or when one invokes the other - see below) they provide similar functionality to "gronknntp", although users may find additional uses for each, individually.

The "nntpmsgids" is configured similarly to "gronknntp" (above) and expects to find a list of newsgroups in the file "config" (default filename as per "gronknntp") and connects to the NNTP server on "hostname", where it then produces a list of the Message-Ids of new articles in those groups, printing the list to "stdout".

The same "SNARFXPOSTMAX" functionality of "gronknntp" is duplicated in this command.

The "gronkmsgids" command reads a list of Message-Ids from "stdin", connects to the NNTP server on "hostname", retrieves the specified articles, and presents them on "stdout" as a "rnews" batch.

The argument "pfcnt" (if specified) should be a small integer (typically between 1 and 3) which specifies a prefetch quantity, viz: the number of article-fetch requests which should queued so that article retrievals occur asynchronously, and the network connection is used to its maximal advantage.

Combined with the asynchronous nature of the "nntpmsgids" command, this provides for exceptionally high throughput of article transferral, as there are few occasions where the link will ever be idle, eg: awaiting for the remote NNTP server to respond.

Such a feed could be implemented via:

    stream \
        : nntpmsgids nntpserver | gronkmsgids nntpserver 2 | rnews

...in the "feeds" file, after setting up "$SNARFCONF/sites/nntpserver" - but it should be noted that this method of retrieval is of slightly higher risk than a standard "gronknntp" feed, since the article-count maintaining program ("nntpmsgids") cannot determine the successful connection of the article fetcher ("gronkmsgids").

Moreover, if "gronkmsgids" fails to connect to your NNTP server for some reason and quits, you might lose articles which "nntpmsgids" will register as successfully retrieved (if it hasn't been killed by SIGPIPE).

It is for this reason that "nntpmsgids" has an extra option, "-gronk".

When invoked with the "-gronk" option, "nntpmsgids" will automatically launch "gronkmsgids" on the same host, with a "pfcnt" of two, and will pipe the list of message-ids directly to it, so that a feeds entry of:

    stream : nntpmsgids -gronk nntpserver | rnews

- is functionally identical to the one previously cited, with the added advantage that "nntpmsgids" can detect a failure in "gronkmsgids", leading to better error recovery. In short: this is the preferred way to pull a feed using this method, though users may find creative uses for "nntpmsgids" and "gronkmsgids" separately.

It should also be noted that "nntpmsgids" currently relies on the non-RFC977 "XHDR" command extension; if this is not available on your NNTP server, use "gronknntp" instead.

runbatch [-debug] "command"

This program reads an rnews batch from "stdin", and for each article that it finds in the input, it invokes "command" (which can be a single command, or a shell pipeline), feeding the article into the command via "stdin".

Obvious uses for this command include breaking open an rnews batch and reformatting and mailing the messages out to a maillist, via "conva2m" and "sendmail".

runfeed [-debug] [-nolock] [feedname ...]

"Runfeed" is the driver script that can be used for many functions in SnarfNews. The "runfeed" configuration file is "$SNARFCONF/feeds", and contains lines of the format:

feedname : puller [ : pusher ]

Where:

• "feedname" is a tag used to identify the command sequence
• "puller" is a command or shell pipeline
• "pusher" is another (optional) command or shell pipeline, separated from "puller" by a colon.

Entries may be broken over several lines using backslash "\" characters.

When a user invokes:

% runfeed feedname

...the "runfeed" program searches its config file for an entry with a matching tag. If the matching entry contains no "pusher" command, then the "puller" command is executed and "runfeed" continues to search for other entries with the same tag.

If a "pusher" command is defined, then the "puller" command is executed, with "stdout" redirected to a temporary file. When the "puller" finishes, if this file is NOT empty, then the "pusher" command is invoked, reading "stdin" from this file.

Thus, the user can (for instance) use "gronknntp" to read articles from a NNTP server by specifying it as a "puller", and then mail the resulting rnews batch to another site by specifying a "pusher" to encode and mail the batch, the complete entry being:

feedname \
	:gronknntp news.foo.co.uk \
	:batchencode | mail snarfnews@bar.co.uk

Whereas, if you don't need to worry about sending out empty batches of articles (eg: you are fetching articles for your local feed) - you might prefer not to specify a "pusher" and just use a pipe:

topup : gronknntp news.foo.co.uk | rnews -v

The "-nolock" option disables locking of each feed in turn, which otherwise prevents multiple instances of the same feed being run simultaneously. Specifying no feedname on the command line will try to run all feeds listed in the "feeds" file.

timeout interval command args ...

This program runs "command" for not longer than a specified "interval" of time (eg: "30s" = 30 seconds, "10m" = 10 minutes, "2h" = 2 hours) - after which time the process it killed using SIGTERM and then SIGKILL, if it has not already exited.

Miscellany

lib/snarf.pl

Socket library functions for NNTP access.

conf/feeds

Configuration file for "runfeed".

conf/filter/default

Configuration file for "filterart". This file contains lines:

    # maximum number of groups an article can be crossposted to and accepted
    # (see also SNARFXPOSTMAX in "snarf" for gronknntp config of same)
    xpost 9

    # max article size specified as nnn (bytes) or nnnK (Kb) or nnnM (Mb)
    size 256K

    # filter out common spams; format:
    #     header [egrep pattern matching a spam]
    header ^subject:.*money.*money.*
    header ^subject:.*make.*money.*fast
    header ^subject:.*make.*cash.*fast
    header ^subject:.*\$\$\$

extras/inn-subscribe

Weenie hack: reads a list of groupnames from "stdin" and invokes "ctlinnd" to create them. Saves typing it in from the command line every time.

secret/example

Example "keyfile" for use in batch encryption. DO NOT USE THIS FILE!

Instead, you should create your own "keyfile" by typing a long jumble of characters into a file, and transporting it via some secure means to the remote site.

People running a recent version of Linux (kernel rev >2.0), or any other OS equipped with a good "/dev/random" random number generator, may like to try:

% make asecret

- which should produce a string suitable for redirecting into a "keyfile" in the "secret" directory; users on other operating systems will have to find a different way to create a high-entropy random text string.

FAQs

    get-remote \
	: runsocks gronknntp news.remote.co.uk | rnews -v

    put-remote \
	: innbatch news.remote.co.uk \
	: runsocks batch2ihave news.remote.co.uk

snarf!:!*,group.name:Tp:/usr/local/snarf/snarf art2mail address %s