UnixWorld Online: Tutorial Article: No. 017

Procmail: A One-size-fits-all mail processor

By Paul Dunne.

March 29th, 1999

This article shows Unix system administrators how they can enhance their e-mail setup by installing procmail, a general-purpose mail processing program for all brands of Unix. I will use Linux as an example system, but what I say should apply to to other Unixes.

Introduction

Getting, Compiling, and Installing

Using procmail with Sendmail

The Basics of Recipes

Questions regarding this article should be directed to the author at paul@dunne.ie.eu.org

Introduction

Why procmail? Well, that's really a question this whole article is devoted to answering. But, in essence, you should consider using it because it is drop-in replacement for /bin/mail that does more, much more.

Some other appealing features are its simplicity -- just one executable and one config file -- and its performance. Written in C, it has a low impact on system resources -- a note accompanging the source distribution claims "1.4 times faster than the average /bin/mail in user-cpu time."

The system requirements are basic:

Any Unix-alike (or POSIX compliant) system.
A compatible MTA (Mail Transport Agent) such as Sendmail, ZMailer, smail, MMDF, mailsurr -- in short, any mailer that can process RFC822- compliant mail.

Getting, Compiling and Installing

The latest version of the procamil sources is available from the Procmail ftp site.

Configuration is a matter of making changes to two files, Makefile and config.h, then running make install.

The Makefile has only a few, self-explanatory variables which you may like to change. config.h is rather more interesting.

This file is in two parts: some stuff that customises your installation, and some stuff you shouldn't need to change unless you're hacking procmail. Here is a list of the C defines which you may need to change, together with thier default vaules, and an indication of whether they are defined at all by default.

sMAILBOX_SEPARATOR "\1\1\1\1\n"
eMAILBOX_SEPARATOR "\1\1\1\1\n": Starting and ending separators for mail. Define if your system uses nonstandard mail separators. Default: undefined.
KEEPENV {"TZ",0}: Any environment variable in KEEPENV will not be unset on startup. Default: defined.
DEFPATH "PATH=$HOME/bin:/bin:/usr/bin": Default path. Define is you do not want the one in PRESTENV below. Default: undefined.
PRESTENV {"IFS","ENV","PWD",DEFPATH,0}: Any environment variable in PRESTENV that is not assigned to in PRESTENV will be unset; any environment variable which is assigned to will be given that value. Default: defined.
TRUSTED_IDS {"root","daemon","uucp","mail","x400","network",\ "list","slist","lists","news",0}: Every user and group in the list is allowed to use the -f option. If the list is empty (i.e. just a terminating 0), everyone can use it. If unauthorised users use the -f option, an extra >From_ field will be added in the header. Default: defined.
NO_fcntl_LOCK
NO_lockf_LOCK
NO_flock_LOCK: Define any or all of these if you do not want to use that style of kernel locking. Default: all three undefined.
RESTRICT_EXEC: Don't allow everyone to fork programs. Default: undefined.
NO_NFS_ATIME_HACK: Use if you're definitely not using NFS mounted filesystems and don't want procmail to sleep for 1 sec. before writing a regular mailbox. Under heavy load, procmail will not do this. Default: undefined.
DEFsendmail "/bin/mail": Use, and chnage if necessary, if the autoconfigured default SENDMAIL is not suitable. Default: undefined.
PROCMAILRC "$HOME/.procmailrc": Default rcfile for every recipient; if this file is not found, delivery will proceed as normal to the default system mailbox. Default: defined.
ETCRC "/etc/procmailrc": Global procmailrc startup file (will only be read if procmail is started with no rcfile on the command line). Default: defined.
ETCRCS "/etc/procmailrcs/": Trusted path prefix for rcfiles which will be executed with the UID of the owner of the rcfile (this only happens if procmail is called with the -m option, without variable assignments on the command line) Default: defined.
console "/dev/console": This sould be defined if you want procmail to use the console (or any other terminal or file -- replace `/dev/console' with the file you want to use) to print any error messages that could not be written to the logfile. This is intended for use in debugging; if you have trouble creating a "logfile"; or if you suspect that the trouble starts before procmail can interpret any rcfile or arguments. Default: undefined.

And, as the comment says, only edit below this line if you *think* you know what you are doing!

Once satsified with the state of the Makfile and config.h, procmail is compiled and installed in once fell swoop with

$ make install

(If not compiling as root, you will have to seperate the steps, with make, followed by su and make install).

It is recommended that procmail be installed suid root. You should note the following:

When in explicit delivery mode e.g. when called from sendmail, procmail will always change uid and gid to the recipient's defaults as soon as it starts reading their .procmailrc file.

When not in explicit delivery mode, e.g. when called from the recipients's .forward file, procmail will always change UID and GID to the real UID and GID of the invoker, thus losing any SUID and SGID privileges.

These two precautions should effectively eliminate any security holes, since procmail will always have the UID of the user hwose commands it is executing.

Using procmail with sendmail

There are a number of ways procmail can be used with sendmail. It can be used as a local mail delivery agent, or as a general-purpose "prog" mailer. If you are using a recent version of sendmail, you can use the m4 macro files to greatly simplify sendmail configuration for whichever use of procamil you desire. In this case, there are four variables and two features that relate to procamil. The features are:

procmail
local_procmail

and the variables are (with default value in brackets):

PROCMAIL_MAILER_PATH	[/usr/local/bin/procmail] The path to the procmail
			program. This is also used by FEATURE(local_procmail).
PROCMAIL_MAILER_FLAGS	[SPhnu9] Flags added to Procmail mailer.  Flags
			``DFM'' are always set. This is NOT used by
			FEATURE(local_procmail); tweak LOCAL_MAILER_FLAGS
			instead.
PROCMAIL_MAILER_ARGS	[procmail -Y -m $h $f $u] The arguments passed to
			the Procmail mailer. This is NOT used by
			FEATURE(local_procmail); tweak LOCAL_MAILER_ARGS
			instead.
PROCMAIL_MAILER_MAX	[undefined] If set, the maximum size message that
			will be accepted by the procmail mailer.

Local mailer

The simplest way in which procmail can be used is as a drop-in replacement for the local mailer. This entails mail delivry as normal, plus extra processing if a .procmailrc file is present in the home directory of the recipient. To do this, follow these steps:

If you are building a sendmail config file from scratch, or you maintain your sendmail config using the m4 macros, then simply define local_procmail in your .mc file, like so:

FEATURE(local_procmail)

and rebuild sendmail.cf. Note that this does not use PROCMAIL_MAILER_FLAGS or PROCMAIL_MAILER_ARGS for the local mailer; you have tweak LOCAL_MAILER_FLAGS and LOCAL_MAILER_ARGS instead.

If you maintain sendmail.cf directly, then edit the local mailer definition. If you're using a sendmail 8.6.x or older, use this:

Mlocal, P=/usr/local/bin/procmail, F=lsSDFMhPfn, S=10, R=20,
	A=procmail -Y -a $h -d $u

If you're using sendmail 8.7 or newer, then use this:

Mlocal, P=/usr/bin/procmail, F=SAw5:|/@glDFMPhsfn, S=10/30, R=20/40,
	T=DNS/RFC822/X-Unix,
	A=procmail -Y -a $h -d $u

Procmail as a local mailer needs root privileges on startup. Further, on some systems, procmail may need to be sgid daemon or mail. Check this by looking at the current mail delivery agent, and copying its permissions. You can also use "make recommend," which will suggest appropriate permissions for your system. The optional lockfile program may also need the same group permissions, to allow it to create and unlink lockfiles in the mail spool directory.

Changing the mail spool directory to the user's home directory

You may wish to store mail in the recipient's home directory, rather than in the traditional /usr/spool/mail. This has some obvious advantages:

Mail is subject to the user's quota limitations.
there is often more room on the home partition(s) than on /usr/mail.

The quota limitations also apply to /usr/spool/mail or /usr/mail if procmail does the delivery. These quota limitations often do not work with the regular /bin/mail since that usually writes the mailbox with root permissions (eluding the quota restrictions).

if you intend to install procmail as the local mailer, and you want mail to be delivered to, say, $HOME/.mail by default, then:

Uncomment, and change if necessary, SYSTEM_MBOX in config.h, and rebuild the prgroam. Procmail will now deliver to this location by default.

Make sure that all users set thier MAIL environment variable to the same value. If a mail client in use at your site does not use MAIL, then either fix the source of that proram, or create symbolic links in /var/mail.

prog mailer

Procamil can also be used a a "prog" mailer. This can then be used a general mail filter. Use the "procmail" feature if configuring sendmail using the m4 macros. The entry in sendmail.cf reads as follows:

Mprocmail, 	P=/usr/local//bin/procmail, F=mSDFMhun, S=11, R=21,
		A=procmail -m $h $g $u

Mailertables

Sendmail and procmail in combination provide a simple way to forward all mail for a given domain to a single person, using the sendmail mailertable feature. This involves defining procmail as a "prog" mailer.

mailertable:

example.com	procmail:/etc/procmailrcs/example.com

/etc/procmailrcs/example.com:

:0	# forward mail for example.com
! -oi -f $1 person@other.host

This would send any mail for (anyone)@example.com to to person@other.host. Within the procmail script, $1 is the name of the sender and $2 is the name of the recipient. Note that if you use this with FEATURE(local_procmail), list FEATURE first.

Basic Recipes

The rcfile, whether the global /etc/procmailrc or each user's $HOME/.procmailrc, can have two types of entry: environment variable assignments, and recipes.

The core of procmail is the idea of a "recipe." This is a rule in the procamil rc file (/etc/procmailrc or $HOME/.procmailrc). Each recipe has three parts: header, pattern, action. The header identifies this as a recipe, and declares options. The pattern, as the name implies, matches a regular expression against the message header, or body, or both. The action does something if the regular expression matches.

:0 [flags] [ : [locallockfile] ]
<zero or more conditions (one per line)>
<exactly one action line>

A second colon, after the 0 on the first line, tells procmail to use locking. Locking is important; you may have two procamil processes running at the same time, both trying to deliver mail to the same location. A second colon alone tells procmail; followed by a filename, it tells procmail to use that file as the lock file. The file will be created, and unlinked only after processing is complete. Thus, if a second instance of procmail should happen along in the meantime, it will see that the file already exists, and wait.

Flags can be any combination of the following (though of course some combinations will not make much sense):

H match the pattern against the header (default)
B match the pattern against the body
D distinguish case in pattern matches (default is ignore case)
A on a match, only follow this recipe if the last recipe without the A flag matched as well
a as "A," and additionally the last recipe must have successfully completed
E The inverse of A, that is only follow this recipe if the immediately preceding recipe was not executed.
e Only follow this recipe if the immediately preceding recipe failed -- that is, the action line was executed and returned an error code.
h Feed the header to the action line (default).
b Feed the body to the action line (default).
f Treat the action line pipe as a filter.
c Generate a carbon copy of this mail. Processing of the mail continues.
w Wait for the action line filter or program to finish, and check its exitcode (usually ignored). If the filter is unsuccessful, then the mail is not considered to have been processed.
W As "w," supressing any "program failure" message.
i Ignore any write errors, as one might receive from an early closed pipe, for instance.
r Raw mode: just write the mail out "as is." Without this, procmail will make sure that the message ends with an empty line.

The condition mostly uses standard egrep regexp syntax, with a few additions. these are:

! invert the condition.
$ evaluate what follows using the sh(1) substitution rules for matter inside double quotes, skip leading whitespace, then reparse it.
? Use the exitcode of the specified program.
< Check whether the total length of the message is less than the specified number of bytes.
> Check whether the total length of the message is greater than the specified number of bytes.
variable ?? match the remainder of the condition against the value of this environment variable.
\ Escape, to quote any of the special characters above at the start of a line.

The action line

! Forwards the message to the specified e-mail addresses.

| Starts the specified program. If the command given contains shell metacharacters, a shell will be started to run it in. Variable assignment before this symbol will assign the output of the program to that variable. A pipe symbol alone, with no program, will send the e-mail to stdout.

{ (followed by whitespace) marks the start of a nesting block. Everything until a closing "}" will depend on the conditions specified for this recipe. For example:

:0
*
mail

This is a very simple recipe that stores mail in a file called "mail." As you can see, the first line of a recipe begins with ":," and the second line with "*." A recipe can have more than one pattern line, but only one header line and only one action line. Don't confuse the star at the beginning of the pattern line with a wildcard, by the way. The default matching action is "match," so since we haven't got any pattern, this recipe will always suceed.

When writing procmail recipes, it is best to test them first. You can do this easily on the command line. Simply invoke procmail, giving the name of the rc file you wish to test as the first argument, and redirect a sample e-mail message to be procmail's standard input, like so:

$ procmail ./test.rc < ./sample.msg

Where sample.msg is of the form

From: somebody@example.com
To: me@example.com

Note the blank line between header and body. Replace the To: address with your own.

Let's test our example recipe.

$ procmail test.rc < test.msg
procmail: [991] Mon Mar  1 17:05:45 1999
procmail: Match on ".*"
procmail: Assigning "LASTFOLDER=mail"
procmail: Opening "mail"
procmail: Acquiring kernel-lock
Folder: mail
$ cat mail
From: somebody@example.com
To: paul

Procmail expects exactly one mail message to be presented to it on its stdin. Procmail uses special environment variables, which you can change in the rcfile. Here is a list of the more important ones.

LOGABSTRACT Should procmail send a one-line summary to the logfile, consisting of the From and Subject fields, where the message when, and its size in bytes.
LOGFILE Where to send error, diagnostic and LOGABTRACT messages.
SHELL what shell to use -- default /bin/sh
VERBOSE turns on extended diagnostics -- default off.

Recipes in Action

Here is a simple /etc/procmailrc as an example. This does little more than log all arriving mail in what may be a more convenient and easily customizable format than sendmail, do some simple MIME conversion (recipe for this taken from procamilex(5)), and implement a very simple "auto-responder," to reply with information to people who write to certain addresses. This is easily changed so that, for example, people writing to "abuse" get a formula message about how concerned you are about spam ;-), with their message then being snet on to the appropriate person at your site. Normally, we don't want to do much to people's mail at a global level, anyway.


# set some environment variables
SHELL=/bin/sh
LOGFILE=/var/log/procmail
LOGABSTRACT=all
VERBOSE=off

# convert MIME types
:0
* ^Content-Type: *text/plain
{
  :0 fbw
  * ^Content-Transfer-Encoding: *quoted-printable
  | $MIMENCODE -u -q
 
     :0 Afhw
     | formail -I "Content-Transfer-Encoding: 8bit"
 
  :0 fbw    
  * ^Content-Transfer-Encoding: *base64
  | $MIMENCODE -u -b
 
     :0 Afhw   
     | formail -I "Content-Transfer-Encoding: 8bit"
}

# auto-responder
:0 H
* $ ! ^$MYXLOOP
* ! ^FROM_DAEMON
* ^Subject: *(get|send|mail|gimme)\/( +(info|\
			articles|\
			pgp|((public )?key)|\
			finger|\
			resume|\
			www.*|\
			geek code))+
{

	GETTING=$MATCH		# What got me here?
	MYFROM="From: example.com's Magic Mail Responder "
	SUBJECT=`formail -xSubject`
	BOUNDARY=`formail -x'Message-Id:' | sed 's/[<>@ ]//g' | cut -c1-65`

	# log this in its own logfile
	LOGFILE=
	AUTOREPLYLOG=/var/log/autoreply-log

  :0 ch:
  $AUTOREPLYLOG

	# This is how I get real headers.
	:0 fhW
	| formail -rY -I"$MYFROM" \
		-I"$MYXLOOP" 

	BOUNDARY=`echo '' ; echo '' ; echo --$BOUNDARY`

	:0 fbW
	| echo '' ; \
	echo $BOUNDARY ;\
	echo '' ;\
	echo 'You asked for it, you got it.';\
	echo '';

	# help
	:0 fWb
	* GETTING ?? help
	|  cat - ;  echo $BOUNDARY ; \
	   echo '' ;  cat /usr/local/etc/company-info; echo ''

	:0
	| ( cat - ; echo "$BOUNDARY--" ) | $SENDMAIL -t -oi 

}
LOGABSTRACT=$ABSTRACTVAL

A User's Eye View

We have seen how useful procmail can be as a local mail delivery agent from a system administrator's point of view. It is worth while taking a quick look at the extra benefits your users can gain when you install the program

First things first: Every user that wants to use procmail to process his or her mail should have a .procmailrc file in his or her HOME directory. You will recall that without this, mail will simply be delivered to the default mail spool file as defined in config.h.

Some simple recipes you might want to put in a "skeleton" .procmailrc for new users:

New Mail Notification

Here is an example recipe to notify a user when they receive new mail.

LOGABSTRACT=no
:0 c
* 
| /usr/lib/mh/rcvtty -bell
LOGABSTRACT=all

This recipe uses the "c" option so that mail it handles will not be regarded as delivered. Thus, we can hand the mail on to further recipes. If we didn't have this here, the mail would effectively disappear -- not what we want at all!

The blank expression is a catch-all -- we want to match any and every message with this.

Finally, the action line uses a program in the MH distribution, that notifies the user that they have new mail. It does this by ringing the terminal bell (-bell -- without this, it would be silent), and writing a one-line message to the screen at which the user is logged on, giving summary details of the message -- who it's from, the subject.

The whole thing is wrapped in two variable assignements, which switch logging off, and then on again. We don't want to log the uninteresting results of this recipe, unless we are debugging.

Handling Mailing Lists

Here is an example recipe for how a user might wish to deal with mailing lists, using the MH mail user agent (MUA).

#procmail-digest
:0 w:$MAIL/lists/procmail/.mh_sequences.lock
* ^From: procmail-d-request@lists.utah.edu
$MAIL/lists/procmail/. $MAIL/inbox/.

This recipe handles mail from the procmail-digest mailing list, a digest version of the procamil mailing list.

It uses the "w" option to write to a lock file, so that if more than one process tries to deliver mail to this folder, they won¹t mess each other up.

We match against the From header.

Finally, we deliver the mail to two places: the user's default folder for incoming mail, inbox, where he can read it at his leisure; and the archive folder for this list, so then when the user is through with the copy in his inbox, he can simply delete it, knowing that a copy has been filed in the right place. Much more convenient than having to do it by hand.

Resources

Procmail comes with a good set of man pages, which are also available on the WWW:

There is a procmail mailing list. To subscribe, send mail with a Subject line of "subscribe" to
procmail-request@informatik.rwth-aachen.de
for the full list, or, for the digest version, to
procmail-d-request@informatik.rwth-aachen.de.

Some other WWW resources include:

Conclusion

In this article, we've looked at how procmail can enhance local mail handling on a Unix server. There is much more about procamil that we haven't covered, including use of its companion programs formail(1) and lockfile(1). The resources given above should be a help in exploring procamil in more detail.