WORKSHOPS

Information Dissemination With Listservers

by Christopher M. Sedore

Electronic mailing lists are an extremely popular form of communication in organizations and across the Internet. Electronic mailing lists fill the communication void between plain electronic mail, UseNet news and the World Wide Web (WWW).

One factor holding mailing lists in place is their privacy. Unlike UseNet news, controlling the users allowed to join or send to a listserve is easy. Another factor is a list's proactive nature--WWW resources require someone to run their browser and go to the pages while mailing lists drop a copy of each message in every member's mailbox, forcing them to read or discard it.

Mailing lists ca n be created in a few different ways. At one end of the spectrum is building simple distribution lists on a Unix system by adding entries to the alias table for the Unix mail system, SendMail. This provides a simple and effective way of providing distribution to multiple users from a single address. The simplicity of this approach is also its downfall--a privileged user on the Unix system must perform all the list maintenance. Each time you make a change, you are manually adjusting an important configuration file in your mail system. Similar facilities are available in most other mail systems, and they suffer from the same drawbacks: high maintenance overhead and ease of making errors during updates.

The other end of the spectrum is commercial listserve software. This software is built for large distribution lists, with additional intelligence to solve some of the probl ems encountered by simpler systems. Associated with these systems are better management, more rob ustness and flexible archive func tionality. Lsoft's ListServ software, for instance, is the industrial strength mailing list software.

From BITNet to the Internet ListServ evolved on BITNet and ran on an IBM mainframe. BITNet's store-and-forward architecture, which is built on slow links, forced the software to address mail delivery in a special fashion. As lists grew, handling thousands of messages individually was impractical. The software needed to have the intelligence to send one copy of a message to a site or region, and have another IBM mainframe running ListServ software distribute it through the region. These large machines continue to handle some of the large mailing lists on the Internet using some of the optimizations developed for BITNet. This software is now available for various Unix platforms, as well as Windows NT.

The Internet brought the end of store-and-forward networking, allowing interconnectivity from any host to any other host. This, combined with faster networks, eased some of the restr ictions that existed on BITNet, but gave birth to others. Still, list servers can be an inexpensive way to distribute information to large numbers of people, but setting them up is considered difficult. To demonstrate what you might be in for, we'll show you how to set up Majordomo, a package squarely in the middle between simple distribution lists and the commercial ListServ package. The Majordomo package, as well as related documentation, including the FAQ, is available via the Web at http://www.greatcircle.com/majordomo .

The Majordomo Collection Majordomo is a collection of Perl scripts and a single C source code file (a wrapper for executing Perl scripts as a privileged user). This collection provides excellent portability: It will run on any major Unix flavor that supports Perl and normal setup semantics (testing for this workshop was done on FreeBSD, a BSD 4.4-based Unix for Intel PCs ). Majordomo supports a broad set of functions, including simple archiving of mailing list traffic, e-mail-based list management tools, automated user subscriptions, digesting and moderated lists. It uses the native Unix mail agent, SendMail, to handle all mail transactions.

The system is built from three Perl scripts: majordomo, resend and archive2.pl. At the heart of the system is the script majordomo. It handles all incoming requests and performs the manipulations necessary to subscribe and unsubscribe users from mailing lists. The second script, resend, rewrites mail headers on outgoing messages, resetting the "Reply-to:" address in mail messages back to the list address and adding other header lines (that is, list description X-headers) to messages. Lastly, archive2.pl keeps archives of mail messages for lists. It is usually included as a recipient of outgoing mail to the list. Although knowledge of Perl is not necessary to install and run the system, you must modify some script p aths if Perl or SendMail are not in their default locations.

Besides the three Perl scripts, an included wrapper program allows the scripts to run as a privileged user (typically "root" or "bin"). The wrapper forces the execution of the application from a directory specified at compile-time in the Makefile, and ensures that the environment is properly and securely set up for the scripts to run.

Majordomo works by leveraging the capabilities of SendMail, which builds into its alias facilities the ability to manage simple lists. You can create lists for SendMail by simply creating a file with a list of the destination e-mail addresses and telling SendMail to include it. This is exactly what Majordomo does, except that rather than manual updates, as described for the alias table earlier, Majordomo handles updates for you in response to commands sent via e-mail. This frees you from having a system administrator bothered by adds and deletes from your mailing lists.

It's Not All Hands Off Mailing list creation and deletion still requires the attention of a system administrator. Creating and deleting lists involves editing the system alias table and creating or removing a set of configuration files for each list. It's basically a copy-and-modify operation, but it requires significant attention to detail when managing more than a few lists and list changes. List modifications (that is, if you wanted to change the outgoing headers on your list) also require editing the alias table. This can be time consuming.

Majordomo installation is relatively easy for anyone with a Unix systems administration background. Our testing did include a few roadblocks. The first--and most frustrating--is the fact that Majordomo v1.93 requires Perl 5.001m, despite the documentation's claim that it will work with Perl 4.036 and some Perl 5.001 implementations. We also were forced to fix the SendMail path in the resend script, since it was not set to the defaul t location. Beyond these issues, Majordomo's documentation doe s a good job of describing the installation process. We'd recommend retrieving the majordomo FAQ (frequently asked questions) before starting the installation, since it has some additional hints on Majordomo installation and troubleshooting.

An Internet Administrative Assistant Majordomo does have limits. As lists grow large, or contain many members whose mail receivers have poor connectivity, SendMail may have difficulty keeping up with mail traffic. This results primarily from the mail delivery methodology of the Internet. It's easiest to think of it this way: If you have a list of 500 people, and you need to give each a message, you could ask your administrative assistant (SendMail) to call each of the 500 people and give them the message. As your administrative assistant calls, he or she finds that some recipients don't answer (the remote host is down), so waiting may occur (the time-out ) and the intended recipient is placed back on the list to be called again later. Other recipients may answer, but put the assistant on hold while delivering the message (poor connectivity), increasing the amount of time necessary to deliver the message.

Although this can be bad on large lists, the next suite of problems begins when you add more messages to the administrative assistant's workload. Your administrative assistant has ways of managing additional messages, including distributing the work to others, using alternate communication channels (voice mail or electronic mailing lists). SendMail, on the other hand, has limited intelligence and resources to apply to delivery problems. Its primary tactic is to sort mail by delivery MX (mail exchanger) while maintaining a connection cache for regularly accessed hosts. This implementation avoids spending inordinate amounts of time connecting and disconnecting.

You have the option of hiring more assistants, or in this case, spawning additional SendMail daemons. This can quickly run into scaling problems. For large lists, you simply can' t start enough daemons on a machine of reasonable size to deal with large numbers of messages. The growth factor is significant: A 500-member list at 10 messages per day is 5,000 messages. While 5,000 messages is not really that many, if they all are heading for sites with poor connectivity or if you're sending your latest network diagram as a MIME attachment, it may be a problem. With only 1,440 minutes in a day, if each message, including your attachment, took one minute, you would not even get through the queue.

MIME Enables Attachments The proliferation of MIME has encouraged more people to attach documents to their e-mail messages: We recall watching the space on our NetWare servers drop 10 MB in a five-minute period as 100 copies of a 100-KB document were delivered. This may be fine for local lists (although perhaps not so great for your mail server), but consider the additional overhead if it were a non-lo cal list.

Solutions to scaling simpler distribution systems exist. The primary one is using local mailing lists as an exploder. These can be simple aliases for small groups, or can be handled with another Majordomo installation. Unfortunately, these solutions require that the users cooperate by subscribing to the local list instead of the main distribution point. Other possible solutions include moderating and digesting the list. Moderating a list requires that you appoint someone to preview each article and forward it to the list if its content is appropriate, whereas digesting collects articles for a period of time (usually a day) and then forwards the collected messages in a single message. Combining these two methods can prove very effective at improving distribution as well as content. Majordomo supports both functions.

Large mailing lists also require a fair amount of management and attention. As lists grow beyond a single organization, accounts are often deleted without users unsubscribing first, leading to bounced messages. In really ugly cases, these bou nced messages can be fed back into the list and fill everyone's mailbox with tens or hundreds of rejection messages.

Generally, lists of this size should be migrated to other communication forums like UseNet news or the Web. Although, as stated earlier, some unique properties continue to make mailing lists attractive for applications where some privacy and proactive delivery are desirable.

Christopher Sedore is manager of network services for The Maxwell School of Citizenship and Public Affairs at Syracuse University. His e-mail address iscmsedore@maxwell.syr.edu.

February 13, 1996