Generally acknowledged as being the 'killer-app' of the Internet, e-mail is more frequently used than even the Web. As more and more people rush online, e-mail will become, even more, the ubiquitous means of communication between individuals, for all sorts of purposes, in many different spheres of activity.

E-mail has achieved a phenomenally wide user base over the last few years, which is due in part to the power and design of the Internet mail protocols that we will be examining later in this chapter. Programming with those e-mail protocols is, however, a subject that has received surprisingly little attention from the mainstream programming textbooks.

One of the secrets of e-mail's amazing success with the general public is that it hides the underlying complexities and details of transmitting and retrieving electronic mail from the end user, allowing for low maintenance and ease of use. Mobile phones and PDAs have also been tremendously successful because, like e-mail, they provide a user-friendly interface to a powerful communication paradigm. These wireless consumer devices are likely to be far more quickly and widely adopted by the largely non-technical general public in the next few years, overtaking the more powerful and complex personal computer.

The market penetration of WAP-enabled mobile phone technology will bring with it greater demand for more flexible, and powerful, mobile computing applications. Wireless e-mail functionality will be at the core of this revolution.

E-mail and WAP are fast becoming the most demanded combinations of technology by both corporations and general consumers.

Within today's e-commerce computing systems, the ability to exchange messages is an important feature, and one which will be demanded far more frequently as the functionality provided by e-mail becomes a de-facto standard, replacing the outdated fax technology for most businesses, and being fully integrated with corporate voice mail systems.

In fact, the ultimate goal for messaging technology for many organizations is a universal inbox in which voice, fax, and email can be viewed in any format by a mobile communicator device. For example voice mail, email and faxes should be viewable as email, audible like a voice mail, or sent to a nearby fax machine or printer.

In this chapter we will:

• Review the history of e-mail and the current e-mail protocol standards
• Look at Sun's JavaMail API
• Build server-side Java programs to deliver e-mail messaging features for WAP-enabled devices
• Briefly examine how to use CDO and ASP to incorporate e-mail into web applications on the Microsoft platform

Introduction to E-Mail

E-mail is an asynchronous message exchange technology. This simply means that when you send an e-mail message the recipient(s) does not have to be available at that instant to receive the mail, but may collect the message at his or her own leisure.

E-mail was one of the first applications to be used on the Internet and has shown a remarkable amount of tenacity. The protocols used to transmit and deliver e-mail have been evolving and changing over the years, and we have seen a wide variety of proprietary protocols come and go. Most of these proprietary solutions are now either obsolete or have been adapted to the open standards adopted on the Internet at large.

The idea of proprietary e-mail systems is no longer feasible in the Internet computing world - systems must interconnect to benefit from the huge installed base of Personal Computers, Macs and workstations, interactive TVs and mobile devices, linked to the Internet.

The History of Internet E-Mail

The ARPANET (Advanced Research Projects Agency Network) was created in 1969 as an experimental project to enable communication between participants in the DARPA (Defense Advanced Research Projects Agency) community. Ray Tomlinson wrote SNDMSG, the first ARPANET e-mail system, in 1972, and e-mail protocols and systems have snowballed since then.

To gain an idea of the worldwide adoption of this technology, here are some figures detailing e-mail usage (taken from NUA Internet Surveys):

• In the early 1990s, there were only 15 million e-mail accounts in the world
• There were 569 million e-mail accounts globally at year-end 1999; this figure is up 83% on the previous year
• It is predicted that there will be in excess of one billion e-mail accounts worldwide by 2002

The International Standards Organization (ISO) spent many years working on the vast and complete X.400 protocol as the de-facto standard for electronic mail.

However, whilst waiting for the final published specification, many vendors developed proprietary e-mail systems that achieved a wide deployment. The ISO e-mail standard, along with X.500, its sister standard for directory services, was simply released too late to achieve market dominance. The PC revolution was in full swing, and other, less expensive implementations such as MS Mail, Lotus Notes and cc:Mail had achieved a critical market share. Despite the vast reach of the ISO and the comprehensiveness of the enterprise (the brief was to design a complete mail specification), the standard was unable to displace the mass of e-mail systems already firmly established.

Acceptance of X.400 was further hampered by the fact that the ISO had missed some fairly vital pieces of functionality, such as the ability to asynchronously access mail messages without a permanent connection to the Internet in a way we will discuss later using the Post Office Protocol (POP3).

Nevertheless, X.400 may see some sort of renaissance as a mail backbone to transfer mail messages between mail servers, and is actually being used as a standard mail backbone protocol by several of the major vendors, including Lotus, Microsoft, IBM, and HP.

X.400 does provide:

• Good support for Binary Large Objects (BLOBS)
• Support for Electronic Data Interchange (EDI)
• Security via X.509 certificates
• Well designed connectivity of mail functionality with the X.500 Directory Service specification

We are now seeing the emergence of devices that integrate the more traditional capabilities of e-mail with the telephony features available to mobile phones, and other wireless devices. These devices are able to leverage the functionality, and familiarity, of both e-mail and wireless technologies. They are also truly portable, unlike most mail-enabled devices that have been used previously.

Short Messaging Service (SMS)

Short Messaging Service (SMS) messages, currently available on most modern mobile phones, have now reached over one billion messages exchanged a month in the European market alone, despite relatively light marketing of the feature by network operators and handset manufacturers.

However, sending an SMS message is unwieldy; you can only send messages in 160 character chunks of text. The editing of SMS messages is usually a cumbersome and laborious process and the user typically receives no warning when the character limit is about to be reached. Furthermore, to use SMS you need to know the mobile number of the person you wish to contact.

What E-Mail and WAP Can Offer

The popularity achieved by the very limited SMS technology indicates that demand for messaging over mobile phones certainly exists, and giving mobile phones all the functionality of e-mail seems to be the next logical step.

E-mail is a substantially more advanced technology than SMS, even if it is only used for simple SMS-like text messages. Message recipients are not limited in how they receive the message when using e-mail. Rather than only being able to access the message from a single mobile phone, the user can choose to access it from whatever client e-mail software he or she prefers, whether that is another WAP phone, a home PC, laptop, or even a UNIX workstation. E-mail, unlike SMS, allows for the recipient to have an address that is more like 'natural-language' than a phone number, and is thus easier to remember. Furthermore, e-mail provides the ability to mail 'group' addresses; for example all@wapbook.org. As we will see later in the chapter, e-mail also has substantial multi-media functionality, and can use a variety of security protocols.

WAP devices and e-mail capabilities seem to be an ideal technological fit since they allow for a useful synergy of personal communication technology: delivering the convenience of portability from mobile phones, whilst allowing instant access to e-mail, providing asynchronous access to written messages.

It is interesting to supplement the figures on e-mail usage listed above, with some corresponding figures on mobile and WAP usage, in order to assess the scale of the potential market that WAP e-mail functionality may reach (source: Durlacher Research Ltd):

• There are currently 300 million mobile subscribers, growing at 50% per annum (PC growth globally is now only about 20% p.a. and falling)
• WAP penetration into the mobile phone market is predicted to be 8% in 2000, 22% in 2001, 50% in 2002 and 85% in 2003
• It is predicted that the m-commerce market in Europe will be worth 5 billion euros during 2000, rising to over 20 billion euros by 2003. E-mail will play a vital part in this boom.
• By 2003, over 50% of Internet access will be by non-PC devices (Meta Group)
• By 2005, 1 billion mobile devices will be used worldwide (Gartner Group)

Key Elements of E-Mail

To help us understand the issues we will discuss later in the chapter concerning JavaMail, CDO and programming e-mail systems, we must first explore the key elements of an e-mail system.

There are essentially two types of protocol used in the e-mail process: transport protocols, and storage and retrieval protocols. We will look at these protocols and the overall mail process in some detail, before moving onto programming e-mail in the next section.

The Mail Process

The sender of an e-mail message uses a program to create and send mail; this program is known as the Mail User Agent (MUA). Once created, the message must be moved to the recipient's mail server over a transport medium - the Internet or a company's intranet for example - using a Mail Transfer Agent (MTA). The message is then delivered to the recipient using a Mail Delivery Agent (MDA) and stored until the recipient chooses to read it using his or her MUA.

So, the most important mechanisms in the mail transport system are:

• Mail User Agent (MUA) - A program used to create and receive mail messages
• Mail Transfer Agent (MTA) - The means by which mail messages are transferred between machines over the Internet
• Mail Delivery Agent (MDA) - The mechanism that delivers the mail message to the recipient's mailbox when mail is delivered via an MTA to the mail server

To understand how to code applications using e-mail functionality, it is unfortunately necessary to understand the large number of acronyms that come with the territory! Some of these acronyms are associated with e-mail protocols, which have evolved into sophisticated messaging tools. Thus, we will now take a look at the following questions:

• What are RFC 822 and MIME?
• What are S/MIME and OpenPGP?
• What is SMTP/E-SMTP, how does it work?
• What are POP and IMAP?
• How does LDAP fit into mail systems?
• How does ACAP fit into mail systems?

Mail format and transport protocols

In this section we will look at the mail protocols associated with formatting and transportation of messages. This will in effect answer the first three questions asked above.

On the Internet, Mail User Agents (MUAs) typically send messages in a MIME encoded format to Mail Transfer Agents (MTAs), which transport messages using SMTP/ESMTP protocols.

RFC 822

In the past, e-mails were sent in a standard format, specified in RFC (Request For Comment) 822, entitled 'Standard for the format of ARPA Internet text messages'. This encoded the mail as plain text in a US-ASCII 7-bit character set format with no multi-part structure to the message body. The need for the inclusion of other language's character sets, and multi-media attachments led to a need for a more complex message structure, which was solved by the creation of MIME.

MIME

MIME (Multipurpose Internet Mail Extensions) defines theı necessary message structure (see RFC 2045-2049) needed to work with different 8-bit character sets and multi-part messages. MIME was constructed as an extension to RFC 822, allowing older MUAs to continue to work, ignoring the new features, formats and extensions. MIME specifies a Content-Type header that can be used to specify a character set or media type for the e-mail message, for example:

                   Content-Type: text/plain; charset=us-ascii

In a typical e-mail message, you may see several attachments indicated by MIME type boundaries. For example the following e-mail header snippet indicates the email it came from contains a simple text section, followed by a Microsoft Word document (Content-Type: application/msword) as an attachment:

                   ------_=_NextPart_000_01BFA0AF.11763546

                   Content-Type: text/plain;

                   ıııı charset="iso-8859-1"

                    


                   ------_=_NextPart_000_01BFA0AF.11763546

                   Content-Type: application/msword;

                   ıııı name="Speaker FAQ.doc"

                   Content-Transfer-Encoding: base64

                   Content-Disposition: attachment;

                   ıııı filename="Speaker FAQ.doc"

                  

                   ------_=_NextPart_000_01BFA0AF.11763546--
           

End-to-End Security

Security services can be added to each step of the communication path, for example by using Transaction Layer Security (TLS), also known as Secure Socket Layer (SSL).

Alternatively, the security can be 'wrapped around' the data being transmitted, making it independent from the transport mechanism. This second type of security is known as 'end-to-end' security. S/MIME (Secure MIME) and OpenPGP (Open Pretty Good Privacy) are two competing protocols that provide 'end-to-end' security for e-mail messages.

S/MIME and OpenPGP provide all the functionality of MIME but, in addition, allow you to sign e-mail, and send it in a secure 'envelope'. LDAP (a directory service which is discussed in the next chapter) and X.509 certificates can also be used to secure Internet mail.

Work is currently being done by the IMC (Internet Mail Consortium), and the IETF (Internet Engineering Task Force), to resolve S/MIME and OpenPGP, which represent two incompatible standards for mail encryption and security, into a single standard.

S/MIME

S/MIME was originally developed by RSA Data Security Inc. and is based on the Public-Key Cryptography Standards (PKCS) #7 Cryptographic Message Syntax (RFC 2315), and the X.509v3 certificate format. S/MIME creates a message digest, which is encrypted to ensure the message cannot be tampered with in transit. In order to 'seal the message envelope', S/MIME encrypts all the contents using a triple-DES key algorithm.

OpenPGP

OpenPGP, which has superseded the PGP/MIME format, is based on the Pretty Good Privacy (PGP) data encryption standard.

SMTP

A transfer protocol is the language spoken by Mail Transfer Agents (MTAs). There are many such protocols in existence, such as X.400, UUCP (Unix to Unix Copy), and SMTP (Simple Mail Transfer Protocol). SMTP is by far the most commonly used and is the standard message interchange protocol on the Internet.

SMTP uses an envelope and body metaphor to structure a mail transfer. The envelope is used to transfer messages and contains information about the message sender and the destination address. The originating address is used to notify the sender of the message when a delivery failure occurs. The SMTP body contains the entire message including the body and header information.

E-SMTP

Extended Simple Mail Transfer Protocol (E-SMTP), which is described in RFC 1869, is an extension of SMTP to allow for an 8-bit character set. The 7-bit US-ASCII character set, which SMTP used, needed to be extended to allow for other European and Asian characters and to cope with multi-media message bodies. This is very much in line with the MIME extensions to RFC 822.

Storage and Retrieval protocols

When a user retrieves mail from his Internet Service Provider (ISP) he will almost certainly be using POP3 or IMAP4, as the retrieval protocol.

POP

The Post Office Protocol (POP3) would notbe necessary if users had a permanent connection to the Internet and received SMTP messages directly. Historically, users would dial into such a permanently connected machine to read their e-mail remotely; this mail server would be constantly connected to the transport medium of the Internet, allowing it to receive mail continuously.

With the growth in personal computing, there was increased demand for e-mail to be stored locally, with only sporadic connections to the Internet. This led to the development of POP3 as a protocol to retrieve mail from a remote mailbox, via the user's MUA, allowing the mail to be read and stored locally.

IMAP

The most commonly used mail retrieval mechanism is still the Internet's POP3. However, the Interactive Mail Access Protocol (IMAP4) is gaining ground fast.

While POP3 stores all messages in a single message queue, IMAP4 allows for a more flexible mail storage metaphor, using a hierarchical folder concept in which to store messages, and providing a richer set of mail manipulation functions.

IMAP was designed with the following goals in mind:

• To provide the ability to store mail in folders besides your own inbox
• To provide a richer set of functionality for manipulating mail folders
• To provide better access to a mailbox in online and disconnected access modes than the simple offline access offered by POP3
• To facilitate access to a user's email from more than one client computer (for example from a mobile device and an office PC)
• To be fully compatible with the open standards of Internet messaging such as RFC822, MIME, and E-SMTP

However, as POP3 is still the dominant protocol in use on the Internet, and as it is slightly easier to program and conceptualize, the examples using mail retrieval in this chapter will use POP3.

E-Mail's Relationship with LDAP

Ever since the development of X.400 and X.500, e-mail services and directory services have been functionally linked. Lightweight Directory Access Protocol (LDAP), which is the subject of the next chapter, is by far the most commonly deployed directory service for storing corporate address book information, and can be used to deliver address book functionality to MUAs. This is particularly useful to mobile users: there is little benefit in storing globally-administered information locally when it can be efficiently accessed from a single central source, thereby eliminating duplication and the need for synchronization.

E-Mail's Relationship with ACAP

The Application Configuration Access Protocol (ACAP) was conceived at Carnegie Mellon University as part of an ongoing project to enhance the university's e-mail and message exchange systems, known as Project Cyrus. ACAP has been proposed as an Internet standard as defined in RFC 2244.

ACAP covers a lot of the same functionality as LDAP in that it aims to store attributes (name/value pairs) relating to a user on a centrally accessible server. ACAP's focus is to allow highly mobile users the ability to retain configuration information and preferences specific to an individual user - for example, implementing a personal address book - and has been used by some MUAs for this purpose. ACAP client software is also designed to work offline, unlike LDAP.

Work has been done to closely integrate ACAP and the IMAP mail protocol, and if these services achieve a wide market penetration, they may provide substantial benefits to mobile users. However, ACAP suffers from not being as broad in its approach as LDAP, which is ahead in terms of both the maturity of the protocol, and its market share.

A Typical E-Mail System

To summarize, it might be helpful to look at some diagrams of a typical mail system using the Internet protocols. From a user's point of view e-mail is sent via SMTP, collected from their mailbox using POP3 or IMAP, and any address book information is searched for using LDAP (or ACAP):

In terms of the flow of a message (either a RFC822, or a MIME, formatted message) from sender to recipient, the transmission path looks like this:

You should now understand how a mail system is structured. We have looked at how a Mail User Agent talks to a Mail Transfer Agent to send e-mail across a mail backbone, how mail is retrieved from a store and how LDAP is accessed for address book information.

Our next task is to look at how we can incorporate this mail functionality into our mobile applications, using WAP and e-mail to deliver real business value.

Programming E-Mail

There are a number of messaging APIs (Application Programming Interfaces) on the market that enable developers to write simple and 'generic' mail functionality into their programs. Before we look at some specific code examples, and consider how to construct applications that make use of messaging functionality, we will now look at the major API sets, which are:

• Common Mail Calls
• Vendor Independent Messaging
• CDO and CDONTS
• JavaMail

Common Mail Calls (CMC) is the X.400 standard mail API set developed by the X.400 API Association (XAPIA), and as such, has a limited application to this chapter since we are dealing with the Internet mail standards - SMTP, POP3, etc - rather than X.400.

Vendor Independent Messaging

Developed by Lotus, Vendor Independent Messaging (VIM) provides a simple set of API calls that provide platform-independent access support for mail systems.

Microsoft's Collaboration Data Objects (CDO)

CDO is an extremely simple-to-use API that is based on COM objects. What this means in practice is that it's the API you will want to use if you are coding in Visual Basic, Visual C++, or scripting languages on Microsoft platforms - which will be the case if you are using ASP to generate WML pages dynamically - we'll look an example of how to do this later in the chapter. CDO comes in two variants, CDO and CDONTS (CDO for NTServer).

CDO is designed to run against Microsoft's Exchange Server - a sophisticated mail server that can send and receive mail and organize all users' mail accounts for a large organization. Exchange Server's functionality is considerable and is accessible practically in its entirety from CDO. That means that although CDO is still relatively simple to use, it is more complex than you might want for very basic send/receive email functionality. For this reason Microsoft developed CDONTS as an alternative. CDONTS can run either against Exchange Server or against the simple SMTP service that is shipped with Internet Information Server (IIS). CDONTS can't do much beyond sending and receiving e-mails, but that means it is incredibly easy to use - as we'll see later on when we look at an example using CDONTS to programmatically send an email with just three lines of VBScript code.

Using CDO, complex messaging functionality can be pulled together with other related COM based programming models such as ActiveX Data Objects (ADO) and the Active Directory Service Interface (ADSI). This linkage will provide mechanisms to manipulate mail, and easily extend CDO applications, to include other areas of personal information management functionality, such as contact and calendar management.

JavaMail

Released by Sun in August 1998, JavaMail aims to revolutionize access to mail systems in the way that Java Database Connectivity (JDBC) revolutionized access to databases. We will be dealing with this at length in the next section. The advantages of using JavaMail include:

• It is available on the majority of current operating systems
• It offers an e-mail API that is both flexible and easy to use
• It offers excellent networking capabilities as standard