Building a Robust Linux Security Solution

The Linux OS is emerging as a viable alternative to "closed-system" security solutions. Here's how you can build a complete Linux security scheme from the ground up.

March 1, 2000

16 Min Read
Network Computing logo

Linux possesses a stable, robust, open-source kernel whose quality is subject to daily scrutiny by thousands of “power users” around the world. As with any other open-source system, once vulnerability is identified, it is typically published and verified within hours—with a fix typically available in days. While Linux still faces serious challenges as a Windows replacement in the office environment, its ability to perform as an enterprise server is no longer in question.

The open-source nature of Linux may require that you devote some time and effort to staying abreast of the latest security advisories and available kernel versions, but it also means you can exercise tight control over system resources, right down to the kernel layer. This is time well spent when you consider the potential consequences of a successful network attack.

Of course, while cutting-edge knowledge of Linux kernel development has its virtues, your security system is probably not the right platform to test-drive the latest and greatest that the “VolksOS” has to offer. Linux kernel versions come in two very distinct lines: the development (experimental) line, which uses odd minor version numbers (for example, 2.1.x); and the stable production line, characterized by even minor version numbers (for example, 2.2.x). New features and drivers are tested on experimental kernel versions, which makes their behavior largely unpredictable.

I strongly recommend against running experimental kernels on your Linux security device. However, I do recommend that you build and install the latest stable version as soon as it’s available. New kernels often address security vulnerabilities, and the version that you got on your distribution is likely to be outdated by the time it gets placed on the shelf.Let’s take a closer look at the building blocks that make up a complete Linux security system and see how all the pieces fit together to secure the enterprise network.


While a good perimeter defense system spans multiple layers, the Network (IP) layer often remains the most vulnerable. From its inception, the Linux kernel was implemented to support filtering of both incoming and outgoing IP packets, based on a user-specified set of access-control rules.

The ipchains utility lets the Linux system administrator interact with the kernel to specify these rules. The name “chains” refers to the sequential nature in which the kernel consults rules belonging to the same chain. The ipchains syntax is succinct and intuitive. For example, if you want to accept all outbound traffic while restricting inbound traffic to SMTP only, you would use the following statements:

% ipchains -A output -s –I eth0 -j ACCEPT
% ipchains -A input -p TCP -dport 25 –I eth0 -j ACCEPT
% ipchains –Aoutput –I eth0 –j DENY
% ipchains –A input –I eth0 –j DENY

The first line issues a directive to modify the output chain to accept packets whose source addresses are in the network block (the local network) to go out the first Ethernet interface, (eth0). Conversely, the second line instructs the kernel to accept incoming traffic from any source (no

–s switch is present), as long as the destination TCP port number (-dport) is the one that corresponds to SMTP (25). The third and fourth lines simply declare a “deny-all” stance, where the default action is to block all traffic on all chains unless otherwise permitted by a previous rule. I strongly recommend that you adopt a similar strategy.In addition to the output and input chains, Linux supports a forward chain that controls packets that flow through the firewall but are destined for another host.

As with many other packet-filtering systems, one drawback to ipchains is that certain types of higher-level protocols (like UDP) are difficult to filter. These protocols often require the ability to perform a “stateful inspection” of the connection table, where the decision to accept or deny a packet-forward request is based on whether or not an outbound connection to the requesting host already exists.

Note that kernel support for ipchains is not enabled by default, so you will have to rebuild a kernel from the source tree, remembering to specify the CONFIG_FIREWALL option during the make config step.


Due to the ever-shrinking IPv4 address space, it is becoming increasingly difficult to obtain enough addresses to assign every corporate host a legal (that is, globally routable) IP number. This has led network administrators to assign reserved addresses to most corporate hosts, using Network Address Translation (NAT) at the perimeter, to preserve the ability to access Internet hosts.

The Linux NAT solution comes in a kernel feature dubbed “IP masquerading,” by which a firewall device can translate several internal (illegal) IP addresses to a single external (routable) address. The mechanism that achieves this is often referred to as either Network Address Port Translation (NAPT) or “one-to-many” NAT.

The Linux firewall device acts as the default router for the community of internal hosts, and replaces the source address of outbound packets with the source address of the firewall itself. It then assigns a high TCP port number as the source of the translated connection, which it uses to keep track of which internal host initiated the connection.

To enable IP masquerading, you must select both the CONFIG_IP_FIREWALL and the CONFIG_IP_MASQUERADE options when you build your Linux kernel. Once you have rebooted the machine and loaded the new kernel, simply issue the following command: % echo “1” > /proc/sys/net/ipv4/ip_forward

Once enabled, Linux IP masquerading can be configured using the ipchains utility discussed previously. For example, if you choose the Class C network ( as your internal address range and would like to mask all your addresses as they traverse the firewall, simply issue the following command:

% ipchains -A forward -s -j MASQA well-known shortcoming of IP masquerading (and NAPT in general) is the inability of outside (Internet) hosts to initiate connections to internal hosts. For this same reason, IP masquerading is a suitable solution for client environments, such as a community of Windows PCs that needs to browse the Web and send Internet e-mail while receiving its e-mail from a POP3 server. For more information on Linux IP masquerading, go to


Network managers often find themselves in need of a higher degree of access control than that offered by a packet-filtering firewall. It’s not always possible to track the IP addresses that should be granted network access, and Network-layer filtering leaves no room for user authentication and session logging. An Application-layer firewall system lets the security administrator specify which users are permitted to use a given network service (for example: HTTP, SMTP, POP3), regardless of the source address the request originated from.

The IETF has blessed the SOCKS protocol as the standard for proxying requests across Application-layer firewalls. When a SOCKS-aware Web browser attempts to establish a connection to an outside resource, the SOCKS server processes the request instead and proxies the connection on the client’s behalf. SOCKS 5 (RFCs 1928, 1929, and 1961) supports strong user authentication, as well as the ability to proxy both TCP and UDP connections.

Aside from the ability to authenticate users, SOCKS firewalls come with a more subtle benefit: Because the proxy server originates all network requests on behalf of the clients, only one legal IP address needs to be deployed. Whilethis isn’t technically NAT, this feature bodes well for security administrators, who prefer not to have internal addresses (even the legal ones) known to the Internet at large.The NEC SOCKS 5 implementation ( includes a freely available and fully functional Linux version of the SOCKS proxy server, as well as the libraries needed to “SOCKSify” both Linux and Windows end-user applications. In addition to user authentication, SOCKS 5 lets security administrators compile detailed logs on user Internet access and includes a set of administration tools to manipulate and summarize these logs.

As network managers gear up for the upcoming conversion to IPv6 (albeit a long conversion), the NEC SOCKS 5 package is emerging as an invaluable transition tool due to its ability to speak IPv4 to the private network side while using IPv6 packets to communicate with the Internet. This translator feature is available as a free patch to the standard Linux SOCKS 5 server.

If you want to implement a SOCKS 5-based security solution for resale, NEC offers a commercial software toolkit called the SOCKS 5 Border Control Framework that enables security product implementers to develop turnkey hardware/software SOCKS 5 packages. For more information on this offering, go to

An alternative to SOCKS can be found in the Trusted Information Systems (TIS) Firewall Toolkit, the precursor to the commercial TIS Gauntlet firewall, now made available by Network Associates. The Firewall Toolkit (FWTK) includes a set of utilities (and full source code) that lets the security administrator set up proxies for any number of well-known TCP services (Telnet, SMTP, POP3, HTTP, and so on). Unlike the SOCKS proxy server, FWTK proxies can be set up on arbitrary ports, instead of the single TCP port number (1080) used to proxy connections through SOCKS. Client applications must be reconfigured to use the FWTK proxy port instead of the original service’s port number.

While SOCKS 5 offers a more comprehensive application proxy solution, the TIS toolkit is a useful alternative in cases where only a single service needs to be proxied. It is also a good alternative when the service is not one of the standard TCP ports (the toolkit includes a general-purpose plug-gw proxy that is designed for such applications). The TIS Firewall Toolkit can be obtained from


There is no reason why you can’t take advantage of a PKI-based solution to address your authentication and authorization problems. For example, if a number of your business partners need access to an extranet Web server in the company’s Demilitarized Zone (DMZ), you can set up a small Registration Authority (RA) server to accept requests to sign X.509 public key certificates and a Certification Authority (CA) to process the requests and sign the certificates.

Both Netscape Navigator and Microsoft Internet Explorer currently support X.509 certificates, and there are free Linux implementations of RA, CA, and Lightweight Directory Access Protocol (LDAP) servers.

Based in Europe and boasting developers scattered worldwide, the OpenCA Project ( was born from the need for an open-source toolkit for rapid deployment of all the elements necessary for a PKI.

The first building block of a Web PKI is a standard Web browser that lets users request a certificate from an RA. The OpenRA is implemented as a set of CGI scripts written in Perl and hosted on a secure Web server. The OpenCA team looked to the proven Apache Web server, adding standard Secure Sockets Layer (SSL) support to it to ensure the privacy of the certificate requests. As soon as the RA receives a certificate-signing request, it queues it for the CA, which is typically kept offline for security reasons. Once the CA signs the request and issues the certificate, the browser can once again visit the RA and load the certificate to the browser’s local cache.The U.S.-based OpenLDAP project ( provides an implementation of LDAP, often used by the RA to store certificates, which can then be queried by PKI subjects. The OpenLDAP distribution includes an LDAP server (primary and replicated), as well as a set of sample clients and a number of administration tools.

Let’s return to the extranet access problem. Your business partner has requested a certificate from your RA, and your CA has approved the request. Your partner can now present this certificate to your secure Web site and use strong authentication to access the extranet services that you offer on your DMZ. This is a straightforward solution, using proven technology (HTTP servers, CGI scripts, and Perl modules) and standards-based protocols (X.509, SSL, and LDAP).

The OpenCA and OpenRA modules can be obtained from The OpenLDAP server is available from The Apache and OpenSSL software can be downloaded from and, respectively.


Developed under the auspices of the IETF, the IPSec standards (RFC 2401) have slowly emerged as the most popular protocols for ensuring the privacy and integrity of data traversing VPNs. Using IPSec’s Encapsulated Security Payload (ESP) at the security gateways, corporations large and small can use the Internet to transfer private data inside encapsulated and encrypted packets, whose source and destination addresses are always those of the security gateways.The Free Secure Wide Area Network (Free S/WAN) project was born as an initiative to offer an open-source, full- featured IPSec implementation for a wide range of target architectures, including the Linux OS. Unlike the Secure Wide Area Network (S/WAN) product offered by RSA Data Security, Free S/WAN is available at no cost, and since all the cryptographic development takes place outside the United States, strong encryption is also offered as part of the standard distribution.

The Free S/WAN package comprises two discrete components: the kernel module and the Internet Key Exchange (IKE) daemon.

As with any other security service, IPSec requires kernel support for the encryption and encapsulation of packet payload. The Free S/WAN modules must be merged into the standard Linux source tree, and a new kernel must be compiled.

The IKE protocol requires two IPSec gateways to exchange authentication and key setup information before the actual IPSec tunnel can be constructed. The information negotiated via IKE generates the Security Association (SA), which contains the tunnel credentials that each endpoint maintains for each of its peers. The Free S/WAN IKE daemon is implemented in user space and uses UDP port 500 for sending and receiving data to the remote IPSec peer. This daemon must be compiled and installed for Free S/WAN to operate correctly.

The main advantage of a Network-layer VPN solution is its simplicity; hosts behind the IPSec gateway need not be aware of its existence at all, as no client modification is necessary. IPSec can protect any service that uses the IP layer, typically with a nominal amount of overhead, even when the strongest data encryption (3DES) is selected (and often using a relatively inexpensive platform).The Free S/WAN IPSec distribution for Linux can be obtained from

If you want to grant your remote users VPN access to your Linux gateway, but you don’t want to install (or maintain) IPSec software on their laptops, you are in luck: PopTop is a freely available Point-to-Point Tunneling Protocol (PPTP) server that can act as an end-point for VPN sessions from standard Windows desktops. Microsoft includes PPTP VPN support on Windows 95/98/NT, and there is even a PPTP client available for Linux clients. The PPTP standard is the subject of Internet Draft RFC 2637 and has been implemented by many commercial vendors.

Be aware that while PPTP is fundamentally sound, Microsoft’s PPTP implementation contains several obvious security flaws. Visit the Counterpane Systems report at before you decide to implement this solution. You should also consider that Microsoft has announced IPSec support in the upcoming release of its Windows 2000 OS.

The PopTop server for Linux can be obtained from, and a PPTP client for Linux is also available from


Linux has finally reached the maturity level necessary to be considered a viable security platform. Open-source solutions have some well-known drawbacks (lack of support, spotty documentation), but Linux makes up for them with a proven design, a rock-solid kernel, and the ever-growing support of software vendors that see Linux as a much-needed revolution in the Unix landscape.

Security managers have quickly learned to ignore much of the hype that has often surrounded Linux’s rapid success. Its real security value is as a stable, robust platform where security vulnerabilities are quickly identified and addressed, and where managers can control system resources down to the kernel. Linux offers all that, and at a price that can’t be beat.

Ramón J. Hontañón is an Internet security engineer at UUNET, an MCI WorldCom company. He can be reached at [email protected].


A host of Linux sites exists on the Internet. Here are a few to get you started:

For information on Linux kernel sources, check out is an educational site promoting the Linux OS. It’s available at

Linux Weekly News features the latest news and reports on Linux. Go to

Information on the Linux certification program from the Linux Professional Institute (LPI) is available at

Linux Today has news and other resources for business professionals interested in Linux. Go to

Which Linux Distribution Is Best?

OK, so your mind is made up: You’ve done your homework and concluded that Linux is the right security platform for your enterprise. The next decision you make will be the most important: which distribution to install?

By now you’ve probably guessed that there is no right answer to this question, but here are a few pointers to helpyou decide which distribution is right for you:

RedHat Software ( was the first mainstream, commercial Linux distribution, which managed to reach out to a market segment typically dominated by Microsoft products. Central to RedHat’s success is a well-polished graphical installation system that includes plenty of online help throughout the entire process. It also pioneered the concept of integrated package installation with its RedHat Package Manager (RPM) utility and package format, designed to simplify the process of adding third-party software utilities to the core Linux system. Furthermore, RedHat offers a certification program backed by a three-week training curriculum. Recipients of the RedHat Certified Engineer (RHCE) distinction are expected to master topics ranging from Unix internals to TCP/IP concepts and network security practices.

While RedHat is the clear choice for the Linux neophyte, it is equally suitable for the network administrator looking for a solid, proven distribution with a wide range of support options.

Caldera Systems’ ( OpenLinux offers an equally seasoned distribution that also supports the RPM package management system and features a graphical installation system. Caldera also boasts a comprehensive training program, both in-house and through their Authorized Linux Education Center (ALEC) program—each designed to prepare students for the Linux Professional Institute (LPI) certificate. The choice between OpenLinux and RedHat is a toss-up.SuSE ( offers a comprehensive package, including a very rich set of X Window utilities and graphics adapter support. By far the largest distribution (five or more CDs), it also offers a suite of open-source security tools, including a proxy server, sample firewall scripts, and a system-hardening utility. SuSE also supports the RPM package format.

SuSE is the most popular distribution among European Linux users and perhaps the best choice for the intermediate-to-advanced user who is looking for a security-conscious distribution out of the box.

Debian’s ( GNU/Linux is the clear choice if you want to go freeware. This distribution is actively updated and supported by a core of devoted (and remarkably well-organized) supporters around the world. The package format is very similar to RedHat’s RPM and just as easy to master. The initial system setup is not as intuitive as RedHat’s, but a seasoned systems administrator will actually find it more efficient, especially on multiple installations.

Slackware ( was one of the first distribution methods to hit the scene, back when the entire Linux system would easily fit on a handful of floppies. While it’s still being actively maintained, its installation is not quite as straightforward as the other Linux offerings. Slackware relies on compressed “tar” archives and does not support the more popular RPM format.

Stay informed! Sign up to get expert advice and insight delivered direct to your inbox

You May Also Like

More Insights