Linux Provisioning Systems to the Rescue

It was only a few years ago that using Linux on enterprise-level servers and workstations meant a low-cost (or downright free) operating system, with high operational costs. Those days are gone; Linux OS distributions have matured and stabilized. The cost of running Linux in your enterprise is now far lower than ever before.

Of course, while Linux may be easy to use and implement on one server, that doesn't mean it's easy to deploy and manage a fleet of enterprise Linux servers or workstations. Sites with a large Linux presence typically must build complicated custom scripts and programs to handle installation, and such customization extends beyond the conventional limitations of vendor-supplied installation systems.

Linux provisioning systems can help ease this burden. They combine controls for OS installation and configuration; application installation and configuration; policy enforcement; and patching in a single interface. They provide a comprehensive approach to Unix and Linux provisioning and eliminate the need for vendor-supplied installation tools and custom scripts and programs. Large Unix and Linux sites can gain tremendous benefit from such systems.

There are, of course, some pitfalls with Linux provisioning systems, such as possible overreliance on one vendor for both the OS and provisioning system. And if you have a Linux provisioning system and decide to switch to another, migration must be carefully planned. Anyone preparing to incorporate a Linux provisioning system should consider those two facets before deployment.The Players

The commercial systems we examined--from Mandriva, Novell and Xandros--all address the typical problems of provisioning large groups of machines, but there are key differences among them. We also checked out a homegrown solution implemented at the University of Wisconsin-Madison's CAE (Computer-Aided Engineering) Center. Red Hat declined our invitation to participate in this review, claiming it's still at the beginning stages of combining Red Hat Network and JBoss Operations Network.

If you're looking for the broadest support of OSs (for instance, Microsoft Windows OSs) and the most in-depth reporting and monitoring, Novell's ZENworks is your best bet (see "More Than Just Linux," page 59). However, ZENworks also is the most difficult to learn to use.

If you are from a predominantly Microsoft shop and are making inroads into Linux, Xandros' xDMS (Desktop Management Server) and Xandros Server Management Console will be familiar and easy to use. Both support GUIs that are similar to native Microsoft applications. Xandros Server Management Console focuses on the management and configuration of servers and the services running on them, while xDMS focuses on installation automation and golden-image creation.Mandriva's Pulse was in the beta stage during our tests. As such, in its current state it would be useful only for seasoned Linux administrators. We fully expect this will improve before the tool is released.

Feature Checklist

Whether you're considering any of the provisioning tools we examined or another provisioning system, you should be aware of the general benefits and drawbacks to the class of tools.

With a commercial provisioning tool, for instance, you can solve one of the perennial problems faced by many systems administrators: documentation. If you have lots of homegrown provisioning tools, someone on staff must write and document those programs--and the latter oftens become a low priority. With a commercial provisioning tool, however, that onerous task goes away.

And speaking of documentation, you will most certainly want documentation regarding the states of the machines at your organization. With a commercial provisioning tool, you can generate a complete set of information regarding your machines. These tools also can produce detailed inventory reports of hardware that's been provisioned. Even better, this information is typically discovered by the provisioning system as each machine is installed or when a client-side program or service is running.

Another benefit of commercial provisioning tools is that they do away with the need for custom programs, which carry the inherent hazard of brain drain. If only one or two people in your organization know the complete workings of a custom provisioning system, that represents a huge liability. Commercial provisioning systems obviously don't hold the same risk.

As one might expect, commercial Linux provisioning systems generally work better when they are being used to provision the same vendor's OS distribution; the provisioning systems we evaluated present fewer problems when used with certain versions of Linux, but they do not require it. Like incorporating any multiple of products from a given vendor, using a provisioning tool from the same company that made the OS distribution will weaken your position in any future negotiations with that company because you are reliant on it. If that vendor falls out of favor, you'll have some thorny issues to work out.

As provisioning systems provide all the tools required to incorporate existing servers, the migration path for a change of provisioning system is straightforward but, as with any major transition, requires careful planning. One possible migration strategy would be to turn off the out-of-favor provisioning system and incorporate the existing servers into the new provisioning system (as if they had not ever been provisioned). Another path, probably more appropriate for desktop workstations, would simply be to reinstall the operating system in the new provisioning system. As the installation process becomes more automated, the cost of a full reinstallation is quickly reduced.Cost Comparison

Cost is clearly a consideration with any commercial Linux provisioning system. Of the systems we examined, ZENworks is most expensive. To illustrate the difference in pricing schemes, we use the OS and provisioning system cost at an example site where there are 10 servers and 1,000 desktop workstations.

Xandros' xDMS costs $495, which includes the xDMS server OS software and five copies of Xandros Business Desktop 3.0; additional Xandros Business Desktop licenses are available for $129.95 per machine. Xandros Server Management Console, which is part of Xandros Server 1.0 Standard, costs $449.99. So, for our example site, with 10 servers, one xDMS server and 1,000 desktop workstations, the total list price would be just over $134,000.

ZENworks costs substantially more. Pricing for Novell's ZENworks is separate from the base SLED (SuSE Linux Enterprise Desktop) or SLES (SuSE Linux Enterprise Server) license. SLES starts at $873 for three years (or $349 annually). Novell's SLED licenses start at $125 per desktop for three years (or $50 annually). It costs $83 for the license to have a device managed by ZENworks for three years. The total cost for our example site would be $217,560.

The least expensive is Mandriva's Pulse client and server. A major reason for its low cost stems from the fact that Pulse will be released under the GNU General Public License, so it will not have a licensing cost. However, a server is still needed, and Mandriva Corporate Server 4 costs $484 with three years of maintenance. Pricing for Mandriva's Corporate Desktop starts at $99.90 per machine (which includes five years of product maintenance). This would cost only $104,740 for our example site.Because Mandriva's Pulse will be released under the GPL, unlike ZENworks or any of Xandros' tools, Mandriva won't be able to add unforeseen costs in the future. This gives your organization more budget control. Although Mandriva would like customers to use Pulse along with its Enterprise Linux OS, the vendor is also taking the extra step to ensure that the tool can work with other operating systems--including Microsoft Windows.

On the other hand, Xandros' tools, xDMS and Xandros Server Management Console, as well as ZENworks, use proprietary licenses for their installation tools.

Etched In Gold

The first step in provisioning a large number of machines is to make a golden image. A provisioning system will make an image from a complete copy of the machine's contents, or a complete copy of the configuration files needed to generate the machine's contents. After the image is created, it is saved to a server and the provisioning system installs it where it's needed, whether on client workstations or on servers. This ability to install a machine from scratch--by booting machines directly off a network server using PXE (Preboot Execution Environment) and/or CD-ROM booting--is the most basic feature of any provisioning system.

In most cases, your organization will likely have a variety of hardware and software configurations, forcing you to pick an approach to provisioning. Either you can make one image for each configuration, requiring lots of staff time, or you can make less specific images that require customization before a machine is useful. Per-task customization is one area where the products we looked at showed the most difference.ZENworks, for instance, leaves application configuration to its YaST (Yet Another Software Tool) software, which comes with Novell's Linux OSs. The drawback is that if you'd like to make a change on multiple machines, you must run YaST on each machine. A striking feature of Xandros' Server Management Console is its slick software-configuration GUI. Unfortunately, the product only configures servers, meaning you must use Xandros' xDMS to make changes on clients as well. Mandriva's Pulse, meanwhile, handles both clients and servers and allows plug-ins to be added easily.

CAE uses a cocktail of open-source tools to take a very generic image and convert it into a running workstation.

Patchwork Project

Patching machines, as even the most cursory glance at industry trends indicate, is critical to the running of your operation. Both ZENworks and xDMS let you build local software repositories and distribute updates from this local cache. ZENworks includes a software package called ZENworks Linux Mirror, which provides fine-grained control over which software packages should be imported into ZENworks. However, ZENworks Linux Mirror is complicated to set up and requires work outside the Web-based interface. Once it's properly configured, though, ZENworks Linux Mirror can download packages from remote ZENworks servers, YaST Online Updates (Novell's standalone system patch-management service) or Red Hat Network servers.

In the version of Pulse we examined, the only way to add patches is to install them manually in the Pulse repository. Clearly, that is something that will be fixed before Pulse is fully released.Show Some Control

A feature ZENworks carries over from its Microsoft Windows background is control of desktop settings from within ZENworks using ZENworks policies. These policies can restrict and change many aspects of the end-user experience. For instance, we were able to assign a policy whereby we disabled every bell and whistle on the gnome desktop. Such control can be exercised over specific machines or groups of machines. However, many organizations with Linux desktops roll out preconfigured, prerestricted desktop environments. For such companies, ZENworks policies would be unnecessary.

ZENworks bundles in a VNC (Virtual Network Computing) server. Similar to Microsoft's RemoteDesktop, VNC provides a method to allow remote control of the consoles of machines controlled through ZENworks.

Although VNC isn't bundled with xDMS or Pulse, it can be added--at the cost of the time it takes to install it. There are several open-source and freely available VNC servers out there. Two we like are RealVNC and Tight VNC.

More Than Just Linux

It's worth noting that both Novell ZENworks and Mandriva Pulse operate on Microsoft Windows OSs. In fact, this is predominantly where ZENworks has made a name for itself. For many years, ZENworks has been used to support, install and maintain Windows workstations at CAE.To be clear, ZENworks has more features for Windows desktops than for Linux desktops. Besides the control that is analogous to what is available on Linux, ZENworks can fully manage profiles, so that a profile can be moved from one machine to another and one version of Windows to another. On Linux machines, most of that information is stored in a user's home directory, so if you copy the entire home directory, the entire profile is copied too.

Also interesting is Novell's claim that the latest version of ZENworks--ZENworks Handheld Management--can manage handheld computers, such as RIM BlackBerry, Palm and WinCE devices, even with low-bandwidth situations and intermittent connections.

Mandriva also sent us a beta version of its Windows client. It is based on Ruby, an open-source interpreted programming language with many useful systems administration features--for instance, it offers a number of ways to process things, just as one could in Perl. Ruby is more commonly used on Linux systems; it does have native Windows support, and Mandriva included for us the prerequisite Ruby binaries for Windows.

How We Tested Linux Provisioning

Testing systems that can manage and install Linux systems automatically can be exceedingly risky on a live network. Unrequested installs on a co-worker's machine are frowned upon.

We tested these products in the NETWORK COMPUTING Real-World Labs® at the University of Wisconsin-Madison Computer-Aided Engineering Center.We used a private network for each vendor's solution. This way, each server could have authority for its portion of the network and be unaware of the others. Machines were added or removed on each private network.

Each of those private networks was connected to the public network through a firewall. We chose to use m0n0wall, which is an easy-to-use, FreeBSD-based firewall. We also used Microsoft Windows XP workstations to make sure they would work properly on the network.

The Novell server was running SuSE Linux Enterprise Server 9.0 with the latest set of patches. We used a mixture of test machines, including SuSE Linux Enterprise Desktop 9.0 and 10.0, and SuSE Linux Enterprise Server 9.0 and 10.0.

The Xandros subnet consisted of two servers, one running Xandros Server Standard 1.0 Edition, and the other running Xandros Desktop Management Server 1.0.1. We also were running additional Xandros Server Standard Edition servers and Xandros Business Desktop 3.0.3.

Our Mandriva network consisted of a Mandriva Corporate Server 4.0 machine and a Mandriva Corporate Desktop workstation.All Network Computing product reviews are conducted by current or former IT professionals in our own Real-World Labs®, according to our own test criteria. Vendor involvement is limited to assistance in configuration and troubleshooting. Network Computing schedules reviews based solely on our editorial judgment of reader needs, and we conduct tests and publish results without vendor influence.

Executive Summary: Linux Provisioning

Linux provisioning tools ease the deployment chores for sites that have hundreds or thousands of Linux machines. They deliver controls for operating system and application installation and configuration, policy enforcement and patching.

Each of the systems we examined offers benefits that address different needs. For companies looking for the broadest support possible and the most in-depth reporting and monitoring, Novell's ZENworks is a good choice, though it is also the most expensive one. Administrators from a mainly Microsoft-oriented background will find the interface of Xandros' xDMS and Xandros Server Management Console familiar and easy to use, but the company's two-product approach is a drawback. Mandriva's Pulse, while the cheapest of the three systems, is still in the beta stage. Once it ships, it will offer good future-proofing since it will be released under the GPL. And companies might also consider cobbling together open-source tools to build their own provisioning system; the Computer-Aided Engineering Center at the University of Madison-Wisconsin has constructed such a system.

For a more in-depth analysis of these products, go to nwcreports.com.Novell Zenworks

Large organizations that have a firm boundary between those who maintain the OS and those who maintain the network should consider ZENworks. This provisioning tool targets OS-only functions, leaving a Novell's YaST to handle service-level configuration of servers and workstations. The UI also has some interesting features that will be a big help in navigating the myriad processes in an installation with many servers.

In addition, ZENworks was the most comprehensive Linux provisioning system of those we tested.

Like Mandriva's Pulse, ZENworks uses a Web browser to navigate its management interface, which is straightforward. One of the elements we especially liked was the "recently used" items. As we were changing configuration on particular servers, the configuration elements showed up in the "recently used" items list. This let us jump around quickly among components that were recently configured, without having to drill back down into different parts of ZENworks. Additionally, quick links to detailed product documentation was readily accessible.

One of the substantial differentiators between the products we reviewed is the conceptual boundaries Novell draws around ZENworks. Novell views ZENworks as a base OS configuration and management tool only. System services are configured using YaST, which is not integrated with ZENworks. On the other hand, both Xandros and Mandriva view both base OS and system service configuration as within the domain of their provisioning tools.Novell claims that this division represents segmentation of the typical enterprise Linux core systems administration staff and service administration staff duties. The company's claims have merit in very large organizations where the duties of machine installation and maintenance are indeed separate from service operation. If your organization doesn't have that separation, having different tools is certainly going to be an unnecessary complication for you.

One useful feature was the machine inventory available from ZENworks. During our tests, we were able to get a lot of information about the machines that we were managing, such as memory and sizes of hard drives, as well as motherboard versions, chassis information and more. Having this information at your fingertips aids not only initial troubleshooting, but also the analysis of your environment.

Using the ZENworks Web-based interface, you can turn the raw information provided by the client service into the many reports a large enterprise might require. If you have decided to replace a certain model of 17-inch CRT monitor with flat panels, for instance, it is easy within ZENworks to create a report of the names of all the workstations with 17-inch CRT monitors. ZENworks comes with a number of preconfigured reports that can be run, including those for new machines in the last day or week, errors in the last day or week, and devices that have been inactive for more than 90 days.

The basic building blocks for installing Linux software applications on Linux computers through ZENworks are RPM (Red Hat Package Manager) packages. Sets of RPM packages are grouped together into bundles or catalogs. Bundles are sets of RPM packages that a machine must have; catalogs are sets of RPM packages a machine could have, either as optional software or to resolve dependencies--important because often certain RPM packages require that other RPM packages be installed. Having the entire set of RPM packages in a ZENworks catalog will provide ZENworks with a method to install additional RPM packages to resolve dependencies.

ZENworks also offers another level of control through the creation and enforcement of policies. These policies can, for example, define what desktop settings a user can manipulate.Any of the above methods (bundles, catalogs and policies) can be applied to computers or groups of computers. If, for instance, you find that your marketing department needs to have a new super-duper software tool installed, you can take the RPM packages, bundle them into a ZENworks bundle, and instruct ZENworks to install the bundle to the marketing department computers.

If you have existing Novell's SuSE Linux servers, incorporating them into a new ZENworks server is easy. With a simple shell script that comes with ZENworks, we created a custom CD-ROM image and mounted that image on an already installed SuSE Linux server. We then installed the ZENworks RCD (Red Carpet Daemon, the ZENworks Linux agent). The server communicated with our ZENworks server and immediately went to work, pulling down several software updates that were not already installed.

Making a copy of that server was easy too. Using the server above as our golden image, we rebooted the machine using PXE into the Novell Preboot Services Menu; there was no need to even burn a CD. From this menu, we then were able to put a copy of our golden client on the server.

Then it was time to make our clone. We went to the machine that was to become the clone and booted it with PXE. We then instructed the Novell ZENworks Imaging Engine to install the image. A few minutes later, our image was saved onto the machine and the new clone rebooted.

Novell says that it uses the internals of ZENworks in Novell's latest versions of SLES and SLED (SuSE Linux Enterprise Server/Desktop). It also says that an update of ZENworks is due soon that will include native support for SLES and SLED 10.Xandros xDMS

Xandros xDMS differs from both Pulse and ZENworks in that it is based on the Debian DEB package management system, instead of the RPM package management system. As a result, xDMS uses native GUI tools to update in the same manner that Debian machines do.

The current Xandros tools come with a clear limitation: One tool installs and configures clients; another tool configures servers, but does not install them. Xandros claims to be working on an update that will combine the tools. Clearly if Xandros does this, the product will be far more powerful. With the version we tested, Xandros Server Management Console has decent configuration-management abilities--only for Xandros servers--but lacks installation automation. xDMS provides installation automation and golden-image abilities but does not have strong configuration management. Each tool complements the other and, were they merged, would make a single, much stronger tool.

Instead of Web browser-based front ends, Xandros provides native clients for management. Xandros' tools are aimed at those familiar with Windows-based tools.

Since Xandros has one tool for desktop workstation installation and configuration and another tool for server configuration, we needed two servers. xDMS comes with its own server OS. We had one Xandros Server 1.0 Standard server and one xDMS 1.0.1 server. The Xandros Server 1.0 Standard server provided the DHCP and DNS services and ran the Xandros Server Management Console, providing the server configuration features. The xDMS server ran xDMS Console application, Administrator application and Repository Builder application. xDMS also ran tftpd (trivial file transfer protocol daemon), needed to provide PXE booting ability for Xandros Network Setup.One aspect of xDMS we liked was the clear integration between the Xandros Network Setup utility and the xDMS GUI. As we deployed our test workstations, the xDMS Console application gave us a very clear view into the status of our test workstations installing Xandros Business Desktop 3.0.

Xandros' other tool, Xandros Server Management Console, like Mandriva Pulse and unlike ZENworks, offers a centralized server configuration tool. Using Xandros Server Management Console we could change features on any server from a single GUI tool. Xandros Server Management Console enables software configuration on any Xandros server configured to report to a particular domain from within one GUI tool. Novell relies on its YaST tool. However, each YaST instance is unique to each server and is not integrated with the others.

Much like xDMS, Xandros Server Management Console has a well-integrated user interface. If you're familiar with the Microsoft Windows administration tools, you will feel right at home using this tool. We installed several Xandros Server 1.0 Standard test servers. Using Xandros Server Management Console's familiar two-paned window, we found it quite simple to change configuration settings on the servers as needed. Furthermore, services could be stopped, started or restarted easily.

Mandriva Pulse

Mandriva's Pulse will be released under the GPL. This means that Pulse will be freely available as open-source software and can be extended. Unfortunately, Pulse is in beta stage. Mandriva says Pulse should be available in the first quarter of 2007.The user interface is quite clean and straightforward. We were able to add and remove packages and change package repositories without difficulty. Obviously, as beta software we found a number of issues that clearly will need to be addressed before the software is released. Given Mandriva's track record, we feel that Pulse will be worth consideration once the product ships in full release.

CAE Custom System

The Computer-Aided Engineering center at the University of Wisconsin-Madison uses a home-grown Linux provisioning system. It does what we need it to, but building your own provisioning system requires a high level of Linux knowledge and wouldn't be something we'd recommend to those without significant Linux experience.

We use custom DHCP and DNS servers that in turn use an Oracle database. As machines are installed and configured on the network, their entries in the database are updated.

The process for provisioning new workstations is as follows: We add the machine along with metadata (including responsible person, room and jack number, and so on) into an entirely home-grown IP address management database. This database is also used by our DHCP and DNS servers to provide both services.Our site uses Debian Linux. Initial installation is handled by an open-source program called FAI (Fully Automated Installation.) FAI, like xDMS, works only with DEB files. If you are using an RPM-based Linux distribution, you might want to consider tools like Red Hat's Kickstart or Novell's AutoYaST (both free).

FAI is responsible for everything from partitioning hard drives to installing a very generic OS; this is typically just enough to get the Linux workstation to be able to reboot and recognize CAE's servers. We then use the open-source tools rsync and/or cfengine to apply the final running configuration for the machine. For more information about rsync and cfengine, see "A Solution to Linux Management".

CAE has two classes of workstations: laboratory workstations and independent servers. Our computer laboratory workstations consist of a large number of identically configured machines, whose image changes on a regular basis. We use rsync to convert a workstation from the initial generic installation provided by FAI to our current running image, as well as to ensure new changes get pushed out (and any local changes are discarded). Rsync doesn't handle exceptions easily, so we do not use it in places where large exceptions are the norm--like on servers.

Application delivery is through two methods. We locally install open-source applications using the Debian-native APT (Advanced Packaging Tool) system. We deliver commercial engineering applications through AFS (Andrew File System).

Some of our tools have graphical interfaces, but they are not consistent and are not as nice as any of the commercial packages discussed above. As with any home-grown tool, we are responsible for our own documentation--a perennial challenge for us.Each of the systems we examined has strengths and weaknesses. Will CAE switch away from its current collection of DIY tools to a tool such as xDMS or ZENworks? Time will tell, but given that what we have is working for us, it is hard to justify the investment. However, these commercial tools are certainly worth looking at and may give you quick ROI, depending on the particulars of your situation.

Jeff Ballard is the Unix systems manager for the Computer-Aided Engineering Center at the University of Wisconsin-Madison. Write to him at [email protected].