Infrastructure Configuration Management

Luckily, a new crop of configuration vendors has risen to the challenge of managing multivendor networks. Although not as aggressive in scope as typical PBNM systems, these new systems are worth the money. At the most basic level of system configuration, they replicate much of what can be done through scripts--including archiving, differencing and transferring configs. They also add higher-level functions, such as device access control and configuration auditing.

One big reason for embracing configuration management is compliance ... meeting and mastering the procedures that assure network compliance to SAS-70 (a security standard), ISO 17799 (a broader best-practice production control spec), or the mother of pushy standards, Sarbanes-Oxley (see "Complexities of Compliance,").

Come Into Our Parlor

We asked AlterPoint, Gold Wire Technology, Intelliden Corp., Rendition Networks, Tripwire and Voyence to send their configuration-management products to our Syracuse University Real-World Labs. Rendition, Gold Wire and Voyence did so, but AlterPoint and Tripwire declined to participate, saying they were between releases. Intelliden, one of the most established vendors in this space, also declined, citing a lack of resources.

All the products we tested approach conformance using rules about how configurations should be defined. These rules are compared by the configuration system against running and achieved configurations. For example, lines in a configuration that specify ACLs are compared against a rule or set of rules for that particular device. If a configuration running on a device or stored in the configuration system's database doesn't contain a rule-specified ACL, a rule-violation exception is logged and it triggers notification. ACLs are just one example: Each rule can handle any configuration requirement within a device-specific syntax, for instance, device password encryption or interface duplicity parameters (full and half duplex).One thing you won't be able to do with these products is create new configurations regardless of the manufacturer and type of device. They do not interpret configurations and try to determine if parameters are correct; they just compare the configuration to rules. And these configuration systems genearlly are configuration-syntax illiterate--in other words, they are clueless about applications. So there is no single, over-arching virtual language that can generate the specifics of a configuration, as was the promise of PBNM. Indeed, the complexity created by this lack of a standard interface to configure network devices contributed to the PBNM vendors' downfall. Today's configuration-system vendors must still deal with this complexity, but none of these systems tries to use a single template or rule that sets all ACLs (to use our example again) across all switches regardless of vendor.

Go Back!

Unfortunately, none of the products we tested could automatically roll a configuration back to an earlier version. The lack of an auto-undo feature does not mean that an earlier configuration version can't be sent to a network device and reloaded; it just can't be done automatically. We thought we'd be able to track authorized changes on a device and create two versions automatically. Then, if a change to a new version caused problems, a policy could roll back to an earlier version. But the vendors say this is not what IT wants. We understand that automatically changing anything to do with configurations is as much a trust issue as a technical challenge, but we'd still like to see it. What do you think? Your input matters because this is a quickly changing category of products. Cases in point: All three vendors released new revisions during our testing window. We obviously couldn't keep changing versions midstream, so expect some new features by the time you read this.

Indeed, these vendors are running a leapfrog feature race, as evidenced by the similar functionality all displayed. Gold Wire's Formulator and Rendition's TrueControl were neck and neck throughout our tests. TrueControl just edged out Formulator's security-focused, fault-tolerant appliance thanks to its intuitive interface and superior control over configurations, while still maintaining an open architecture. Voyence's VoyenceControl wasn't far behind, sporting an all-Java application interface and well-thought-out configuration creation. However, its price, control over configurations and ease of use did not compare as well.

All prices listed are for our test scenario. A breakdown of warranty, maintenance, professional services and new-device costs is contained in our pricing chart.Rendition Networks' TrueControl is very easy to use yet doesn't skimp on the control and creation of network-device configurations. Seemingly a completely different animal compared with Formulator, under the covers TrueControl's no-nonsense command-line roots and focus on helping operators make it a strong contender.

At the heart of compliance, for TrueControl and the other products tested, is the task of combining configuration syntax rules into a cohesive policy. TrueControl made it easy to get started on the compliance path. With its familiar Win32 navigation and logical design we quickly set up rules that tracked our configurations. These rules didn't just let us compare entire configurations, we also could check portions of configurations. For example, we checked passwords for every TTY and AUX interface. And we could set up a rule that said, "Check all routers to make sure that they have passwords."

TrueControl was the only product to let us retrieve configuration data from network devices based on a centrally configured frequency, similar to periodic polling. With the others we had to schedule batch jobs. Devices flagged for rule compliance were checked and alerted on when out of compliance. In addition to e-mail, external program execution and syslog notification, TrueControl provides a status-console-like display, with red-encoded devices showing a quick view of network-configuration compliance. Formulator also offered script execution, but TrueControl made this easy to specify as a notification option from within the user interface.

All the products we tested can gather network-device information, such as version, memory and storage used (and abused), but only TrueControl let us deploy network-device operating systems or images.

For reporting, TrueControl's easy-to-use GUI gave us a "home page" for publishing reports. Because the underlying storage is Microsoft SQL, we could send ODBC feeds to report writers, such as Crystal Reports.TrueControl, like Voyence, let us use templates to create configurations. We simply populated static and variable parameters via a scripting interface. On the downside, TrueControl can associate a single set of parameters with a single device only. In contrast, VoyenceControl can deploy to many devices.

TrueControl does sport, however, a very long list of devices supported, and like Formulator, is heavy in Cisco support. And, TrueControl was recently chosen for resale by Nortel. Not only does this give Rendition the inside track with Nortel device support, it forces it to go very deep in this area. Unfortunately, our switches from Alteon, owned by Nortel, were not yet supported. (Do we hear a whip cracking?)

Also missing from TrueControl is the specific notion of a baseline identifier for a configuration. Rather, configuration baselines are managed via policies and rules. However, custom fields are available in the database, allowing for text strings, much like Formulator's tags.

TrueControl 3.0. Rendition Networks, (888) 876-4626, (425) 636-2148. www.renditionnetworks.com

Gold Wire has had skin in the configuration-management game for awhile, having survived the PBNM shakeout. Its Formulator appliance has been reinvented and refocused on archiving, managing and verifying that network device and system configurations are in conformance. Gold Wire has begun integrating Formulator with HP OpenView and Remedy, looking to support context launches, traps and ticket-number tracking.

Authorization and workflow, important pieces of compliance control, are not part of Formulator--a key difference between it and VoyenceControl. But this lack points to what we consider a strength in the openness of the underlying Formulator architecture. Instead of enforcing work flow through Formulator's interface, other applications, like Remedy, can access configuration control easily and thus bolt on work flow--and its authorization process--to the control of configurations.Gold Wire has made some progress in making the GUI more navigable, but the strength of Formulator is that all its Perl-like underlying commands are completely accessible and secure. The command line and GUI use the same role-based authentication.

We added our devices to Formulator using automatic discovery, bulk CSV file load and manually. We found Formulator's autodiscovery on par with rivals--that is to say, mediocre--so auditing and editing after the fact are a must. That's why we prefer to simply load a CSV file that has the right device lineup.

Of note is Formulator's support for Unix and Linux servers. This is a strategic direction for Gold Wire, one we think makes sense. Even though only access restriction and tracking are supported--not OS or systems management--the feature is well-integrated. For example, we could log in to our Unix systems using a Formulator proxy, and OS and hardware information was added to the database. However, our OS configurations were not stored.

A powerful piece of Formulator is its ability to control configuration changes and track changes using proxy logons and keystroke logging. Indeed, all three products we tested support proxy sessions or logons (called "pass-throughs" in Formulator). This simply means the Formulator device server logs on to a network device for a user, granting access based on group memberships and roles. When we created a proxy session through Formulator to a switch or router, we didn't need to know the user ID and password. This is not device access control--limiting who can enter configuration mode in IOS. Rather, you're determining which devices can be accessed and whether configurations can be gathered or new configurations pushed to these devices.

In addition, when we were creating and tearing down proxy sessions, Formulator took snapshots of the configurations, providing us with comparison reports and detailed before-and-after access records, linked to the device and time. Included in the session detail when we established a proxy session was a complete record of our keystrokes, including all the keystrokes that took place while issuing configuration commands.The depth of Formulator's proxy includes daisy-chained sessions, where you connect from one router to the next, issuing a few commands to each. The beauty of Formulator's approach is that it associated our commands to the devices on which we actually issued them. In contrast, in the other vendors' daisy-chained proxy sessions, even though all commands--including telnet or SSH session commands--showed daisy-chained session establishment but the records for subsequent session keystrokes were associated with the session of the first device in the chain.

Like those in TrueControl, Formulator's reports are query-driven, and we found predefined editable queries in a number of categories, including audit, verification, configuration comparison, diagnostics and session log. Formulator's GUI made it easy for the SQL illiterate among us (ahem) to build and modify reports, and we could execute them at the command line or using the GUI.

For notification, Formulator includes the usual--syslog, SMTP--and is flexible. And, as we set up notifications, we found the trigger criteria we could select equally flexible. We unearthed tons of stuff in the database that we could use for unique notification treatments, including job tag/names, groups of users, groups of devices, external servers sending events to Formulator, session types, date ranges, event types and on and on. And, these attributes were also available as selection criteria when we were creating reports or looking for particular devices or configurations. We created geographical groups for our devices, for example, and those were available as filters to view particular configurations as well as groupings that could receive different event notification.

Formulator didn't try to edit and manage creation of configurations the way VoyenceControl did, focusing instead on managing deployment. Luckily, we found looking at differences between configurations and copying configurations for deployment to other devices easy breezy. For example, a common view of side-by-side configurations was highlighted with color-coded lines--red showing deletions; green, adds. Formulator's device support is on par with Rendition's and better than VoyenceControl's.

Formulator's underlying command-based application is accessible without messing with the GUI, if that's your cup of tea; every function is command-line executable. Although the GUI doesn't present predefined views of devices and their status or customizable lists of reports, devices and views in a personalized space, it does pull its weight in terms of easy access to forms, pull downs and lists for command creation. The advantage of this is, if a parameter exists in the database and a report on it is needed frequently, it's doable without too much difficulty in the GUI. In addition, the CLI is well-documented, and when we created a command within the GUI, it was displayed in command form so that it was easy to execute--even we were able to get specific syntax correct.Formulator was the only appliance we tested, and we liked its high-availability features and automatic failover. The primary advantage of the appliance is Gold Wire's certification that it's hardened. Moreover, Formulator is the lowest priced entry, a rarity for an appliance. Pricing is figured on the base platform plus a per-user license fee; device modules are licensed on a per-device basis.

Formulator 200 3.4. Gold Wire Technology, (888) 585-9473, (781) 398-8800. www.goldwiretech.com

VoyenceControl's scripting and configuration templates offer automation--and thereby conformance--of network configuration management. Voyence has focused on the processes that IT performs in order to maintain configurations, attempting to automate and thus codify production networks. VoyenceControl's automated, modular scripting and template tools will enable configuration life-cycle creation, maintenance and auditing--after your engineers invest some time creating custom templates. Shrink-wrapped templates are few, but according to Voyence, fitting the product's automation to your environment is part of the training and implementation process. We think that building template libraries is a process that Voyence could improve over time.

Once created, templates can be linked, creating a single source for configuration updates, but also modularizing portions of configurations. These über templates can be maintained separately to represent best practices and linked into a configuration when it is moved to a network device. This separation of tasks will support the separation of functions required for conformance.

Audits are accomplished using what Voyence calls term-letts. These scripts, which we managed using the same scripting environment we used for templates, can execute most any command and let us collect and compare configurations, to prove what we thought was pushed out onto the network actually was. We could maintain a consistent network configuration by periodically running term-letts to check and compare what was against what should be.

VoyenceControl, like rivals, employs rule-based compliance, but Voyence breaks away from the pack with its automated configuration deployment. As with the other two products, rules requiring or forbidding particular configuration statements-such as not allowing the addition of a new OSPF area--are supported. We were able to automate deployment of a batch of configurations.Unique to VoyenceControl is the ability to extract from and insert into configurations device-specific data, like IP addresses. This is GUI-driven and not difficult. Automatic data insertion is accomplished via CSV files, which VoyenceControl appends with variables assigned to configuration files. In this way, each line in a CSV file can correspond to device particulars, making it easy to audit and change variables in network-wide configurations. Meanwhile, subsection configuration leveraged an Explorer-like paradigm from within the Java GUI, letting us pinpoint sections of configurations relating to interfaces, routing protocols and ACLs, for example. That made it easy to navigate long configs.

We also created workspaces in VoyenceControl where we could edit and house our configurations in progress. This provides coordination between projects that affect configurations. None of the other vendors had similar predeployment--and visually represented--containers. Formulator and TrueControl do, however, have flexible grouping functions, which give you multiple sorts of the same device.

VoyenceControl's more advanced device-configuration manipulation provides a lot of automated control, and puts policy enforcement in the network engineer's hands. The other vendors downplayed the importance of this kind of automation, stating that their customers are nervous about the accuracy of the data that would automatically populate these configurations, but we feel that this is a process, not a technology problem, and should be tackled as such.

VoyenceControl was also the only product to present us with an authorization function as part of its batch job process. When we were ready to deploy a configuration it was queued as a job using a scheduler. Before the job can run it has to be authorized, a function that can be assigned via role-based security permissions. Although we like this idea, we question the likelihood that an IT organization will want to use a configuration product to manage job authorization. Our experience tells us this process is likely to live in a service-desk application--if it's a process at all.

We could access VoyenceControl's reports online, in hard copy or through customized portals. The canned reports, Change Audit Report, Connection Report, IP Address Report and OS Report, are straightforward and schedulable. No e-mail publication of reports and no report writer go on the minus side of the ledger. Voyence says its next release will address this.VoyenceControl also had respectable device support, again long and wide in Cisco. The company was willing and able to create new drivers very quickly: When our Alteon routers popped up as unsupported, within a couple of hours Voyence had created a device driver for us, and they said this is SOP, not special treatment. The company also does not charge for the service. Not bad!

VoyenceControl is licensed per device under management. For small networks, the price per managed device is $250, regardless of type. The minimum system price is $20,000. Quantity discounts are available, and there is no incremental licensing fee for the device servers. There is also no per-user license fee. While there is generally no requirement for additional application servers, if you choose to incorporate one, the price is $20,000.

Unique to VoyenceControl in this review were topology maps that showed Layer 3 connectivity, representing the current state of the network. This is similar to what we've seen from HP's Network Node Manager or Aprisma's Spectrum network-management system. If Voyence offered Layer 2 connectivity maps based on configuration and traffic analysis (a hard task to be sure), then we'd get excited. As it is, Layer 3 mapping is a nice addition, but nothing to write home about.

The Java-based VoyenceControl console is a bit clunky. In addition, the documentation was less thorough than we'd have liked. Fortunately, Voyence includes training in its purchase price, and we suggest you take them up on the offer.

VoyenceControl, $107,000 (as tested). Voyence, (972) 931-3444. www.voyence.com BRUCE BOARDMAN, executive editor of NETWORK COMPUTING, tests and writes about network management and systems. He has 12 years' experience managing networks and distributed computing for a financial service provider. Write to him at Bruce Boardman at [email protected].

"The Survivor's Guide to 2004: Network and Systems Management,"

"Layer 2 Discovery,"

"Network Monitoring for Small Budgets,"

FYI: Sixty percent of all network performance and availability failures are the result of inappropriate configurations, according to analyst firm Enterprise Management Associates.Configuration-management vendors are looking to make running a heterogeneous network less laborious. These tools don't have as big a bag of tricks as the now mostly defunct PBNM crowd offered, but they won't cost you the earth, either. Rather than try to create new configurations across a range of devices, the products we tested from Gold Wire, Rendition Voyence approach conformance from a rules-based angle: First, you define how configurations should be, then they raise a flag when an exception is noted. After weeks of testing, Rendition's TrueControl edged out Gold Wire's Formulator appliance by the thinnest of margins thanks to TrueControl's easy-to-use interface and exceptional configuration control. Voyence's VoyenceControl made a respectable showing as well. Your needs and in-house skill sets will dictate which is best for you; we'd be happy to make a home for any of them in our labs.

IT regulations do not lay out cookie cutters for compliance. Rather, they are usually broad, loose guidelines, like ISO 17799, or laws that dictate an end to meet but don't specify the means to that end, such as Sarbanes-Oxley Section 404. Although the regulations that implement these laws may provide a little insight, your implementation must comes down to reason and reliability.

For example, in Section 404, management is required to maintain an internal control over financial reporting. That control must provide "reasonable assurance" regarding the reliability of financial reporting and the preparation of financial statements for external purposes in accordance with generally accepted accounting principles. It does not specify that the control needs to be, say, a DBMS or some other technology, only that it must provide a "reasonable assurance" of accuracy. In other words: Reliable.

In the future, courts will sort out what is reasonable assurance and what is not based on the facts of cases where investors are led astray by misleading financial reports. Our assertion: It will be more than JBOLF (Just a Bunch of Log Files) and less than a complex knowledge management system. --Sean Doherty

We pointed each configuration-management system at our test network . First, we determined what access community strings and telnet passwords were defined on each device. Because these systems were supporting other tests by editors in both our Syracuse University and Green Bay, Wis., Real-World Labs, there was little consistency in configuration. So, testing happened on our production systems, and interruptions caused co-workers to get grumpy, just like in the real world.Device types were primarily switches and routers, but we did include a Cisco firewall and Sun Solaris server, just for fun. The device list included items from Nortel, 3Com, Foundry, Extreme, Juniper, HP and Cisco. Representative, but by no means exhaustive.

We applied access security and automated nightly configuration backups. We also created new configurations, compared running and new configurations, and reported on all this activity. We exercised access-control capabilities by creating user and device groupings with varying levels of access. We poked at each system's graphical interface and spent some time using the command lines in Formulator and True Control. Voyence uses Java only.

R E V I E W

Configuration Managers

Sorry,
your browser
is not Java
enabled

Welcome to

NETWORK COMPUTING's Interactive Report Card, v2. To launch it, click on the Interactive Report Card ® icon

above. The program components take a few moments to load.

Once launched, enter your own product feature weights and click the Recalc button. The Interactive Report Card ® will re-sort (and re-grade!) the products based on the new category weights you entered.

Click here for more information about our Interactive Report Card ®.

0