Testing Across the Miles

Our test bed comprised a diverse group of about 100 devices in both our Syracuse, N.Y., and Green Bay, Wis., labs. Remotely managing most of those devices was a breeze, thanks to the products' behind-the-scenes support for telnet, SNMP, SSH and TFTP for access. In fact, device support in general was no sweat. As you can see in our comparison charts on pages 68 and 70, the configuration managers we tested have grown to support switches from most major vendors; our leaders even handle firewalls, VPNs and load balancers. There hasn't been much movement toward combining systems and network configuration management, but some do gather basic Linux and Unix systems information. Opsware's Network Automation System and AlterPoint's DeviceAuthority support MySQL, Microsoft SQL and Oracle, but CatTools lacks database support.

Policy enforcement and production control have taken the biggest leap since we last looked at this product category (we tested budget-priced offerings in "Controlling Change, Cheaply,", and enterprise-class suites in "Driven to Conform,"). For example, AlterPoint's DeviceAuthority and Opsware's Network Automation System let us define policies and ensure compliance with them.

Policies are defined states--desired or undesired--associated with devices. A simple example would be to check for "public" SNMP community strings, and when found, notify operations, management and the responsible network engineer of the violation. Polices are not limited to this simple parsing; they also can include checking running configurations and any information stored in a database associated with a particular device's inventory record. This might be a specified version of a configuration, hardware, OS, custom asset fields, or organizational determinations, such as "edge switch."

DeviceAuthority and NA System have made strong advances in this area. For example, both support configuration autoremediation. It's possible to change configuration-policy violations automatically without operator intervention, but most network engineers aren't comfortable with this level of automation .

We also found that these systems have become easier to use, so much so that both Kiwi and Opsware scored perfect 5s in this category, which we graded based on interface navigation, available help, multiple paths, shortcuts, window clutter (actually, lack thereof) and ease of remote usage. Opsware's NA System is more complicated than Kiwi's CatTools, but NA System exposes most functions near the surface, making navigation simple. We loved having a "My Favorites" function on every screen so those well-worn paths didn't wear out our clicking finger. In addition, NA System offers a search function on every page that allowed for quick connections to a device through the proxy telnet or SSH, and let us find just about everything--not just devices, but modules, diagnostics, tasks, sessions, events, users and ACLs.Meanwhile, CatTools is aimed at the network engineer and small and midsize businesses. Inexpensive and easy to use, it has solid configuration management. We didn't find policy management, compliance reporting or workflow functions, and it doesn't support database repositories, but it does have a decent list of supported devices (not all Cisco!) and solid diagnostic and configuration-management tools.

DeviceAuthority trailed a point behind the ease-of-use leaders mainly because we had to deal with two interfaces: a Java Update interface and a separate HTML Audit interface, running on Win32 and IE 6, respectively. The HTML interface was cleaner and easier to use, but you must create update jobs and perform certain administration and configuration-management chores in the Java app. The Java app will run the HTML interface within a window, and the HTML interface uses the same authentication as the Java app, but session information is maintained separately. So we had to log back on to both interfaces after time-outs--an annoying process made more tedious when the HTML interface would spawn a new window, say, for a report, to which we then had to authenticate.

Although subpar ease-of-use might not be a deal-breaker in shops with experienced network administrators, more-intuitive GUIs and systems administration will generally translate into lower care and feeding costs. Speaking of price, check out our pricing chart on page 75 for a detailed look at what these products will set you back.

Compliance is a selling point for almost everything IT is purchasing lately except maybe new mousepads, and Opsware's NA System is no exception. Its Compliance Center can help crack the reporting nut by orienting all of its (considerable) reporting in relation to Sarbanes-Oxley, COBIT, COSO, ITIL, GLBA or HIPAA. Summary and more-detailed links gave us explanations of what each standard requires and how they are interrelated. We were impressed, but then, we haven't had a lot of experience with some of these reporting types. So we asked some reference customers that are knee-deep in compliance reporting and found that they weren't as excited as we were, mainly because they were, well, knee-deep in compliance audits and had learned what it takes to placate auditors. They did, however, appreciate the cross-referencing of reports to various areas outlined by these standards.

The three participating vendors are a good representation of the state of this technology. Kiwi's CatTools is easy to use, straightforward and a great value. It's not going to manage policies or report on SOX compliance, but it will keep your configurations safe and let you manage the network without any heavy lifting. AlterPoint's DeviceAuthority includes policy management, a single-sign-on proxy and serious database support, making it a top contender.But our Editor's Choice goes to Opsware's NA Systems for its excellent production control, configuration management, reporting, ease of use and architecture. It was awash in unique features as well, and we don't mean just its compliance angle; for example, its device-reservation capability let us create multistep tasks that prevented other update jobs from running against targeted devices. That meant other jobs wouldn't try to update the same devices. This may not avert much heartburn in smaller shops, where most of the work gets done by a few network engineers, but it's easy to see how such coordination would eliminate head-scratching in large and decentralized networks.

We think Network Automation System is the best network-configuration product we tested. Its feature set, though strong overall, really excelled in production control, reporting, ease of use and architecture.

Like its rivals, NA System let us use tasks to accomplish our configuration- and device-management goals, including deploying passwords, building backup configurations, updating device OSs, running diagnostics and issuing CLI commands. One task unique to NA System is the automatic synchronization of running and start-up configurations. All we had to do was point to a device or group of devices and run the task.

Tasks can be scheduled or run interactively. Scheduling recurring tasks is a breeze--we could set schedules weekly by day, for a specified number of iterations, and on a particular day of the month. The only options we didn't see were end of month, end of quarter and end of year. Also unique to NA System is a calendaring feature that let us view all the jobs we scheduled on any day. Call us demanding, but we want views by week, month and quarter as well. These batch features mean you won't have to come in at O'dark-hundred to make updates--instead of getting time and a half, you'll be investing in more beauty sleep.

NA System was the only product to include a workflow engine for configuration and OS updates. Setup was wizard-based and simple. We created workflow rules for our Syracuse and Green Bay labs, each with a different set of devices and users. We accomplished this using basic roles--originator, approver and FYI recipient--that combined with groups of devices and specific actions to create workflow rules. Within the workflow, we could assign an impact priority to tasks requiring approval. These are user-defined, but low, medium and high values are configured on NA System by default. When we submitted a job, we also rated its impact. Approvers can disallow any task they deem too risky.

ACL management is one of NA System's best features. As an option, ACLs can be parsed when a supported device is added to inventory. Unfortunately, our Cisco 7401 wasn't supported, but our 7200s were. With Opsware's help and a bit of luck, our 7401 managed configurations using a 7200 driver, and we began parsing out ACLs. Once that was done, we could search and comment on specific ACLs. Being able to add inline comments--unique to NA System--not only let us annotate directly above each ACL process, it also kept comments persistent even when the ACL was changed, creating an annotation for workflow and historical audits. But the best part is that we could insert a single line into an ACL for a single device or insert an ACL change across a number of devices. The latter function is supported by a "Batch Inset ACL" task that then changes a line for a particular ACL number, or handle (a name assigned to a group of specific ACL numbers). This capability means an ACL can be deployed across different device types consistently, regardless of the number assigned.

With the ACLs parsed, we could add single-line updates and push them to multiple devices using included batch insert and removal scripts. ACLs support both security and QoS (quality of service) applications.

Events, like other network-management applications, are used to alert operators, network engineers and management to failed and successful activities on network devices. We found the usual notifications, including e-mail, executables, logging and status indicators on the management console--red for a problem and green for everything's okey-dokey.NA System's SingleView event viewer is simple to use. Scrolling lists of all event types, times and device groups made it easy to focus on key events. Once we created an event view, a single click created a CSV file for quickly loading events into a spreadsheet. And we weren't limited to event viewing: Every view includes an "add to favorites" selection for quick access to any part of the NA System interface. The only downside was it didn't save our event filters in this link; rather, the link showed the last filter used.

Events displayed in SingleView are configurable, and all event types didn't have to be on, or even logged. We controlled notification and enabling of events from within the administrative menu. In fact, NA System can track users when they log on to a network device. Users can be denied access to a device through a proxy logon, a capability shared by all the products. NA System takes it a step further, parsing the user name from a syslog message and/or adding the user to the system automatically, thus at least attempting to ensure attribution of externally made configuration updates.

Policy management ideally lets IT maintain the state and security of network devices. NA System does both. By creating rules that we then applied to groups of devices, we ensured that rule compliance or violations were reported. And we could return to a previous configuration version if necessary.

In both NA System and DeviceAuthority, rules and the devices to which those rules apply are the basis for policies. Setting a policy was easier in NA System than in DeviceAuthority thanks to Opsware's rule-creation wizard, though the way DeviceAuthority separated the rules from the polices made rule reuse simpler.

NA System uses regular expressions to match configuration text in policy rules. For example, we wanted to check for consistent SNMP community strings. We reviewed each SNMP configuration for public strings as well as our desired string.

Reporting in NA System is impressive: We found everything from a network-health overview to compliance hand-holding to more inventory and system status reporting than you can stick with a shake. This is helpful when auditing a network for the correct OS versions to support BGP or VoIP, for example. We especially liked the unique network-status report--this graphical view gave us a quick heads up when things went awry. An overview pie chart summarizes risk percentages, split among low, moderate and high risk. A score reflecting best-practice health info summarizes configuration status as OK or in violation; trouble areas include Policy and Software Versions. For example, NA Systems includes CERT advisories about OS versions; we could choose whether to designate our OSs as compliant versions. If an OS that wasn't defined by us as compliant showed up on the network, we got a heads up via a compliance report. Other handy reports include unsynchronized running and startup configurations, device-access failures and configuration changes in the past 24 hours. These are broken down by the specific occurrences or devices that led to violations. We liked the way this report sorted by device groupings so we could zero in on the health of our little corner.We found more than 40 canned reports covering devices, tasks, sessions, compliance, workflow and overall stats. We e-mailed a few reports to ourselves on a daily basis. Reports came as HTML, with links back to specific device events on the NA System server. Sweet!

NA System's architecture claims support for many third-party management vendors, including BMC Remedy, Computer Associates Unicenter, Hewlett-Packard OpenView NNM and Service Desk, IBM TEC/NetView, Micromuse NetCool, Nortel Networks Optivity and Smarts inCharge. This support included bidirectional traps and bidirectional service desk integration. Support for authentication services was the widest of the products tested and included TACACS+, LDAP, Active Directory and RSA SecurID.

We were most impressed with the monitoring and management of system functions, including HTTP, LDAP, memory, SMTP, syslog and TFTP services. The admin and user interfaces are one in NA System, with access restricted by user role. Administrative control over system services includes a view of each service's most recent status, and we could rerun the status check interactively. Finally, we could configure warning and error thresholds for available disk space; available RAM; and millisecond delay for TFTP, SSH andsyslog messages on the NA System.

Opsware Network Automation System 4.0. Opsware, (408) 744-7300. www.opsware.comDeviceAuthority should be on your shortlist thanks to its strong features, even though it trailed NA System in production control and reporting. DeviceAuthority has all the basic configuration-management bells and then some. We were impressed with the well-laid-out hardware and software inventory--device parts, like interfaces, CPU, flash storage and software images, along with running and start-up configs, are displayed in an Explorer-like hierarchal tree with right-screen details, easing navigation. Tasks will run basic CLI commands or Perl, TCL and Java.

A highlight is DeviceAuthority's conditional job processing, which it refers to as workflow and validation. For example, we specified a job to check if our targeted devices were running IOS 12.2x. If they were, we ran the job and if they weren't, we failed it.

DeviceAuthority doesn't have its workflow-approval process built-in like NA System does, but it claims third-party integration with service desks from HP and BMC. Hourly, daily, weekly and monthly job recurrences are options, but we missed being able to calendar more business-oriented schedules, like end of month or every other Tuesday.

The product's event-management capabilities include myriad types of notifications, flexible filtering and event de-duplication. All tasks support triggering events that fail, warn or succeed, and we could choose to see, for example, only failing tasks. In addition to task-related events, DeviceAuthority receives events from TACACS+, RADIUS, HP OpenView and SNMP traps. The event viewer provides a filter that we used to limit based on time, type of event, address, host name, make, model and class. We wanted a filter based on our device groups as well.

DeviceAuthority organizes policy creation into rules and policies. Rules are specific actions or desired states, and policies are a collection of rules and the devices to which these rules are to apply. This clear separation, not found in rivals, was a huge help in auditing how rules are applied. Even navigating through long lists of rules was a breeze using the DeviceAuthority model.

DeviceAuthority's Network Intelligence Model (NIM) defines device attributes in categories including ACL, chassis, interface, Spanning Tree, static routes and VLAN. We used these NIM attributes throughout DeviceAuthority to create dynamic target groups for reporting, configuration and software management, and they made it easy to define the right database field when looking for a set of devices. For example, we created a policy that required 128 MB of RAM in routers running 12.2* IOS, then applied that policy to the Cisco routers in our lab.Reporting covers configuration, hardware and software changes, along with inventory, credentials and policy. Output options include HTML, PDF, XML, CSV and text. We could use a number of preconfigured reports, or we could custom build them. The whole process is wizard-driven and let us select custom search criteria to populate the report or use the groups of devices we had already created. We set up periodic reports with e-mail distribution; these reports sport hot links to events on the server. But unlike NA System, there is no overview "health" reports or compliance hand-holding.

DeviceAuthority isn't hard to use, just a bit tiresome now and again. As we mentioned, the suite required us to hop between two interfaces. This setup has its flaws, but on the upside, right-click context and function jumps are comfy navigation hooks that the DeviceAuthority Java app has over the pure NA System HTML. And when we had to update our license, DeviceAuthority handled the entire function from within the Audit browser. No FTP of the license file, and no system console access required. NA System required us to copy the file to the system.

DeviceAuthority has good architectural support for databases, third-party authentication services and role-based access, but not as deep as some of the options available on NA System. For example, we set up users and groups with access roles, and limited particular device groups to specific devices. All good, but the roles were static, and even though they were well-documented and thought out, we couldn't find a noncoding way to create new or custom roles. However, we did talk with one of DeviceAuthority's major users (sorry, can't disclose the name, but think really big bank). The company said it has very stringent security requirements--including temporary centrally controlled passwords as well as tokens--and found DeviceAuthority a good fit.

Device support is good. And like NA System, DeviceAuthority offers good third-party management support, including BMC Remedy, CiscoWorks RME, ConfigureSoft ECM, HP OpenView, IBM Tivoli TEC/NetView, and Secure Elements CLASS 5 AVR.

DeviceAuthority Suite 3.5. AlterPoint, (888) 228-3422, (512) 536-8300. www.alterpoint.com

Easy is as easy does" sums up CatTools. It lacks an HTML interface (though Kiwi says it's working on one, along with a SQL database), but the product's Win32 interface is a pleasure to use. Defining the entire functional scope are simple tabs, including those for device, activity log, info log, display, compare, report, TFTP and activities. CatTools finished last because it didn't have big-scaling, role-based access, third-party integration architectures.With CatTools, we could schedule predefined connectivity, configuration and report jobs or tasks. Scheduling is very flexible, with fixed recurrences like minute, hour, day or week, as well as custom intervals. We set up tasks to run every day at midnight, excluding weekends, for the next six months. Any scheduling calendar can be saved and applied to different jobs, avoiding redefining the same frequency over and over.

Connectivity tasks check device status using pings. We provided a list of devices we wanted pinged from a second group of devices, giving us connectivity status from various points in the network.

When testing configuration tasks, we could back up and update configuration files as well as run CLI commands on a device or group of devices--static and dynamic groups are supported in CatTools. We created dynamic groups using a filter function, which let us select two different device attributes.

Event management lacks DeviceAuthority's de-duplication and NA System's ability to reach outside system-created events to assign user responsibility, but basic monitoring for changes and reporting on events as a result of system tasks works well. There's no event view; tasks we ran created a log, showing error counts for failures. Each task in the log linked to the results for it.

Kiwi CatTools supports configuration backup on all the devices listed in the device support chart. Support for the automated entering of commands to the CLI and configuration is available on some devices, including those from Cisco, Extreme Networks, Foundry Networks and 3Com, as is support for the collection and reporting of port, ARP, MAC, VLAN, DNS and version information.

CatTools' reporting is simple text or HTML, with reports covering MAC address, ARP cache, port/MAC mapping, IOS software/hardware and interface error counters. Once we scheduled the reports, they were e-mailed to us and stored locally on the CatTools server.

Kiwi says large organizations like Cisco, HP, IBM, Microsoft and 3Com use CatTools to manage thousands of devices, even though the vendor had targeted the product for the midmarket (100 devices or so). Version 3, due for release in about six months, will feature more enterprise-class bells and whistles, Kiwi says, including MySQL, Microsoft SQL and Access database support, a Web front end, and an even easier-to-use interface. We'll be keeping an eye out--any company that does this good a good job at the basics is worth watching.Kiwi CatTools 2.2 Enterprise. Kiwi Enterprises. www.kiwisyslog.com

Network configuration management has always been a great concept. IT operations suck up between 50 percent and 80 percent of the IT budget, according to Gartner. So managing changes that affect topology and transaction flow is key to reducing complexity and cost. But in the real world, it's been a challenge to control even tested and tracked changes that went through production channels. Forget managing one-off tweaks or keeping track of security-device reports.

Time to lose the negativity. Vendors have mastered the basics, such as tracking and limiting change, providing access to those who need it when they need it, and automating repetitive tasks, such as changing passwords and ACLs (access-control lists), and are adding slick features, like reporting on the compliance aspects of network configuration and managing more than just switches--wireless devices, firewalls, load balancers, even VPN gateways, are fair game.

We tested AlterPoint's DeviceAuthority, Kiwi Enterprises' Kiwi CatTools and Opsware's Network Automation System in our Syracuse University Real-World Labs. Kiwi's affordable CatTools is easy to use but lagged behind its rivals in more advanced functions like policy management. DeviceAuthority let us manage policies and more, but Opsware's NA System took our Editor's Choice; we liked its comprehensive feature set and its raft of unique features, including compliance reporting.

We set up the products under test in our Syracuse University Real-World Labs® and managed about 100 device configurations from both the Syracuse lab and our Green Bay, Wis., business-applications lab. This gave us a healthy mix of Cisco Systems, Extreme Networks, Hewlett-Packard and Nortel Networks devices to target as we backed up, changed and updated configurations and software across these devices.We monitored for changes and set up groups of devices, policies and users to determine how well the products worked in larger federated environments. All three vendors pointed to actual deployments of thousands of devices as evidence of their scalability. We talked with reference customers who affirmed that they were indeed installed and managing these numbers. Still, if size matters to your group, you should do your own due diligence. We know you will.

All Network Computing product reviews are conducted by current or former IT professionals in our Real-World Labs® or partner labs, according to our own test criteria. Vendor involvement is limited to assistance in configuration and troubleshooting. Network Computing schedules reviews based solely on our editorial judgment of reader needs, and we conduct tests and publish results without vendor influence.

R E V I E W

Network Device Configuration Management

Sorry,
your browser
is not Java
enabled

Welcome to NETWORK COMPUTING's Interactive Report Card, v2. To launch it, click on the Interactive Report Card ® icon above. The program components take a few moments to load.

Once launched, enter your own product feature weights and click the Recalc button. The Interactive Report Card ® will re-sort (and re-grade!) the products based on the new category weights you entered.

Click here for more information about our Interactive Report Card ®.