The Evolution of Patch Management

Patch management is a nightmare. Critical patches are announced at the whim of vendors. Security and operations teams must drop everything to close holes in software before attackers exploit the vulnerability. Even in the best of circumstances, patch management requires close cooperation across operational disciplines that include security, operations, applications, and business units. Patches must be tested to ensure that they don't affect essential business systems, tracked to ensure that they've been deployed, and reported on for executives and auditors who want bottom-line summaries of risk posture and compliance.

Patch management products can provide immediate relief, but a new trend is emerging that folds patch management into a larger security or configuration management system. Pure-play patch management vendors that don't respond to this trend will find themselves marginalized, whether by Microsoft and its automated patching systems, or by established software distribution and asset management vendors that are adding patch management to a larger portfolio of security and configuration management features.

Security and configuration management systems deliver patches, but they also assess and monitor the overall status of an asset, including applications running on the machine, allowed services, open ports, and so on. These systems track changes and remediation efforts and continually monitor the state of the assets to detect machines that fall out of compliance.

In general, what sets the two systems apart is the vendor doing the selling. Companies that tend to sell to security architects emphasize security, while those that sell to IT operations teams emphasize configuration. The thing that really counts, however, is the management.

That's because the key to a more secure environment depends as much on process as it does on products. In some cases, a product will enforce an ad hoc process on an organization, but the business is best served when all the units responsible for the health of the information systems establish explicit protocols for addressing the issues related to patch and configuration management.To illustrate the importance of establishing explicit patch management protocols, we'll examine how two organizations face the daunting task of configuration management. WesCorp, the nation's largest corporate credit union, has deployed a process-driven approach in which patch management is part of a larger security management system. Within this setup, clear guidelines and procedures articulate the steps taken to address the overall security of the company. Meanwhile, Hannaford, a grocery chain with 140 stores, relies on a product-driven approach to solve the immediate need to secure the organization. We'll also look at how patch management products are evolving into broader configuration management systems.

THE $24 BILLION QUESTION

If you could sum up Chris Hoff's job in one question, it would likely be, "How much risk am I facing today?" For Hoff, it's a $24 billion question. That's the sum total of assets he's tasked with protecting as the director of enterprise security services for WesCorp, the largest corporate credit union in the United States.

To help him answer that question, Hoff has an extensive set of tools at his disposal, including a vulnerability management service from Qualys, a risk analysis system from Skybox Security, and a collection of patch management products, including PatchLink and Microsoft's Software Update Service (SUS). He's also considering an enterprise vulnerability management solution called Hercules from Citadel Security Software.

Each tool plays a specific role in answering the essential question. Qualys scans the network, reports on vulnerabilities, and can set priorities based on the severity of the vulnerability and the value of the asset. That information is then fed into Skybox View, along with router and switch configurations, firewall rules, and a network map that describes the links between each device on the network. Using that information, Skybox can show Hoff which assets are exposed and how effective the intervening controls are or aren't.From there, Hoff and his security team can prioritize which assets need further protection, whether from patches or from configuration changes on the vulnerable systems or intervening devices. That information is passed to the IT operations team, which uses the patch management products to deploy software fixes, ensure that those fixes have taken, and report on completed jobs.

These tools are essential in helping Hoff answer his question, but more importantly they represent only part of a larger process he has implemented, a process that extends across the key groups that exercise influence over the information infrastructure.

"The real chore is what happens when you throw [remediation] over the fence to IT operations," says Hoff.

The reason is that mitigation involves many people in the IT organization, so workflow issues complicate the process. "Four or five different units have to sign off on a patch," says Hoff. "There's a tremendous amount of work."

However, that work has a purpose. For Hoff, it begins with his security group, which conducts vulnerability scans, evaluates vulnerability and exploit data, and narrows actionable items into a set of critical exposures on critical assets. This information is then passed on to the IT operations team.But before IT operations can begin their work, the security team's analysis has to get past the application owners and business units. These units can approve or deny any remediation strategies, including patches, based on how those remedies will affect the business. If a patch is denied, Hoff says the business unit must sign off that they understand and accept the risks associated with not patching the system. "Business units don't like it when you make them accept a business risk because they get the blame," says Hoff.

Once patches and remediation strategies are approved by the relevant groups, Hoff says the IT operations team works within a service level agreement negotiated with the security group to ensure that changes are made in a reasonable time.

This workflow is tracked via a ticketing system. Tickets with specific patching duties are assigned to the operations team so that the work can be tracked from initiation to completion. The tickets can also reference all the documentation required from each unit in the organization. "At any point, someone can get a clear picture of what they are doing," says Hoff.

HAVE PRODUCT, NEED PROCESS

Aaron Merriam, a systems service specialist for Hannaford, a New England grocery chain, has his hands full."When I started five years ago, there were 12 Windows NT servers and now there are 600. Many have vendor-specific applications, so it's a nightmare to manage," says Merriam. Hannaford also has about 3,900 desktops that add to Merriam's patching burden. Before turning to a tool to automate deployment, Merriam created and distributed patches manually. Sometimes patches failed to take, exposing the company to virus outbreaks.

"By the time we got hit with the fourth major virus, we started looking at patch vendors," says Merriam. "We're short-staffed, so we wanted something easy to use and configure. It needed to update patches automatically and list the patches that a machine has or needs."

That tool turned out to be BigFix. Before purchasing the tool, it took half a day or more just to distribute one patch to all the machines that needed it. With BigFix, it takes 10 to 15 minutes. He also likes that BigFix can track the status of the anti-virus clients on the desktops. It has a plug-in to tell which machines don't have anti-virus installed, which definition services are set to manual, and which ones are out of date.

But while BigFix rode to the rescue, Merriam knows there's still more to do. "We've got the tool and it works well, but it's the process we're struggling with."

The source of that struggle is that other business units worry that patches will disrupt business operations.To be fair, those fears may be justified because Hannaford doesn't have the staff to do rigorous testing on software fixes. "We'll look at the patch description and compare it against our infrastructure," says Merriam. "But critical patches generally just go out."

At this point, Merriam says there's no clear policy in place that gives one group or another final say over a change. Disputes between himself and the applications group have to be mediated by supervisors, which complicates his ability to deploy patches during regular maintenance windows. "Until a hard and fast policy is on a piece of paper, we're in limbo," says Merriam.

However, he notes that compliance issues may make patching a higher priority. Regulations such as Sarbanes-Oxley, in which controls to ensure the integrity of information sources must be audited, give Merriam more clout. He also says the IT department is creating a new position to put one person in charge of change and configuration management. "We're going to a process where the security team approves patches, and the operations team rolls them out," says Merriam.

POLICIES IMPROVE AUTOMATED PATCHING

Dozens of products are available to help with the problem of patching. Some products begin from a patch deployment perspective, while others are born from an asset tracking or systems management perspective. What they all have in common is a move away from simple patch automation toward policy-driven monitoring.For instance, with an automated patching tool you associate a patch with a specified group of desktops or servers, and the patch is deployed. If the patch requires a system reboot, the auto-patching tool may give the administrator the ability to set a reboot time that won't affect business operations. This is where strict auto-patching tools stop.

But what if a user or program uninstalls the patch? If you aren't monitoring the system regularly, you won't know until your next scan. Although you can redeploy the patch, that asset was vulnerable during the window between scans and may become vulnerable again if other changes are made between assessments.

Compliance monitoring, which is usually agent-based, can enforce policies continuously. If a patch is uninstalled, the compliance monitor will detect the change and alert an administrator. This reduces the window of vulnerability because non-compliant machines can quickly be brought into line again. Administrators may also be able to detect the root cause of non-compliance--for instance, by finding out who keeps uninstalling the patch.

THREE KINDS OF PRODUCT, ONE GOAL

Three major product categories are now driving toward policy-driven monitoring: patch management, security management, and software configuration management. However, each category is incorporating capabilities from the others, so the distinctions between the three are blurring rapidly.Patch management products such as those from BigFix, PatchLink, Shavlik Technologies, and St. Bernard Software emerged with a laser-like focus on analyzing, distributing, and tracking patches. These products are typically employed by organizations that are overwhelmed by the burden of deploying patches manually, or that have been disappointed by an existing software distribution tool that didn't deal well with patches. Patch management products grew out of a need to automate patch deployment.

But these vendors realize that the days of patch management are numbered as pressure mounts from two fronts. First are the established software distribution and asset management vendors that are adding patch capabilities to their products. Second is Microsoft, which includes patch management in Systems Management Server (SMS) 2003 and Windows Update Services (WUS). Microsoft's offerings cut the bottom out of the market and will force pure-play vendors to evolve or become marginalized.

Hannaford's Merriam echoes an opinion that would make any patch management product manager cringe. "One reason we went with BigFix is it's a point solution. If at the end of the year some other configuration management product comes out, we'll just dump this."

To that end, patch management products such as BigFix are pushing hard into overall security management. Using an agent-based architecture, BigFix lets administrators distribute software, start or shut down specific services, close file shares, track software licenses, and change registry and file settings on host machines.

Security management products from companies such as Citadel, Configuresoft, BindView, and Ecora Software offer patch distribution as part of a larger focus on security issues. For instance, Citadel's Hercules product is an agent that resides on desktops and servers. The agent monitors the asset for five general elements: patches, configuration errors, unsecured accounts, backdoors, and unnecessary or unwanted services.Finally, there are the software configuration management products. This group, represented by vendors such as BMC, Altiris, and LANDesk Software, got their start as software distribution tools. These tools approach asset management from a policy-driven perspective, in which agents monitor the OS, applications, and content for deviations from a standard baseline. If a machine goes out of compliance, administrators are alerted. In some cases, automated remediation capabilities will kick in.

BMC's Marimba includes a suite of products, such as OS Management, Application Management, Patch Management, and Configuration Discovery, which can be purchased la carte or as a set. It also ties into BMC's popular Remedy ticketing and workflow system so that changes can be managed through normal procedures.

LANDesk and Altiris take the notion one step further. For example, the LANDesk Management Suite includes software distribution, license monitoring, OS imaging, and asset inventory management. Two years ago, the company also added a patch management module, and in 2004 it launched the LANDesk Security Suite, which can detect and remove spyware and assess the potential threats to machines by examining Internet Explorer settings, unnecessary services, unwanted programs such as file-sharing software, and anonymous user accounts.

DIFFERENT STROKES

As these three product categories move toward a common feature set, it's important for different groups in the organization to get input into the buying decision because each group will have its own needs to look out for.Mark Nicolett, vice president and research director at Gartner, says security configuration products tend to provide more detailed information about assets than IT operations groups are used to. "Operations people aren't normally trying to standardize down to system and registry settings," he says. However, system and registry settings have become more of an issue because of spyware and adware--one spyware program can make thousands of changes inside the registry.

General configuration settings may also be a source of conflict. A security team may recommend hundreds of configuration changes to harden assets, but if there's no present threat acting as an impetus for those changes, IT operations may be less inclined to go forward because of the likelihood that some of those changes will cause self-inflicted availability issues.

Another issue to be aware of is that many patch management and security management products don't integrate with trouble ticketing tools. This means the security team and the IT operations team may have to manage and integrate separate ticketing systems for remediation.

In the end, the best solution is to let the management process influence the product decision. Look for a product that can satisfy the needs of all the interested parties, including the security group, IT operations, and application owners. Whether you call it vulnerability management, security management, or configuration management, the common goal is to facilitate remediation for everyone involved. Says WesCorp's Hoff, "Rather than dump a 600-page vulnerability report on IT operations to sort out, we take a shared responsibility to make sure the process is streamlined and feasible because otherwise they throw up their hands."

Andrew Conry-Murray, technology editor, can be reached at [email protected].

From Patch to Process

Many organizations find that their IT department's priorities aren't set by the staff, but by software vulnerabilities and the attackers who exploit them.

"A process is needed so that organizations can identify vulnerabilities and other weaknesses in the environment and fix them before they are exploited or attacked," says Mark Nicolett, vice president and research director at Gartner, a consulting firm.

While a patch management tool can help, the problem is that the root cause of a vulnerability isn't always related to a patch. Root causes generally come in two forms: known vulnerabilities (which may or may not have an associated patch), and configuration policies that affect the risk posture of an asset.Step one of the process is to create policies regarding the secure configuration of assets. This paves the way for assessing the environment to find assets that are out of compliance. Once you have a baseline, you can bring assets back into compliance. However, because IT resources are limited, you'll have to set priorities. Priorities will differ from enterprise to enterprise based on the value of the assets, their effect on business processes, the criticality of the vulnerability, and regulatory issues.

Next comes remediation. This includes its own set of challenges, both technical and political. On the technical side, the vulnerabilities must be analyzed to determine how critical they are, if an exploit currently exists, whether patches are available, and what other steps can be taken. If patches are available, the organization must decided whether to deploy them immediately or during regular maintenance cycles. Patches must also be tested, tracked, and reported on.

Then there are the political complications. In many organizations, the security staff is tasked with finding and analyzing vulnerabilities, but redressing those vulnerabilities often falls to IT operations, which in turn must answer to application owners and business managers if services are disrupted.

After remediation comes monitoring, in which assets are continually assessed to ensure that previous patches are still in place, that configurations are correct, and that changes haven't been made that affect an asset's compliance.

At this point, the process starts all over again, resulting in a process-based cycle that drives the organization, rather than the organization being driven by vulnerabilities, patches, or attackers.

Risk Assessment: Patch Management

Patch management solutions have had several years to work out the kinks of patch delivery. They now focus on new capabilities for configuration management, such as dealing with registry and system settings, security policy enforcement, and so on. In addition, traditional asset tracking and management products from companies such as BMC, Altiris, and LANDesk are adding patch capabilities to their portfolios.

The majority of solutions are agent-based and will thus require some deployment effort, though network scanner-based products are also available. Patch management is also most efficient when rolled into a policy-driven security or configuration management system. Such a system requires considerable effort up front to create and deploy across the multiple silos (security, IT operations, application managers, and so on) in today's network environment.

By automating patch deployment, patch management systems provide significant benefits. Companies can substantially reduce the time required to update assets and enjoy greater protection from worms and other exploits. When folded into a broader configuration management system, it also helps track deviations from policy-based security standards.The most significant risk from a patch deployment system is the potential for a patch to adversely affect the host's OS or applications. A configuration management system can alleviate some of this risk through testing, or by allowing application owners or business units to decline a patch if there are strong concerns about its effect on business systems.