Avoiding a False Sense of Security

It is widely known that there are problems associated with trying to develop a rating system for vulnerability assessments. Numerical schemes based on statistical models fail to take into account that a single exploitable vulnerability on one machine can lead to a compromise. Subjective rating systems make it difficult to determine whether the overall security posture of a network is improving over time.

In any rating system, however, there has to be some value that represents the highest attainable rating for that system. Is a network that receives this highest rating completely safe? If the highest rating doesn't indicate that the network is perfect, is the rating giving a false sense of security?

In a sample rating system, networks can be rated on a five-point scale from Poor to Excellent. So if a network is rated as Excellent, what does it really mean? In this rating system, it would mean that there is no vulnerability that is exploitable on any of your systems, and that the devices are not providing any information that can be used to gain knowledge about your network or systems. In essence, it's perfect from a security point of view.

So, if we rate a network as Excellent, or even Good, does that give a false sense of security to our customers? If our customers don't fully understand that security is a never-ending process and that they have to remain vigilant to keep that rating, there could indeed be such a misperception. All it takes is one critical vulnerability to be identified on one system, and the overall rating can plummet to Poor overnight.

Some users, trying to glean trending information about their network health, have been confused by shifting ratings: From month to month, a rating may move from Poor to Good, then back to Poor. How can they determine if their network posture is getting better or worse? When asked this question, I advise the user to look at what's causing the changes. Is there suddenly a spate of new vulnerabilities being identified by key software vendors? Are they performing system upgrades to their environment that introduce new vulnerabilities into the environment? Maybe new employee hires with bad security practices are introducing new problems it could be time to offer another round of security training.What can be frustrating to many IT administrators is when security assessment ratings are used as a job performance indicator. The security assessment rating is for the security posture of the network at the moment the network was tested. It is not an audit finding that encompasses multiple months of information or behavior in the network. Nor is it an indicator, direct or indirect, of how well the IT staff is maintaining the security of the network. It simply provides an indication of how vulnerable the network is to being exploited at the time of the assessment. Even trying to use the number of vulnerabilities found can be a misleading indicator of job performance.

An assessment run on one day that indicates no high-level or critical vulnerabilities may get a good rating, while the next day another assessment is run, indicating that every machine on the network has a critical vulnerability and is subject to being exploited. The assessment simply shows that in the previous 24-hour period, some new vulnerability has been identified and the existence of that vulnerability on the customer's network can now be tested.

Was the vulnerability there the day before? Probably: Most newly discovered vulnerabilities are found to have existed since the inception of the software or hardware. However, what didn't exist was the public knowledge of how to exploit that vulnerability and compromise the system.

Network and computer security assessments are just one tool in the measurement of the security posture of a given network. Each network and organization is unique and must choose an acceptable level of risk for itself. The network assessment simply provides a single point-in-time reference as to the security condition of the network – a snapshot. As long as there are new problems found in software and hardware, either through accidental discovery or deliberate searches, the instantaneous nature of the vulnerability assessment rating will fluctuate.

Over time, organizations should see that older vulnerabilities are being fixed on their networks and not being reintroduced when new devices are placed on the network. They should also see that when the posture of their network changes, it is due to a recent and identifiable cause. Determining how well your organization responds to these changes in security posture is far more important that trying to keep the posture constant. Only by looking at what is causing changes in rating can an organization truly understand if its security program and posture is improving. Only by using vulnerability assessment as one tool in the overall evaluation of the network, and not the sole value, can network administrators make these critical determinations.— Rick Fleming, Co-Founder and CTO, Digital Defense Inc.