Bug Bounty Programs Beat Internal Researchers

9 Android Apps To Improve Security, Privacy

In-house bug busters: You're not offering maximum value for money.

That's one conclusion reached by three University of California at Berkeley researchers who studied bug bounties paid over a three-year period by Google and Mozilla to fix bugs in the Chrome and Firefox browsers, respectively. The researchers' question: Is paying external parties to discover bugs worth it?

The answer seems to be yes. The researchers found that rewarding external bug hunters was between 2 and 100 times more cost effective than hiring full-time employees to accomplish the same task. The study was based on the average daily cost of Chrome rewards being $485, and Firefox $658. "If we consider that an average North American developer on a browser security team (i.e., that of Chrome or Firefox) would cost the vendor around $500 per day (assuming a $100,000 salary with a 50% overhead), we see that the cost of either of these [bug bounty programs] is comparable to the cost of just one member of the browser security team," wrote researchers Matthew Finifter, Devdatta Akhawe and David Wagner in the study.

Not only is the cost of a single employee comparable to that of a bug bounty program, few full-time employees appeared to match the bug-spotting power of a crowd-sourced program that pays cash. Notably, in the three-year period studied, Google paid out 371 bug bounties, while its most prolific internal Chrome bug hunter found 263 vulnerabilities. Mozilla paid out 148 bounties, while its best-performing employee spotted only 48 bugs.

[ Is your copy of Windows acting strangely? Read Microsoft Patch Tuesday Fixes Six Critical Bugs. ]

Of course, this is no call to get rid of internal employees tasked with finding and obliterating bugs in the codebase. Such personnel are an essential part of any secure development program. Furthermore, the study found that most external researchers couldn't afford to live off of bug bounty programs. Out of 82 external security researchers who reported Chrome bugs, for example, only three earned more than $80,000 from Google during the three-year period studied. A single Firefox bug hunter did earn a total of $141,000, but the next most prolific hunter earned just $42,000.

That's why, according to the study, businesses would likely be well-served by hiring their best external bug reporters as internal security researchers, and paying them to find as many vulnerabilities as possible. The researchers said that Mozilla, as well as Google's Chrome team, each appear to have at least three such employees. The study also suggests that every company that develops software should consider running a bug-bounty program to not only help spot bugs in code, but to find them as early possible in the development lifecycle, when remediation costs are relatively lower.

What do bug bounty programs need to succeed? The higher the profile of the program the more attention it gets and the more successful it is, said the researchers. On the down side, bug bounty programs might be less cost-effective where -- unlike browsers -- not every critical security vulnerability necessarily equals a major risk. Regardless, patching speed remains crucial. "High variance in time-to-patch is not appreciated by the security community," said the researchers. In other words, patch quickly or your would-be bug reporters might turn to other channels.

About 30 software vendors and projects -- including Cryptocat, Kim Dotcom's Mega, PayPal, Samsung and Qmail -- now reward information security researchers who report bugs. But some big names continue to remain on the bug-bounty sidelines, including Adobe, Apple and Oracle.

Google started the trend of offering cash for bugs about three years ago, and has so far paid out more than $800,000. "Our vulnerability reward programs have been very successful in helping us fix more bugs and better protect our users, while also strengthening our relationships with security researchers," Google security team members Adam Mein and Michal Zalewski said in a June blog post. Recently, the company raised the payout ante to a maximum of $7,500 for serious bugs, saying the move reflected the increasing difficulty of finding new bugs in Google code.

Many other businesses have since followed suit and launched their own programs. Facebook's bug bounty program, for example, promises minimum rewards of $500 and no maximum. Each bug is awarded a bounty based on its severity and creativity, according to the guidelines. Residents of any country that's not under U.S. sanctions (e.g., North Korea, Libya, Cuba, etc.) may participate.

Microsoft last year dipped its toes in the water by creating an annual BlueHat Prize. In June it announced that it would also offer a bounty program, paying up to $100,000 for anyone able to bypass the attack-mitigation defense technologies built into Windows 8.1 Preview. Microsoft said the move was driven in part by its desire to catch more bugs in its brand-new product before would-be attackers had time to weaponize them.

HP's Zero Day Initiative, among other programs, pays cash for bugs to enable it to add patches for vulnerabilities to its security products, while privately informing vendors and allowing them to ready a fix.

Alternately, bug hunters can work with an independent vulnerability broker such as the Grugq, a South African native based in Bangkok who takes a 15% commission and last year told Forbes that he handles only high-end exploits. "I refuse to deal with anything below mid-five figures these days," he said.

So if software developers can snag a critical vulnerability for less than that amount through their bug bounty programs, they've arguably bagged a bargain.