Gunnar Kudrjavets

Paranoia is a virtue

Paying money to people for finding bugs

In the beginning of the August Mozilla Foundation announced the Mozilla Security Bug Bounty Program. Short summary: if you report a critical security bug and they agree with you then you’ll get $500 cash prize. What can I say; soon people will be selling bugs on eBay and it’ll be currency like any other ;-)

When it comes to exploring the power of money as a motivational factor and its effects on software testing, here’s one experiment I always wanted to do.

Two teams consisting of people with identical technical skills (as close as you can get) will have fixed amount of time to test a specific piece of software. They test exactly the same functionality and teams work in isolation. First team will get paid some amount of money for every bug they find. Let’s assume that triage committee will go through every bug and assign some weight to the bugs and for example priority 1 bugs will cost $100, priority 2 bugs will cost $75, priority 3 bugs will cost $50, and priority 4 bugs will cost $25 (the decomposition into priorities and specific monetary amount can of course be whatever you want it to be). Second team will be motivated just by the pure technical challenge and the spirit of competition.

Which team will finally discover more bugs? Which team will finally discover more "valuable" bugs? How different the results will be? Is it order of magnitude or is it less? How will be the results of experiment depending on the specific type of software? What happens if we reverse the teams and rerun the experiment again at some point in time? Will the results be different if triage committee will constantly triaging the bugs and letting the first team know that they’ve already earned $n? Will the results be different if the second team won’t know that first team will get paid for bugs?

Psychology graduate students with interest in software engineering, where are you ;-)

Posted: Aug 20 2004, 07:12 PM by gunnarku | with 6 comment(s)
Filed under:

Comments

Mikhail Arkhipov (MSFT) said:

If we only shoot for a number of bugs, then repro steps might begin to suck. Good tester not only finds a bug, but also minimized repro steps in order to save developer time. Good tester also takes time to install symbols and debugger and obtain a callstack for an assert, crash or hang. She can also spend additional time preparing repro machine if bug reproes only in a particular configuration or on localized OS.
# August 20, 2004 11:54 PM

Gunnar Kudrjavets [MSFT] said:

IMHO the incomplete repro steps shouldn't be a problem here. Before running any kind of competition or experiment we can establish a very clear quality bar for the bugs. The main issue here is how many real bugs we can find. One can do always postprocessing later to optimize the repro steps etc.
# August 21, 2004 12:05 AM

TrackBack said:

Microsoft's Gunnar Kudrjavets mentions the Mozilla Security Bug Bounty (get $500 by finding and reporting a security bug) and ponders the effect of money in the bug reporting process. Naturally, Gunnar's concept of a bug-finding competition with some teams getting...
# August 21, 2004 6:08 AM

Brant Gurganus said:

Money doesn't really inspire me when I look for bugs. My bug hunting generally involves using the product for daily use. If I see something fishy, I file a bug. These can range from usability nits like using checkmarks in menu items where a radio-item marking is more appropriate.

When I learn about a testing tool, I'll hook it up to the program and again use it under normal use. For example, I've used the Microsoft Application Verifier program to discover that in several cases, Mozilla retrieves data directly from the registry instead of via the appropriate API call. I've also discovered that the way in which Mozilla products start the error reporting tool upon crash is insecure because it uses a buggy API of Windows in a way that allows, for example, "C:\Program.exe" to be run instead of "C:\Program Files\mozilla.org\Mozilla Firefox\components\talkback.exe". I don't really care about the importance of the bug, that is for the developers to decide. Any issue diminishes from the quality of the product and if they are not filed when discovered, no matter how trivial, they may never be filed and therefore never get fixed.
# August 21, 2004 8:34 AM

Michael Swanson said:

I've tried a similar approach, but to keep costs down, we used a point system (similar to the way you've broken things down by bug severity). At the end of stabilization, the person with the higest score received the highest payout, the next person received less, etc. I found that if you only reward the person with the highest score, if they take a commanding lead, the others will slack off, because they don't think they'll ever be able to beat him.
# August 21, 2004 12:07 PM

Shital Shah said:

Unless its your own code or the bugs are really hurting you (i.e. you are one of the real user who can't live without app rather then official QA), its a tedius and boring task to find a bug and even more so to document it. My guess is, the only reason a person would want to look for bugs in an app that doesn't matter to them would be anticipation of reward or continuation of job. Unlike writing code or doing painting, for most mortals, consciously put effort in looking for bugs in someone else's application is not a fun thing to do.

Financial reward per well documented bug is definitely a innovative way to improve your app.
# August 23, 2004 5:17 PM
Leave a Comment

(required) 

(required) 

(optional)

(required)