What is the desirable way to "close a bug"?
This is one the subjects I have to explain very frequently so this blog entry is mainly for purposes of storing my answer somewhere in the permanent location. Let’s start with the moment where Alice (developer) fixed a bug and assigned it to Bob (tester) for closing. First of all, I never liked the term "closed/closing" when it comes to the bugs, IMHO the better term is "verified/verifying", but that’s not important right now. In ideal world Bob in general will do the following:
- Inspect and understand all the changes which have been made to fix a particular issue and verify that these changes haven’t introduced any unwanted or erroneous behavior. Please note that this automatically implies that Bob understands the architecture, design, code, and overall system functionality enough to make any conclusions like this.
- Verify that the bug doesn’t reproduce anymore under specific conditions where it was found. This includes setting up environment for reproducing the bug, running through the exact steps when bug was initially found, and confirming that nothing unexpected happens.
- Looks at the unexpected test failures (if any) and uses his intellect to determine if any of the new failures are relevant to the changes made to fix the particular issue. This is usually very tricky because n different bugs get fixed every day and unless you have a private build with only this specific change then investigative work like this will take time.
- Checks if there’s a test case which would have found the issue and if not then happily develops a new test case or set of test cases. This is the place where some people argue that developer should be the one developing appropriate unit test if applicable.
- If Bob is really committed then he’ll dig into the root cause analysis part of the bug and check the entire code base for similar patterns which caused the original issue. If Bob is really paranoid then if applicable he tries to determine a pattern in the code and develop a plug-in for his favorite static source code analyzer to detect issues like this in future automatically.
- Some ad hoc testing around the area where bug was found. I'm referencing Glenford J. Myers's quote "The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section." (Page 15, "The Art of Software Testing")
Of course this isn’t easy and takes some time, that’s why I highlighted the words "in ideal world" ;-)