Gunnar Kudrjavets

Paranoia is a virtue

Opening bugs is hard and takes some courage

The classical definition of a bug tells us that bug is "an unwanted and unintended property of a program or piece of hardware, especially one that causes it to malfunction." Let’s explore this subject a little more and use another established description:

"A software error is present when the program does not do what its end user reasonably expects it to do."

This is from G. J. Myers, Software Reliability: Principles and Practices. New York: Wiley-Interscience, 1976. We notice that our new definition is more relaxed. The key is in word "reasonably". What is reasonable depends quite a lot from who is using the software and what he/she thinks about specific feature, specific behavior, performance etc. That’s where it gets complicated.

There are lots of cases where it’s very easy to determine that something is bug. For example if you click on “File” menu and your application crashes right away then it’s probably a bug; if your text editor produces documents which it can’t later open then there’s most likely something wrong about this behavior also. But what about your server product starting up slowly? Is this a bug or is the behavior reasonable taking into account the specific hardware, other applications running at the same time, and all the things which product needs to do internally before starting up? Here's where we reach the gray area.

Opening bugs is difficult. What makes it complicated is that basic human psychology kicks in. Imagine a typical situation where developer and tester work in pairs. A developer codes up an implementation of something and then it is tester’s job to check if something works. Next thing you know is that test engineer is opening bugs left and right with titles "This crashes", "That doesn’t work properly", "Incorrect dialog box here and there" etc. Most of us don’t like other people pointing out mistakes in our work (at least I don’t ;-)). I bet that rationally we understand that this is how it’s supposed to be in typical software development lifecycle and are happy that somebody found the bugs before the customer had a chance to find them.

For the person opening bugs it’s hard also. Nobody opens perfect bugs and sometimes people make mistake. Sometimes we run performance and stress runs with incorrect configurations. Sometimes people have a wrong version of some binary installed on their machine and they waste 0.5 days of developer’s time to only discover that everything works as it’s supposed to work and the problem is with the configuration itself. Sometimes bug opener misunderstood the specification and caused unnecessary cycles to be burnt on triage meetings. Problem is that you can’t do this very often. People will start noticing, people will start complaining, and your personal trustworthiness start going downhill. I know that one metric that some test leads/managers use is that they look at what the percentage of bugs opened by certain individual is resolved as "Won’t Fix", "By Design", and "Not Repro". I don’t have a strong opinion about this, but I’m afraid that measurements like this will encourage people to take the easy way out. It would be much more beneficial to hunt for bugs which are certain to be bugs rather than spend precious time talking with your counterparts in development and program management and figuring out the really complicated things.

In some cases the conflict is inevitable. Let’s take the hypothetical slow start-up scenario again. In my subconscious it’s very clear: product just starts up slow, starting up slow isn’t reasonable, therefore it must be bug. But at the same time I also know that for the product to start up it needs: a) check dependencies and if necessary start up some other services; b) create a pool of resources for the later usage; c) check a couple of other things on the way and then finally report to SCM (Service Control Manager) that "Hey, I’m ready!" That’s the downside of being inside the product development. You understand all this and it kind of seems logical and it's difficult to pretend that you're the actual end user. The problem is that the users don’t care (like I don’t care how my cell phone exactly works or how my car’s engine operates) if it’s managed or unmanaged code, is it creating a thread pool or not, does it have dependent services or not, does it has to load a number of other DLL-s or not etc.

Here’s where good test engineers, who will hopefully put their feet on floor and refuse to leave the triage meeting till bugs get accepted by triage, come handy ;-) I personally encourage this kind of resistance a.k.a. “being customer advocate“ ;-) and would prefer to have little clashes during the milestone rather than two months after shipping read in newsgroups that product we shipped is unusable. As civilized human beings we try to avoid the confrontation, but sometimes I think we do this too much.

Posted: Jun 27 2004, 05:21 PM by gunnarku | with no comments
Filed under:

Comments

No Comments

Leave a Comment

(required) 

(required) 

(optional)

(required)