This is one the subjects I have to explain very frequently so this blog entry is mainly for purposes of storing my answer somewhere in the permanent location. Let’s start with the moment where Alice (developer) fixed a bug and assigned it to Bob (tester) for closing. First of all, I never liked the term "closed/closing" when it comes to the bugs, IMHO the better term is "verified/verifying", but that’s not important right now. In ideal world Bob in general will do the following:
- Inspect and understand all the changes which have been made to fix a particular issue and verify that these changes haven’t introduced any unwanted or erroneous behavior. Please note that this automatically implies that Bob understands the architecture, design, code, and overall system functionality enough to make any conclusions like this.
- Verify that the bug doesn’t reproduce anymore under specific conditions where it was found. This includes setting up environment for reproducing the bug, running through the exact steps when bug was initially found, and confirming that nothing unexpected happens.
- Looks at the unexpected test failures (if any) and uses his intellect to determine if any of the new failures are relevant to the changes made to fix the particular issue. This is usually very tricky because n different bugs get fixed every day and unless you have a private build with only this specific change then investigative work like this will take time.
- Checks if there’s a test case which would have found the issue and if not then happily develops a new test case or set of test cases. This is the place where some people argue that developer should be the one developing appropriate unit test if applicable.
- If Bob is really committed then he’ll dig into the root cause analysis part of the bug and check the entire code base for similar patterns which caused the original issue. If Bob is really paranoid then if applicable he tries to determine a pattern in the code and develop a plug-in for his favorite static source code analyzer to detect issues like this in future automatically.
- Some ad hoc testing around the area where bug was found. I'm referencing Glenford J. Myers's quote "The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section." (Page 15, "The Art of Software Testing")
Of course this isn’t easy and takes some time, that’s why I highlighted the words "in ideal world" ;-)
Through the discussion in internal e-mail alias I discovered James Bach’s blog. As he describes it: "Rapid, heuristic, exploratory software testing. An intellectual, anti-clerical approach to testing." I have now spent already significant amount of time reading through the entries and there’s plenty of good stuff ;-) Bach’s company has also pointers to a number of interesting articles and presentations.
In addition you may want to take a look at other blogs at Florida Institute of Technology. Most of them are unfortunately inactive, but what’s there is IMHO worth reading.
This is just a humorous story about how metrics can be sometimes (incidentally?) used even for programmer’s benefit. A number of years ago (before I joined Microsoft) I worked together with a programmer who had developed a piece of financial software and one financial institution wished to purchase full rights for this specific application. They wanted to be able to do whatever they wished with the source code and because of some reasons the programmer who wrote the code was to receive the noticeable percentage from the profit coming out of this deal.
At some point in the process there was a discussion about how to determine the price for the full rights to the source code? One high-level participant proposed that we should just count the lines of the source code and multiple them by some specific amount of money. His logic was simple: the more there is code, the more it should cost. AFAIK there were some people who agreed with him. Of course people discussing this weren’t programmers ;-)
I don’t know how the deal finally ended, I wasn’t privy to this information, but I would assume that if LOC was the major factor in this deal then the programmer who owned the code would have spent his entire time adding comments to the source code and changing the bracing style ;-)
One of the funny problems software people experience IRL is that it’s kind of complicated to explain to other people what are the majority of books you’re constantly reading about (at this point any other representative of any other profession can scream "It's not only the software people, it's us too!") ;-) Here’s the sample conversation from the real life (I’m in the process of reading the John Robbins’s debugging book):
She: What are you reading?
I: Some sample code.
She (puzzled): What kind of code?
I: It’s mainly C++ and Win32 API.
She (still puzzled): What's this code about?
I: It teaches you how to implement your own debugger.
She (more puzzled): What's the debugger?
Unless your other side does professionally what you do then it’s kind of hard to have meaningful conversations about the books you’re currently reading. The example above could of course be reversed and I could be asking questions like "How this MMPI test thingy works?" etc. So, during my last vacation I decided it’s time to read something related to my girlfriend’s areas of expertise and after that have a discussion about this. I ended up with the following three books:
- "Funky Business: Talent Makes Capital Dance" by Jonas Ridderstrale, Kjell Nordström, Kjell Nordstrom. Here’s their main web site. One of the coolest and funniest business books I’ve ever read (I haven’t read many, so don’t take my opinion too seriously ;-)).
- "Girl, Interrupted" by Susanna Kaysen. If you have seen the movie (I actually had to rent the movie after reading the book to satisfy my curiosity) then I would strongly recommend reading the book also.
- "Man's Search For Meaning" by Viktor E. Frankl. Quote from Amazon.com: "Man's Search for Meaning" by Viktor E. Frankl is among the most influential works of psychiatric literature since Freud.
Next level will be the stage when we both will drop stuff related to our profession and start reading something neutral ;-)
In the beginning of the August Mozilla Foundation announced the Mozilla Security Bug Bounty Program. Short summary: if you report a critical security bug and they agree with you then you’ll get $500 cash prize. What can I say; soon people will be selling bugs on eBay and it’ll be currency like any other ;-)
When it comes to exploring the power of money as a motivational factor and its effects on software testing, here’s one experiment I always wanted to do.
Two teams consisting of people with identical technical skills (as close as you can get) will have fixed amount of time to test a specific piece of software. They test exactly the same functionality and teams work in isolation. First team will get paid some amount of money for every bug they find. Let’s assume that triage committee will go through every bug and assign some weight to the bugs and for example priority 1 bugs will cost $100, priority 2 bugs will cost $75, priority 3 bugs will cost $50, and priority 4 bugs will cost $25 (the decomposition into priorities and specific monetary amount can of course be whatever you want it to be). Second team will be motivated just by the pure technical challenge and the spirit of competition.
Which team will finally discover more bugs? Which team will finally discover more "valuable" bugs? How different the results will be? Is it order of magnitude or is it less? How will be the results of experiment depending on the specific type of software? What happens if we reverse the teams and rerun the experiment again at some point in time? Will the results be different if triage committee will constantly triaging the bugs and letting the first team know that they’ve already earned $n? Will the results be different if the second team won’t know that first team will get paid for bugs?
Psychology graduate students with interest in software engineering, where are you ;-)
Whenever I read proceedings of the ACM SIGSOFT (Special Interest Group on Software Engineering) I find myself thinking that probably I’m wasting my time. Most of the articles could IMHO be classified as "esoteric software engineering" which is probably fine when the reader is working on his/her Ph.D. thesis or is scientist. My personal interest lays in practical applications (yet another chance to cite Fred Brooks - "A scientist builds in order to learn; an engineer learns in order to build") so I'm mainly interested in how I can apply anything what's described IRL ;-)
Latest issue of SIGSOFT is dedicated to software testing and analysis and has a couple of interesting articles I have found to be the most applicable to my work related duties:
- Nurit Dor, Stephen Adams, Manuvir Das, Zhe Yang: "Software validation via scalable path-sensitive value flow analysis". This article mentions a tool called ESP (Error detection via Scalable Program analysis).
- David Saff, Michael D. Ernst: "An experimental evaluation of continuous testing during development".
- T. J. Ostrand, E. J. Weyuker, R. M. Bell: "Where the bugs are".
- Darrell Reimer, Edith Schonberg, Kavitha Srinivas, Harini Srinivasan, Bowen Alpern, Robert D. Johnson, Aaron Kershenbaum, Larry Koved: "SABER: Smart Analysis Based Error Reduction".
If you’re interested in subjects mentioned below then check these articles out on your next trip to the library, at least I felt that my time was well spent ;-)
A couple of posts ago I already mentioned the "Software Entropy" article by Pragmatic Programmers. Today I deleted 545 files (data files, documents, source code, and test cases) from our code base. Mainly because nobody in my team was using these files, to the best of our knowledge nobody else was using them, and most of them were obsolete.
IMHO the live code base should not be used as a data storage facility and must contain only the data necessary at current point in time. Once people start using source control system as tool for backing up something, adding every edition of the document as a new file or borrowing someone else’s code and just in case checking it in then chaos is inevitable. During the first version of Speech Server were learned in very hard way that it's not a good idea to have 3-4 different copies of one of our most critical test applications being checked in under different directories. Sooner or later you'll shoot yourself in the foot.
Backup software and appropriate measures should be used for backup. If you want to see the change history then one should use relevant version control tools. If you want to retrieve old document describing how things were three years ago then use specific archive location. If you want to store some tools you wrote then add the source code, not the binaries. Average software project in itself is very difficult to handle and lately one of my main guiding principles has been KISS principle. Less files, less problems ;-)
P. S. Tomorrow I’ll be probably hiding out in cafeteria with my trustworthy laptop because based on Murphy ’s Law at least one these files I deleted was definitely used by someone who’ll come looking for me ;-)
Coming back from the vacation is like jumping into another dimension, there’s just too much stuff to do. But as a guilty pleasure I took this week an hour off for reading the latest issue of Dr. Dobb’s Journal and noticed that there’s definitely a new evolving trend in hiring software engineers - publishing puzzles in announcements. Not that asking puzzle questions during the interviews is something new, but I haven’t noticed too much advertisements containing them. If you want to do some mental work then here’s one from Google ;-)
Which one of the possible answers (there are multiple patterns) is the right one? I really don't know, there are number of answers one can argue are the right ones ;-)
Sometimes it’s good to associate people's faces and blog entries they compose. When you see them walking in the cafeteria then you can say “Hey, you're the person who got it all wrong in your blog ;-)“ Here are two pictures of me during my last week’s vacation in San Francisco. My error in judgment (not using any sunblock) meant that after second day my face was in constant pain and I looked very unhappy ;-) That’s what happens when you’re used to Seattle and occasionally visit California.
San Francisco Bay Area rules!
One of the things I’m constantly thinking about is how to measure the value of someone's actions which prevented bugs reaching our customers. How do I quantify something like "If Alice hasn’t fixed this buffer overflow which Bob discovered then three months from now we would had to issue a security bulletin and spend $X amount of money as a result of it all." It’s very hard (rather impossible) to prove that if some bug hasn’t been discovered by us then somebody would have discovered it in outside world and there’ll a disaster on our hands. Well, in case when your utility accidentally overwrites the boot sector then it’s quite clearly a bad thing ;-) But what about bugs which don't have so clear impact? Not every memory leak is a showstopper. Not every buffer overflow is a security hole.
Quite large percentage of bugs has a root cause which is relatively easy to fix: variables not initialized properly, missing call to release some kind of resource, string being not terminated properly etc. Here are a couple of situations we had while developing both external and internal tools:
- Missing call to
closesocket() and therefore leaking socket descriptors caused us and our partners to spend days diagnosing the root cause of the issue. Fix was very simple, just one line of the code.
- Occasionally some of our BVT-s crashed for no reason at all. Every time we spent hours troubleshooting the problem, unable to understand why it happened. Finally we had an opportunity to capture the crash information and found out that the problem was string which wasn’t properly terminated.
- We had to spend significant amount of time during shipping our first version because in one of the components we were using was tiny memory leak. Just a couple of bytes, but in a couple of days it all accumulated so much that OS started running out of virtual memory.
Probably every programmer can continue this list with hundreds of examples, there’s nothing new here and let’s note the fact I’m being very modest when talking about the cost of bugs. I’m not talking about bugs which caused products to be late or things like Code Red or Nimda. If you have some time on your hands then check out "Collection of Software Bugs".
What tends to happen quite often IRL is that after we’ve hit some kind of blocking issue then somebody spends day and night chasing some bug down in his code, fixing it, and we pat him on the back and say "Good work, that’s the spirit!" instead of asking the question "If Trent had asked Eve to review his code before checking it in, would it sill have happened?" Or let’s take simplified example and two hypothetical development teams: team A and team B. Team A stores all their string constants in resource files because they think it’s a right thing to do going forward. Team B thinks that "We’re US English only, let’s not bother." At some point the decision is made that the product needs to be shipped at international markets. Team A doesn’t do much when it comes to these string constants. Team B spends weekend fixing the code, testers test it during the night, and finally it’s ready. Guess who looks as a hero to the public’s eye? Of course team B when the actual prize should have been team A’s because they followed the proper engineering practices from the beginning.
But how do you effectively measure this? Do you take notes about everything during the entire year and later analyze all this? Can you even compare one person’s commitment to spend the entire weekend fixing bugs against other person’s thorough approach to use proper engineering practices and prevent these bugs from happening? Lots of things to philosophize about ;-)
More Posts « Previous page
- Next page »