August 2004 - Posts
Today I attended one excellent talk covering software engineering culture in general, things which currently need improvement, some internal case studies etc. Test to development ratio was also pointed out in this talk and one interesting thought was mentioned during this presentation. Microsoft is different from most of the software companies by development and test organizations having very similar sizes. Hypothetical little feature team may consist of 6 SDE-s, 4 SDE/T-s and 1 STE for example. What would happen if ratio of testers and developers will be 1:10 for example and at the same time you would have a top notch development team who’ll be obsessive about code reviews, unit testing, proper engineering practices etc? Natural reaction is that the product quality will be doomed ;-) However, one of the projected outcomes was that the product quality will be self-correcting and will stabilize itself automatically to the level that would be acceptable for shipping? You know, all these factors like people developing the software understanding that they’re the ones responsible for product quality and there’s nobody else ;-) Major assumption is of course that peer pressure and strong personal accountability for the quality will be applied.
P. S. Everyone who has shipped any products where testing was done only by developers can claim that they’ve participated in this experiment already and if their product made enough money then they’ve succeeded ;-)
This is one the subjects I have to explain very frequently so this blog entry is mainly for purposes of storing my answer somewhere in the permanent location. Let’s start with the moment where Alice (developer) fixed a bug and assigned it to Bob (tester) for closing. First of all, I never liked the term "closed/closing" when it comes to the bugs, IMHO the better term is "verified/verifying", but that’s not important right now. In ideal world Bob in general will do the following:
- Inspect and understand all the changes which have been made to fix a particular issue and verify that these changes haven’t introduced any unwanted or erroneous behavior. Please note that this automatically implies that Bob understands the architecture, design, code, and overall system functionality enough to make any conclusions like this.
- Verify that the bug doesn’t reproduce anymore under specific conditions where it was found. This includes setting up environment for reproducing the bug, running through the exact steps when bug was initially found, and confirming that nothing unexpected happens.
- Looks at the unexpected test failures (if any) and uses his intellect to determine if any of the new failures are relevant to the changes made to fix the particular issue. This is usually very tricky because n different bugs get fixed every day and unless you have a private build with only this specific change then investigative work like this will take time.
- Checks if there’s a test case which would have found the issue and if not then happily develops a new test case or set of test cases. This is the place where some people argue that developer should be the one developing appropriate unit test if applicable.
- If Bob is really committed then he’ll dig into the root cause analysis part of the bug and check the entire code base for similar patterns which caused the original issue. If Bob is really paranoid then if applicable he tries to determine a pattern in the code and develop a plug-in for his favorite static source code analyzer to detect issues like this in future automatically.
- Some ad hoc testing around the area where bug was found. I'm referencing Glenford J. Myers's quote "The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section." (Page 15, "The Art of Software Testing")
Of course this isn’t easy and takes some time, that’s why I highlighted the words "in ideal world" ;-)
Through the discussion in internal e-mail alias I discovered James Bach’s blog. As he describes it: "Rapid, heuristic, exploratory software testing. An intellectual, anti-clerical approach to testing." I have now spent already significant amount of time reading through the entries and there’s plenty of good stuff ;-) Bach’s company has also pointers to a number of interesting articles and presentations.
In addition you may want to take a look at other blogs at Florida Institute of Technology. Most of them are unfortunately inactive, but what’s there is IMHO worth reading.
This is just a humorous story about how metrics can be sometimes (incidentally?) used even for programmer’s benefit. A number of years ago (before I joined Microsoft) I worked together with a programmer who had developed a piece of financial software and one financial institution wished to purchase full rights for this specific application. They wanted to be able to do whatever they wished with the source code and because of some reasons the programmer who wrote the code was to receive the noticeable percentage from the profit coming out of this deal.
At some point in the process there was a discussion about how to determine the price for the full rights to the source code? One high-level participant proposed that we should just count the lines of the source code and multiple them by some specific amount of money. His logic was simple: the more there is code, the more it should cost. AFAIK there were some people who agreed with him. Of course people discussing this weren’t programmers ;-)
I don’t know how the deal finally ended, I wasn’t privy to this information, but I would assume that if LOC was the major factor in this deal then the programmer who owned the code would have spent his entire time adding comments to the source code and changing the bracing style ;-)
One of the funny problems software people experience IRL is that it’s kind of complicated to explain to other people what are the majority of books you’re constantly reading about (at this point any other representative of any other profession can scream "It's not only the software people, it's us too!") ;-) Here’s the sample conversation from the real life (I’m in the process of reading the John Robbins’s debugging book):
...
She: What are you reading?
I: Some sample code.
She (puzzled): What kind of code?
I: It’s mainly C++ and Win32 API.
She (still puzzled): What's this code about?
I: It teaches you how to implement your own debugger.
She (more puzzled): What's the debugger?
...
Unless your other side does professionally what you do then it’s kind of hard to have meaningful conversations about the books you’re currently reading. The example above could of course be reversed and I could be asking questions like "How this MMPI test thingy works?" etc. So, during my last vacation I decided it’s time to read something related to my girlfriend’s areas of expertise and after that have a discussion about this. I ended up with the following three books:
- "Funky Business: Talent Makes Capital Dance" by Jonas Ridderstrale, Kjell Nordström, Kjell Nordstrom. Here’s their main web site. One of the coolest and funniest business books I’ve ever read (I haven’t read many, so don’t take my opinion too seriously ;-)).
- "Girl, Interrupted" by Susanna Kaysen. If you have seen the movie (I actually had to rent the movie after reading the book to satisfy my curiosity) then I would strongly recommend reading the book also.
- "Man's Search For Meaning" by Viktor E. Frankl. Quote from Amazon.com: "Man's Search for Meaning" by Viktor E. Frankl is among the most influential works of psychiatric literature since Freud.
Next level will be the stage when we both will drop stuff related to our profession and start reading something neutral ;-)
In the beginning of the August Mozilla Foundation announced the Mozilla Security Bug Bounty Program. Short summary: if you report a critical security bug and they agree with you then you’ll get $500 cash prize. What can I say; soon people will be selling bugs on eBay and it’ll be currency like any other ;-)
When it comes to exploring the power of money as a motivational factor and its effects on software testing, here’s one experiment I always wanted to do.
Two teams consisting of people with identical technical skills (as close as you can get) will have fixed amount of time to test a specific piece of software. They test exactly the same functionality and teams work in isolation. First team will get paid some amount of money for every bug they find. Let’s assume that triage committee will go through every bug and assign some weight to the bugs and for example priority 1 bugs will cost $100, priority 2 bugs will cost $75, priority 3 bugs will cost $50, and priority 4 bugs will cost $25 (the decomposition into priorities and specific monetary amount can of course be whatever you want it to be). Second team will be motivated just by the pure technical challenge and the spirit of competition.
Which team will finally discover more bugs? Which team will finally discover more "valuable" bugs? How different the results will be? Is it order of magnitude or is it less? How will be the results of experiment depending on the specific type of software? What happens if we reverse the teams and rerun the experiment again at some point in time? Will the results be different if triage committee will constantly triaging the bugs and letting the first team know that they’ve already earned $n? Will the results be different if the second team won’t know that first team will get paid for bugs?
Psychology graduate students with interest in software engineering, where are you ;-)
Whenever I read proceedings of the ACM SIGSOFT (Special Interest Group on Software Engineering) I find myself thinking that probably I’m wasting my time. Most of the articles could IMHO be classified as "esoteric software engineering" which is probably fine when the reader is working on his/her Ph.D. thesis or is scientist. My personal interest lays in practical applications (yet another chance to cite Fred Brooks - "A scientist builds in order to learn; an engineer learns in order to build") so I'm mainly interested in how I can apply anything what's described IRL ;-)
Latest issue of SIGSOFT is dedicated to software testing and analysis and has a couple of interesting articles I have found to be the most applicable to my work related duties:
- Nurit Dor, Stephen Adams, Manuvir Das, Zhe Yang: "Software validation via scalable path-sensitive value flow analysis". This article mentions a tool called ESP (Error detection via Scalable Program analysis).
- David Saff, Michael D. Ernst: "An experimental evaluation of continuous testing during development".
- T. J. Ostrand, E. J. Weyuker, R. M. Bell: "Where the bugs are".
- Darrell Reimer, Edith Schonberg, Kavitha Srinivas, Harini Srinivasan, Bowen Alpern, Robert D. Johnson, Aaron Kershenbaum, Larry Koved: "SABER: Smart Analysis Based Error Reduction".
If you’re interested in subjects mentioned below then check these articles out on your next trip to the library, at least I felt that my time was well spent ;-)
A couple of posts ago I already mentioned the "Software Entropy" article by Pragmatic Programmers. Today I deleted 545 files (data files, documents, source code, and test cases) from our code base. Mainly because nobody in my team was using these files, to the best of our knowledge nobody else was using them, and most of them were obsolete.
IMHO the live code base should not be used as a data storage facility and must contain only the data necessary at current point in time. Once people start using source control system as tool for backing up something, adding every edition of the document as a new file or borrowing someone else’s code and just in case checking it in then chaos is inevitable. During the first version of Speech Server were learned in very hard way that it's not a good idea to have 3-4 different copies of one of our most critical test applications being checked in under different directories. Sooner or later you'll shoot yourself in the foot.
Backup software and appropriate measures should be used for backup. If you want to see the change history then one should use relevant version control tools. If you want to retrieve old document describing how things were three years ago then use specific archive location. If you want to store some tools you wrote then add the source code, not the binaries. Average software project in itself is very difficult to handle and lately one of my main guiding principles has been KISS principle. Less files, less problems ;-)
P. S. Tomorrow I’ll be probably hiding out in cafeteria with my trustworthy laptop because based on Murphy ’s Law at least one these files I deleted was definitely used by someone who’ll come looking for me ;-)
Coming back from the vacation is like jumping into another dimension, there’s just too much stuff to do. But as a guilty pleasure I took this week an hour off for reading the latest issue of Dr. Dobb’s Journal and noticed that there’s definitely a new evolving trend in hiring software engineers - publishing puzzles in announcements. Not that asking puzzle questions during the interviews is something new, but I haven’t noticed too much advertisements containing them. If you want to do some mental work then here’s one from Google ;-)
Which one of the possible answers (there are multiple patterns) is the right one? I really don't know, there are number of answers one can argue are the right ones ;-)
More Posts