Gunnar Kudrjavets

Program testing can be used to show the presence of bugs, but never to show their absence!

This quote is from Edsger W. Dijkstra and is present in number of his papers, but probably is best known from Dijkstra’s Turing Award Lecture in 1972. My current post is about the most common misconceptions about software testing I've encountered. I have found that blog is a good way to maintain a knowledge base/repository and if some FAQ relevant to subjects I’ve written about comes up then I can recommend the person to use Google and search blogs.msdn.com for some specific phrases and read up ;-)

I should probably reiterate that these are all my personal opinions:

• Testing proves that something works correctly e.g., some complex algorithm is implemented properly. First of all, testing doesn’t prove anything. Proving the correctness of some algorithm or that some piece of code behaves as specified is a very difficult task. I remember that years ago when I was young and naïve CS student we had a seminar on formal methods in software engineering and spent quite some time proving correctness of some trivial search algorithms (basic linear search, binary search). About a month ago I attended a seminar about quality assurance and quality control in airplane software engineering and I asked presenters if they use formal methods at all? I was told that no, but maybe somebody in NASA is using them to prove that something really works as specified. So, don’t feel bad if your WinForms application isn’t mathematically proved to be correct ;-) There’s this really cool article “They Write The Right Stuff” (here’s Wiki link) about the shuttle software development everyone should read at least once and think about it.
• Quantity of test cases has something to do with the testing coverage and quality. If somebody says that we should measure programmer’s productivity by number of lines of code he/she writes then everyone will look at this person and start laughing. Most people understand that this way of measuring someone's productivity is just silly. The same is applicable to number of test cases. This is one clear example where quality matters more than quantity. For me personally there’s no difference if some team maintains 10 or 10,000 test cases. The number of different combinations and permutations grows so quickly that it’s no-brainer to come to work during the weekend and on Monday morning have 1000 valid, but useless test cases in your test case management system.
• Verification and validation is the same thing. Nope. Here’s the definition of verification (stolen from IEEE/ANSI standard):

  Process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of the phase.

  It’s very hard and very expensive to do the verification of something and to the best of my knowledge there are not much test teams out there who do verification. What most teams do is called validation which is per IEEE/ANSI:

  Process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements.

Next time - why do I think that to test something you should be able to have the same technical skills and knowledge as if you were the developer on the same project? I’ve been having this opinion forever, but this doesn’t seem to be very popular belief (I haven't found anyone yet who shares it  - people could start posting “Me too!“ comments ;-)) and I was pretty shocked when Edward Kit’s "Software Testing in the Real World: Improving the Process" describes the same attitude in Chapter 13 "Organization approaches to testing." At least I don’t think that I’m totally foolish anymore ;-)