Gunnar Kudrjavets

What is BVT (Basic/Build Verification Test)? (Part III)

Grand final. If I would have to summarize everything I have to say in one sentence, it’ll be: paranoia is a virtue. Yes, repeat after me: paranoia is a virtue ;-) So, here’s our problem – how to write code what is responsible for verifying the correctness of some other piece of code? I don’t have a good answer for this, if I would, I’ll be currently retired and living from profits of “GK's patented technique for developing high-quality software“, spending my days reading good literature, writing some cool code, working on PhD, raising kids, and going to gym in the evenings ;-) Or maybe not, I'm too much of a workaholic anyway. Some people say that the only possible way to go is formal methods and languages like Z. Unfortunately I’ve never met anyone (possibly I just socialize with wrong people ;-)) who has successfully done anything with formal methods outside the academic circles. The cost is too high and there aren’t many knowledgeable people around either. Anyhow, here are a couple of pragmatic things we’ve learned during last couple of years in our group:

1. Do absolutely your best while writing the BVT code. Pick your best people, give them more time than you would usually give, and if somebody starts complaining about time spent, think about this: how much time (which can be directly  converted to money) you spend each time when false alarm happens? Somebody from test organization needs to investigate the failure, somebody from the development organization needs to investigate also, somebody needs to explain what happened and why etc.
2. Review, review, and review once more. Anyone who has ever read Steve McConnell’s “Code Complete” or any major papers by Capers Jones, knows that formal code inspections and personal desk-checking of the code are one of the best ways to detect defects. For references see “The Software –Quality Landscape” from “Code Complete, 2d Ed“.
3. Check every single return value for every single API call which can fail (basics again, but nobody bothers to do it). Here’s the difference between writing production code and writing the BVT code. In BVT case you can afford to check for every error possible – performance isn’t as important as it would be for speech recognition engine for example. Believe me, Murphy's Law works, everything can go wrong and fail ;-)
4. Log as detailed information as possible per every failure (basics again). Timestamps, thread ID-s, expected return value, actual return value, break automatically into debugger if necessary etc. I’ve seen my share of horribly written code which just says: “Test failed” and that's it. This doesn’t get you any closer to the root cause and quickly resolving it.
5. Use automated check-in systems. In our group we use system called "Gauntlet". To shortly summarize: "Gauntlet" is an automated way of building all the versions of your product and running the designated subset of test cases per every change to your codebase. Jonathan Caves has written longer post about how "Gauntlet" is used in Visual C++ team. We’ve been using "Gauntlet" for some time now and it proved to be extremely useful because of the following reasons:
   • We have non-existent number of build breaks. Nothing gets changed in the main codebase unless all the flavors of build succeed.
   • We greatly reduced the number of BVT breaks due to actual problems in the product. Yes, it still happens, but the main reasons are now infrastructure related problems, hardware related issues, and problems with the automation.
   • Psychologically the stress level in the team has been reduced. While making major changes to some interface for example, lots of people were concerned about how it’ll affect build and BVT results next day. Yes, you can do peer reviews, buddy builds and use all the other possibly countermeasures, but there’s only so much human can do in time allocated for the task to be completed. Penalties for build and BVT breaks are quite severe, so nobody wants to break anything second time ;-)
6. Use "rolling build" 24h a day. If you don’t have automated check-in system, here’s how the basic implementation may look like. Set up the automated process for doing the following:
   • Make sure that the source tree is synchronized and up to date.
   • Build everything and if there are any problems then send e-mail to your team.
   • Run selected set of tests on this fresh build and if there are any problems then send e-mail to your team.
   • Optional step: you may want to send e-mail also every time “rolling build” succeeds. This will indicate that nothing is broken yet. Lots of e-mail? Yes, but that's what e-mail filters are for.
   • Go back to the first step.
7. Use two-phase process to declare some test cases as BVT-s. The first phase is just letting the BVT to be part of daily processes for n days, but not to treat the results the test produces as official ones. After team decides that code looks solid enough - it’s been passing for number of days on all the editions under number of different configurations, the test case is "promoted" to be official. This approach proved to be also extremely useful in ironing out all the possible issues and increasing the trustworthiness of BVT code.

Notice that I’m not talking about TDD (Test Driven Development), unit testing, check-in tests, and other related subjects. Mainly because my knowledge in these areas is mainly theoretical and without seeing application of these techniques IRL during a couple of milestones, I don’t feel competent to be talking about this. You may check Jay Bazuzi’s blog for TDD and Marco Dorantes's blog for interesting agile software development and design related stuff.