Gunnar Kudrjavets

Hunting down the missing calls to Dispose()

Today I had five different meetings. Yes, that’s how my average day looks like. Between these meetings there are slots of time I use to read and write e-mail and do something I call "real work" ;-) The interesting discovery for today was that I found a pattern in our code base where we create number of different objects implementing IDisposable interface and we accidentally forgot to call Dispose()  method on them. As almost anyone traveling in the managed world knows, this means that the objects won’t be properly cleaned up till garbage collection kicks in at some point which because of the nondeterministic nature of garbage collection may happen whenever CLR decides it’s necessary.

Why do I care ;-) Well, because I work on the server product and we really-really-really want everything cleaned up in timely fashion to keep the working set as low as possible. When working set keeps increasing or handle count keeps growing then we start getting pretty nervous around here.

The typical code pattern may look like this:

{
    ...

    StringWriter sw = new StringWriter(CultureInfo.InvariantCulture);

    //
    // Do something with 'sw', but never call 'Close()' when finished.
    //
}

Everyone knows the saying "Where There's Smoke There's Fire". For the practical applications I’ll borrow the quote from Glenford J. Myers’s "The Art of Software Testing" (page 15):

"The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section."

So, if you’ll ask me what I’ll be doing tomorrow between meetings then the answer is that I’ll be checking the related assemblies for the similar patterns ;-)

The joy of pedantic people - hunting down "TODO:" comments in code ;-)

Because of the recent organizational change we inherited a new code base consisting mainly of C# code. When it comes to the unmanaged code then I usually do the following before even reading any lines of new inherited code:

1. Make sure that code compiles cleanly with /W4, /Wp64, and /WX.
2. Enable the following compiler options (if not enabled already): /GS and /RTC1. Then run some subset of tests on the new build in hope of seeing some errors.
3. Run PREfast or PREfix on the code base and look through every single warning.

With managed code it’s not so easy anymore. Half of the problems typical to C/C++ family of programming languages are suddenly gone. It’s not so straightforward to point out mistakes in other people’s code anymore. But the wickedness in me couldn’t rest and after some brainstorming I suddenly had it!

In our group we open a bug per every "TODO:" comment in the code to make sure that everything is accounted for and nothing gets away before shipping. Quick search in bug database showed that I was the first one to stumble across this and more than twenty bugs later I’m feeling that today was a good day ;-) Sometimes there’s only so much needed for happiness.

On the other hand, if a couple of years ago somebody would have pointed out these "TODO:" comments in my code then I would have definitely answered: “Those who can, do. Those who can't, just complain about other people’s code.” Life is strange sometimes.

Opening bugs is hard and takes some courage

The classical definition of a bug tells us that bug is "an unwanted and unintended property of a program or piece of hardware, especially one that causes it to malfunction." Let’s explore this subject a little more and use another established description:

"A software error is present when the program does not do what its end user reasonably expects it to do."

This is from G. J. Myers, Software Reliability: Principles and Practices. New York: Wiley-Interscience, 1976. We notice that our new definition is more relaxed. The key is in word "reasonably". What is reasonable depends quite a lot from who is using the software and what he/she thinks about specific feature, specific behavior, performance etc. That’s where it gets complicated.

There are lots of cases where it’s very easy to determine that something is bug. For example if you click on “File” menu and your application crashes right away then it’s probably a bug; if your text editor produces documents which it can’t later open then there’s most likely something wrong about this behavior also. But what about your server product starting up slowly? Is this a bug or is the behavior reasonable taking into account the specific hardware, other applications running at the same time, and all the things which product needs to do internally before starting up? Here's where we reach the gray area.

Opening bugs is difficult. What makes it complicated is that basic human psychology kicks in. Imagine a typical situation where developer and tester work in pairs. A developer codes up an implementation of something and then it is tester’s job to check if something works. Next thing you know is that test engineer is opening bugs left and right with titles "This crashes", "That doesn’t work properly", "Incorrect dialog box here and there" etc. Most of us don’t like other people pointing out mistakes in our work (at least I don’t ;-)). I bet that rationally we understand that this is how it’s supposed to be in typical software development lifecycle and are happy that somebody found the bugs before the customer had a chance to find them.

For the person opening bugs it’s hard also. Nobody opens perfect bugs and sometimes people make mistake. Sometimes we run performance and stress runs with incorrect configurations. Sometimes people have a wrong version of some binary installed on their machine and they waste 0.5 days of developer’s time to only discover that everything works as it’s supposed to work and the problem is with the configuration itself. Sometimes bug opener misunderstood the specification and caused unnecessary cycles to be burnt on triage meetings. Problem is that you can’t do this very often. People will start noticing, people will start complaining, and your personal trustworthiness start going downhill. I know that one metric that some test leads/managers use is that they look at what the percentage of bugs opened by certain individual is resolved as "Won’t Fix", "By Design", and "Not Repro". I don’t have a strong opinion about this, but I’m afraid that measurements like this will encourage people to take the easy way out. It would be much more beneficial to hunt for bugs which are certain to be bugs rather than spend precious time talking with your counterparts in development and program management and figuring out the really complicated things.

In some cases the conflict is inevitable. Let’s take the hypothetical slow start-up scenario again. In my subconscious it’s very clear: product just starts up slow, starting up slow isn’t reasonable, therefore it must be bug. But at the same time I also know that for the product to start up it needs: a) check dependencies and if necessary start up some other services; b) create a pool of resources for the later usage; c) check a couple of other things on the way and then finally report to SCM (Service Control Manager) that "Hey, I’m ready!" That’s the downside of being inside the product development. You understand all this and it kind of seems logical and it's difficult to pretend that you're the actual end user. The problem is that the users don’t care (like I don’t care how my cell phone exactly works or how my car’s engine operates) if it’s managed or unmanaged code, is it creating a thread pool or not, does it have dependent services or not, does it has to load a number of other DLL-s or not etc.

Here’s where good test engineers, who will hopefully put their feet on floor and refuse to leave the triage meeting till bugs get accepted by triage, come handy ;-) I personally encourage this kind of resistance a.k.a. “being customer advocate“ ;-) and would prefer to have little clashes during the milestone rather than two months after shipping read in newsgroups that product we shipped is unusable. As civilized human beings we try to avoid the confrontation, but sometimes I think we do this too much.

Estimating and scheduling - the most exciting parts of the milestone

When looking at my current job responsibilities then estimating and scheduling are the things which really make me happy. Maybe it’s common to all Virgos, but organizing the tasks, tracking down all the fuzzy issues, and clarifying all the little details is something I really enjoy doing ;-) The problem statement at the beginning of the milestone is very simple: everybody wants you to do all kinds of different stuff and your job is to figure out how much it’ll take, who’ll do it, what are the priorities and risks associated with every tasks, how the dependency graph looks etc. Cool, isn't it?

One thing which slightly shocked me at Microsoft in the beginning was the variety of tools leads/managers use to manage software projects. Some people use Excel, some people use Project, some people use Word, some people use bug tracking database, and some people use homegrown tools. Currently I’ve calmed down in regards to this and treat this as writing code with different editors. It’s not important what you used to create these lines of C# as long as it behaves as specified and has the required quality. IMHO the things which really matter are that you can always answer the following basic questions:

1. On what people in my team are currently working/worked/will be working?
2. How much time something takes/took/will take?
3. How much the current schedule deviates from the original one?

If whatever way you chose to manage your own project can provide answers to these inquiries then you should be very happy. That’s significantly better than average software project is doing anyway. I personally use Microsoft Project, but I’m not in any way "Project guru." Probably I even exercise less than 5% of Project’s functionality. Though I have to admit that working with Project makes me every time feel a little bit guilty because I never have time to spend some hours reading the documentation and discovering possibly new wonderful ways of making myself more efficient ;-( Anyhow, back to playing with my Gantt diagram.

I feel like a professional mover...

For last three weeks all I have been doing is moving things from one place to another, packing them at source and unpacking them at destination. Here’s the list of moves:

• Office move two weeks ago.
• My personal move during last weekend.
• My best friend’s move during this weekend.

My hands are bruised, my body hurts in all kind of different places and I’m already starting to adjust to the rhythm of working five days, moving during the weekend, then working again, then moving again...

The best thing about my personal move was that once again I found that Spartan lifestyle is a good thing. The less stuff the better. My place doesn’t have the following things:

• Armchairs.
• Bed.
• Coffee table.
• Couch/sofa.

I kind of managed to live without them, but I feel that the time of buying furniture and adjusting to more comfortable lifestyle is slowly but surely closing in. Maybe it’s time to cut down on working and actually start enjoying the wonders of life ;-)

Can't write code, shouldn't be testing the code either

When it comes to the software engineering then I have number of strong opinions. I wouldn’t call them radical but just different. One belief I have is that people who haven’t spent years developing software shouldn’t be testing the software either. Somehow it just doesn’t make sense and feel right - how can you tell if something is wrong if you never have gone through the pain of trying to do the right thing and therefore never understood the clear differentiation between working software and non-working one?

I’ve been referencing Edward Kit already in number of my posts. Here’s another quote (Chapter 13, page 164) from him:

There are many managers who unfortunately only see testing as a training ground for development, not a place to put their best developers. Too bad. I count myself among the managers who will not hire a person to do testing unless that person is qualified today to join the development group responsible for developing the same product that he or she would be testing.

Pretty strong words, but IMHO the correct ones. One of the basic observations in regards to this subject I made during last couple of years is that the more code you write (or have written) the better chances you have to accomplish something in software testing related positions. (Wow, I just reinvented the wheel!) Constantly reading and writing code assures that you’ll understand what this is all about. One of my recommendations to everyone has been that before testing anything at all the person should spend significant amount of time doing source code inspections and catching the first wave of bugs this way. Static source code analyzers have their limitations, buddy code reviews aren’t always as thorough as they could be (time constraints, pressure to get your own job done, natural tiredness at the end of the day), the development teams usually don't have time to spend days inspecting their code, and I feel pretty confident about arguing with anyone that the time initially spent on inspecting code gives order of magnitude more in return than just starting to hammer on the product. Catch with this is of course that it takes years of product development experience to acquire the skills of looking at the code and telling that something is wrong.

Development experience in the same application domain grows tester’s efficiency I would say again by order of magnitude. Most of the people who have gone through the entire product development cycle have learnt the hard way about how much it costs to find bugs earlier in the cycle vs. two days before shipping. Let's take a basic database application with the web interface for example. If you have done the database development yourself you know bunch of stuff about stored procedures, triggers, query optimization, indexing, transactions etc. and you can start thinking where the big problems might be. If you have done web development before then you'll know where the problems related to web-based solutions reside and so on. Yes, one can take a testing textbook and do all the classical stuff, come up with number of test cases, define equivalence classes, do a bit of performance testing, do a bit of usability testing, do a bit of security testing, do a bit of stress testing etc., but this will be IMHO just scratching the surface lightly. I would bet that in lots of root cause analysis reports when people look at why certain class of bugs hasn’t been found the answer is that nobody knew to look for them.

What am I saying here? That testing in its current form is inefficient and everyone else is wrong and I'm right? Definitely not, but I think thoughts described in this post are rather interesting subject to explore. Maybe someone could run a couple of experiments with the team consisting only of developers and divide them into two parts: coders and verifiers. One could make them periodically change roles and mix all this with a shot of Extreme Programming to look trendier and compare this to traditional software development organization developing the project with similar qualities. Or something like this.

P. S. Since I stopped drinking coffee, I have been writing these strange and provoking posts on software testing and quality assurance. I don’t know what will happen if I’ll also start having breakfast regularly ;-)

Program testing can be used to show the presence of bugs, but never to show their absence!

This quote is from Edsger W. Dijkstra and is present in number of his papers, but probably is best known from Dijkstra’s Turing Award Lecture in 1972. My current post is about the most common misconceptions about software testing I've encountered. I have found that blog is a good way to maintain a knowledge base/repository and if some FAQ relevant to subjects I’ve written about comes up then I can recommend the person to use Google and search blogs.msdn.com for some specific phrases and read up ;-)

I should probably reiterate that these are all my personal opinions:

• Testing proves that something works correctly e.g., some complex algorithm is implemented properly. First of all, testing doesn’t prove anything. Proving the correctness of some algorithm or that some piece of code behaves as specified is a very difficult task. I remember that years ago when I was young and naïve CS student we had a seminar on formal methods in software engineering and spent quite some time proving correctness of some trivial search algorithms (basic linear search, binary search). About a month ago I attended a seminar about quality assurance and quality control in airplane software engineering and I asked presenters if they use formal methods at all? I was told that no, but maybe somebody in NASA is using them to prove that something really works as specified. So, don’t feel bad if your WinForms application isn’t mathematically proved to be correct ;-) There’s this really cool article “They Write The Right Stuff” (here’s Wiki link) about the shuttle software development everyone should read at least once and think about it.
• Quantity of test cases has something to do with the testing coverage and quality. If somebody says that we should measure programmer’s productivity by number of lines of code he/she writes then everyone will look at this person and start laughing. Most people understand that this way of measuring someone's productivity is just silly. The same is applicable to number of test cases. This is one clear example where quality matters more than quantity. For me personally there’s no difference if some team maintains 10 or 10,000 test cases. The number of different combinations and permutations grows so quickly that it’s no-brainer to come to work during the weekend and on Monday morning have 1000 valid, but useless test cases in your test case management system.
• Verification and validation is the same thing. Nope. Here’s the definition of verification (stolen from IEEE/ANSI standard):

  Process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of the phase.

  It’s very hard and very expensive to do the verification of something and to the best of my knowledge there are not much test teams out there who do verification. What most teams do is called validation which is per IEEE/ANSI:

  Process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements.

Next time - why do I think that to test something you should be able to have the same technical skills and knowledge as if you were the developer on the same project? I’ve been having this opinion forever, but this doesn’t seem to be very popular belief (I haven't found anyone yet who shares it  - people could start posting “Me too!“ comments ;-)) and I was pretty shocked when Edward Kit’s "Software Testing in the Real World: Improving the Process" describes the same attitude in Chapter 13 "Organization approaches to testing." At least I don’t think that I’m totally foolish anymore ;-)

Static source code analysis is finally taking off

I was reading through all the posts at http://blogs.msdn.com/ I missed during my vacation time and it seems that (static) source code analysis tools are in. Roberto Santos has started code analysis blog where he talks about FxCop and PREfast. Since I used lint first time a number of years ago I’ve been hooked up with its derivatives and any other tools which make possible for me to write better code and find errors before users see them. I don’t subscribe to many RSS feeds, but this one I’ll be definitely reading ;-)

Check out also the following article on CNet - "Will code check tools yield worm-proof software?" I’m telling you - soon we’ll see people wearing T-shirts with "My code compiles cleanly with PREfast!" on them ;-) But be careful because analyzers aren’t infallible. Common sense and an ability to think clearly is still developer’s best friend.

Software testing is not software quality assurance!

At least twice a month I have an argument with somebody about differences between software testing and software quality assurance. So I decided to write up a permanent piece about this. What bothers me is that lots of people use these two terms interchangeably without understanding the major differences between them. Even worse, I’ve been noticing myself starting using these terms interchangeably ;-(

But that’s not the point. When it comes to software testing there are usually number of names which come immediately to everyone’s mind: Boris Beizer, Cem Kaner, Glenford J. Myers, James A.Whittaker etc. All these people are very well known authorities, but for me personally Myers’s "The Art of Software Testing" is the book. Possibly because of the name it reminds me Knuth's work about programming ;-) So, how Myers defines testing? Very simply ("The Art of Software Testing", Chapter 2 - "The Psychology and Economics of Program Testing", page 5):

Testing is the process of executing a program with the intent of finding errors.

That does it for me. For alternative definitions you can check all your favorite software engineering manuals and this FAQ. The best book discussing the definitions of testing I’ve found so far is Edward Kit’s "Software Testing in the Real World: Improving the Process". It provides eight different historical definitions of testing and also IEEE/ANSI definitions. Chapter 4, if anybody is interested.

This brings us back to a second part of this riddle - what is software quality assurance? I borrowed this from FAQ because I’m too lazy to type in paragraphs of text from some thick software engineering manual:

Software quality assurance involves the entire software development process - monitoring and improving the process, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and dealt with.

There are lots of things in software world which are close, but are not the same thing if you dig deeper. For example: { JavaScript and JScript, Linux and Unix, HTML and XHTML }. My friends tell me that I’m wasting my time and worrying too much about little things like this instead of trying to enjoy life and reduce my stress level ;-) Probably they’re right, but it takes while to change one’s personality.

P. S. Some cool provoking discussions from Wiki: QualityAssuranceIsNotQualityControl and QualityAssuranceIsNotResponsibleForQuality.

My New York City impressions

Coming back from vacation is like coming back from another reality. You discover hundreds of unread e-mails waiting for you; first day at work starts right away with the fire drill which lasts till the end of the week; your office is being moved from one building to another; in addition you need to move your own belongings from one apartment to a place you just bought. In one sentence, everything’s back to normal.

My girlfriend and I spent nine days at NYC. After first thirty seconds outside the hotel we immediately realized two things:

1. When in Rome, behave like Romans do.
2. Adapt, blend in, improvise, and overcome.

Following this piece of enlightenment it became much easier – at the end of the day we were crossing streets like old-timers, looking strangely at people who were studying street maps in the middle of side-walk, and classifying everyone who walked slower than majority of people as tourists ;-)

As nine days is quite a bit of time, we made a decision that we’ll explore the entire Manhattan on foot - no bus, no taxi, and no subway. Sounds crazy, doesn’t it? At the end it turned out to be a very good decision. We spent first two days going through all classical tourist attractions: Broadway, Central Park, Empire State Building, Times Square etc. In the beginning of the third day we actually started to relax a little bit which caused another problem - when there are two people with different interests for sightseeing then you eventually end up in the painful negotiation process trying to achieve the win-win situation. We both spend a significant amount of our usual workday finding the middle ground in all kind of different decisions, but at the same time we’re both also used to get what we want. Three-four years ago my negotiation tactics were quite simple: we do everything my way or we won’t do it all ;-) Life makes its corrections, I’ve grown older and for next couple of days we reached a compromise - she took me to the Metropolitan Museum of Art and tried to educate me on various subjects and I got my desired visit to Brighton Beach (the only time we used subway in NYC). Brighton Beach is probably kind of nostalgic place for everyone who spent part of their childhood in former Soviet Union. To me personally it reminded Estonia in the middle or at the end of eighties ;-)

Though I don’t even pretend that I care or understand anything about art, Metropolitan Museum of Art was impressive, but unfortunately overwhelming. After half a day all I could say was "Yeah, another Monet, whatever, let’s get something to eat." At the same time I wasn’t classified as totally uncivilized human being, because without twisting words in my mouth I was able to admit that "The Phantom of the Opera" we went to see next day on Broadway, was excellent.

Next couple of days we just spent hanging around across Manhattan. There are number of things which I was told before going to vacation: "NYC is dirty", "Manhattan Financial District is extremely crowded", "people in NYC are unfriendly", "NYC is too fast for you" and so on. Fortunately I found none of these predictions to be true (at least based on the short amount of time I spent in the city if it counts for anything). The fact I can confirm is that NYC is expensive ;-) At least compared to the life I’m leaving in Seattle area.

To summarize: as in all good fairytales, we felt at the end of the first day that we love NYC (how cliché is this ;-)) and any chance we get we would like to come back to this place.