Test Coverage and testing angles

How do you know if you have good coverage on your new code?

Try and remove a line or a constraint check. If all tests still pass – you don't have enough code coverage and you probably need to add another unit test.

The best way to make sure you are adding the correct test is to not let yourself uncomment that line or check until you can produce a test that will fail until you uncomment it. It may be hard but if you can’t think of a way this would fail, you probably don’t have a good excuse for writing that line of code, do you?

 

Why is this a problem? Because you never know when the next developer will show up and try to play with your code, maybe try to optimize it, or wrongly delete some precious line of your code.  If you don’t have a test that will fail, they will never know they made a mistake.

 

You might also want to try and replace various usage of parameters that are passed in to your method with consts. For example given this method,

 

Public int Sum(int x, int y, bool allowNegatives)
{
   If(!allowNegatives)
{
        If(x<0||y<0)
        Throw new Exception();
}
   Return x+y;
}

 

here are some variations of testing for test coverage:

   

If(!true)
{
        If(x<0||y<0)
        Throw new Exception();
}
 
Or
 
   If(!allowNegatives)
{
        If(false||y<0)
        Throw new Exception();
}
 
Or
 
If(!allowNegatives)
{
        If(x<0||false)
        Throw new Exception();
}

If all the tests still pass, you’re missing a test.

 

Another red flag: If you have only one test that checks for various equality to values.

Seeing this:

Assert.AreEqual(3,retval);

Happen only once in relation to some method usually means you can safely return 3 as a value and all the tests for this method will still pass.

 

Published Sunday, April 03, 2005 12:54 PM by RoyOsherove

Comments

Sunday, April 03, 2005 8:35 AM by Nauman Leghari

# re: Test Coverage and testing angles

A really good observation. I just realized that this technique can be very effectively used in unit testing existing code. Which is difficult in normal circumstances.
So,

a) comment out existing functionality that we want to check before writing any test. b) Then write tests
c) Uncomment only the lines which are affected by the previously written test.

Any comments?
Sunday, April 03, 2005 9:42 AM by Roy Osherove

# re: Test Coverage and testing angles

Nauman:
It might be possible, some of the time. But sometimes testing existing code means testing a whole bunch of code together (no simple way about it). I don't see myself commenting out 50 lines of code just so a test will fail.
I'd personally only use this if I have specific tests, not generic ones.
Monday, April 04, 2005 6:39 AM by Paul Bartlett

# Re: Test Coverage and testing angles

I believe there are actually tools that make these sort of modifications automatically to the executables, but I can't recall off hand whether they are for Java or .NET (i.e. modify bytecode or IL). If I manage to nudge my memory sufficiently I'll post another comment...
Monday, April 04, 2005 11:05 AM by Paul Bartlett

# re: Test Coverage and testing angles

The approach is called "mutation testing", and there are some tools available (http://www.xpdeveloper.com/xpdwiki/Wiki.jsp?page=MutationTestingTools) but unfortunately none I've seen for the .NET environment.
Monday, April 04, 2005 11:17 AM by Paul Bartlett

# re: Test Coverage and testing angles

Wrong again! Peli has written about it (http://blog.dotnetwiki.org/archive/2004/05/22/255.aspx) and mentions nester (http://nester.sf.net) which I think I missed in the list I linked to in my previous comment...
Monday, April 04, 2005 3:34 PM by TrackBack

# OdeToCode Links For April 4

Monday, April 04, 2005 11:04 PM by TrackBack

# New Team System Stuff - 2005-04-05

Visual Studio Team System

Today, we have a double-header of Team System webcasts:

9:00 AM PDT &ndash;...