Try to avoid multiple asserts in a single unit test - ISerializable - Roy Osherove's Blog

Try to avoid multiple asserts in a single unit test

Consider Assert failures as symptoms of a disease and Asserts as indication points or blood checks for the body of the software. The more symptoms you can find, the easier the disease will be to figure out and remove. If you have multiple asserts in one test - only the first failing one reveals itself as failed and you lose sight of other possible symptoms.
 
 
 
I've written about this before but I want to stress this again because this topic is more important than you might think.
 
Void Sum_AnyParamBiggerThan1000IsNotSummed()

   Assert.AreEqual(3, Sum(1001,1,2); 
   Assert.AreEqual(3, Sum(1,1001,2); 
   Assert.AreEqual(3, Sum(1,2,1001);
}
 
There are several disadvantages to using multiple asserts in one test case.
The main one is this:

Sometimes you get a better  idea of what's wrong with the code if you have multiple test failures.
 
When you have multiple asserts in one single test, only the first one fails and the test execution stops for this test case. The rest of the lines in the test code never get checked - you're losing some more points of view (symptoms) that you might consider important in order to quickly discover where the problem stems from.
 
Another example - you're changing a piece of code and suddenly 6 tests that might seem unrelated break. If you're lucky you might realize that these tests all have one thing in common - they use some common object or method that does not work anymore. That is - it's easier to realize that the problem probably does not lie within the tests  - they are correct and logical,  but their failure is merely a symptom of a deeper problem that needs attention.
 
Now assume the same scenario with only one test failing - a test with multiple asserts.
Since you now see only one "symptom" you might "diagnose" the disease wrongly because you don't have all the facts. That means you'll probably spend more time investigating this problem and struggling with it  -- one of the things I see most often in these cases is this:
 
once you make one assert pass somehow, another one "reveals" itself in the same test and fails as well. You don't get the big picture and have to realize it step by step--Along the way you you might even make "corrections" to the code which might not have been needed but "fixed" it from a different view - fixed the symptom but not the problem. You might have introduced more changes to the system and even made it less stable in the process - all the while taking more precious time.
 
 
More notes to consider:
 
If I was writing this test after  the production code was already written, I might not have a problem with this in some situations (pressed for time etc..). But if this test is written before the tested code is written, it might prove less optimal for several reasons:
 
 You'll  spend a longer amount of time writing code to make the test pass.
   The longer you spend without running tests, the less confident you become.
 
 
You're not focusing on a small problem anymore.
You're focusing on a combined, larger problem, more generic  which should be solved ("this and this and this should all pass at once"). It's easy to get stuck on large problems. It's  easier to keep going while continually solving small problems one after the other.
 
 
 
The more time you spend not running the tests, the more code you write that has not been tested,
and which will probably depend on more code that has not been tested. You re more likely to present unforeseen bugs.
The previous paragraph should not come as a surprise because the more code you write between each unit tests, the smaller the likelihood that your test actually covers that code. That s because:
 
 
 
Trying to make a test with three ASSERTs pass, is just like trying to make three separate test cases pass in a single bounce:
You'll have to write more code that handles more cases and you ll probably end up writing it more generic that you would have if you were just trying to fix one test case at a time. You won't really know if your tests cover all the generic code you've just written as you would have known by making the tests pass one by one, in the simplest way possible.
 
 
 
Once you have multiple ASSERTs in a test case, it's very easy and human to keep adding more asserts into the existing test than it would be to add them using new tests. Even if those were totally different asserts in nature, the broken window theory works very well here, and leads to less maintainable and less readable tests.
 
   
 
You're working more time without the "success" feedback of the green bar.
Like it or not, the psychological effect here is important. It leads to a better coding experience and more confidence and happiness at work.
 
 
Hopefully all of these reasons should make you think twice before just jumping in without attention to your tests. You'd be surprised at how fast these symptoms add up to make the testing experience either easier and faster or much more repetitive and painful that it could have been.
Published Thursday, April 14, 2005 11:42 PM by RoyOsherove

Comments

Thursday, April 14, 2005 9:05 PM by Dale Emery

# re: Try to avoid multiple asserts in a single unit test

I picked up this piece of advice somewhere a year or so ago, and it's had a big impact on my testing.

There are two situations in which I use multiple asserts. The first is when additional information would be noise. For example:

Assert.IsNotNull(actualValue);
Assert.AreEqual(expectedValue, actualValue);

If I put these into separate methods, both tests fail (one assertion failure, one null reference exception). The null reference exception doesn't give me useful information beyond what I already know from the IsNotNull test. In fact, it's distracting noise. In this situation, I prefer to put the assertions in the same test method.

The other situation is when I'm testing a sequence of operations:

testSubject.doStep1();
Assert something about step 1
testSubject.doStep2();
Assert something about step 2
testSubject.doStep3();
Assert something about step 3
...

Even in this case, I prefer to test each step separately. But I'll test them together if, for example, I don't have a way to prep for doStep3() other than executing doStep1() and doStep2(). In other words, if I don't have enough control over the test subject that I can directly establish the entry conditions for doStep3().

I've started to consider that situation to be a design smell. Rather than splitting the tests, I now try to change the test subject to give me the direct control I want.

But even so, something tells me that I may want to test the sequence as above. The jury is still out on this one.

But in the vast majority of situations, my strong preference is to write one assertion per test.
Friday, April 15, 2005 8:22 AM by David Chelimsky

# re: Try to avoid multiple asserts in a single unit test

In this case:

testSubject.doStep1();
Assert something about step 1
testSubject.doStep2();
Assert something about step 2
testSubject.doStep3();
Assert something about step 3

I'll usually do something like this:

public void testStep1()
{
testSubject.doStep1();
Assert something about step 1
}

public void testStep2()
{
testSubject.doStep1();
testSubject.doStep2();
Assert something about step 2
}

public void testStep3()
{
testSubject.doStep1();
testSubject.doStep2();
testSubject.doStep3();
Assert something about step 3
}

Not to say that there's not a design smell in there to uncover, but this gives you the separation of assertions you're looking for. In fact, it gives you very useful information about the sequence depending on what fails. I've even seen cases where testStep1 fails but the other 2 pass because something in the implementation of doStep2 righted the wrong imposed by doStep1.

Cheers,
David
Thursday, April 21, 2005 5:23 PM by Gary Feldman

# re: Try to avoid multiple asserts in a single unit test

I think you've mischaracterized the problem.

In

Assert.AreEqual(3, Sum(1001,1,2);
Assert.AreEqual(3, Sum(1,1001,2);
Assert.AreEqual(3, Sum(1,2,1001);

you don't just have multiple asserts. You also have multiple invocations on the function under test - and that's the flaw.

Compare that with, for example:

void test_ParseUserNameSimpleCase1() {
user = ParseUserName("John Q. Doe");
Assert.AreEqual(user.firstname, "John");
Assert.AreEqual(user.lastname, "Doe");
Assert.AreEqual(user.middlename, "Q.");
}

This has multiple asserts, all of which are really necessary for this particular case, but only one call on the function under test. Doing this as three separate test methods would just make it harder to read the tests and figure out what was being tested. If, for example, you left out the middlename assertion, it would be easier to spot this here than if the assertions were scattered. It's arguable whether separating them out would make it easier to diagnose any problems.

Also, there are times when it's justified to have multiple calls to the method under test, if it's stateful. But in such a case, it would be suspicious if the assertions verified anything other than the final state. In other words, I agree with David's example above, except that I would permit multiple assertions at the end, after all calls on testSubject.doStepX were done.

Finally, just because NUnit today stops at the first assertion failure doesn't mean that that's cast in stone. I see no reason that there couldn't, in theory, be AssertAndContinue.xxx, AssertAndAbort.xxx, and even AssertAndAbortFixture.xxx. In other words, the core problem isn't really putting multiple asserts into a test method; it's a limitation of the particular test framework.

Gary

Saturday, May 14, 2005 10:05 AM by Dustin Campbell

# re: Try to avoid multiple asserts in a single unit test

I agree with this, you might consider changing "multiple asserts" to "multiple tests".

One other point, in my opinion this is OK:

testSubject.doStep1();
Assert something about step 1
testSubject.doStep2();
Assert something about step 2
testSubject.doStep3();
Assert something about step 3

However, your assertion should include a *good* message that describes what went wrong. That will direct you to the problem spot.
Tuesday, September 12, 2006 1:45 AM by jokiz's blog

# MbUnit vs NUnit

Roy Osherove has illustrated the need to Avoid Multiple Asserts in a test method. Basically i agree with

Monday, September 18, 2006 9:21 PM by jokiz's blog

# MbUnit vs NUnit

Roy Osherove has illustrated the need to Avoid Multiple Asserts in a test method. Basically i agree with

Tuesday, September 26, 2006 2:41 AM by jokiz's blog

# MbUnit vs NUnit

Roy Osherove has illustrated the need to Avoid Multiple Asserts in a test method. Basically i agree with