Avoid multiple asserts in a single unit test: revisited

James Avery argues that what he'd really like to see in a unit test framework is the ability to run (and fail) multiple asserts within the same test.

As some background, I personally am in favor of the "single assert per unit test" idea for several reasons, the most prominent of them is that currently, all unit test frameworks I know in .NET will fail the test on the first failing assert call.

The reason this is how it works is that the framework detects failure by catching a special type of exception (AssertionException if I recall correctly) that is thrown by the assert method. you can't really catch an exception and keep on going to the next line unless the framework automagically instruments each line of assert code in the test with a try-catch (which would be one way to archive James' request)

So currently, what that means that if you have other asserts after the one that failed - they won't run and you won't know if they would have succeeded or not, thus giving you only a partial picture of what's failing and what isn't, which is sometimes a terrible way to try and discover a bug. the other reasons why you'd want to have only a single assert per test are that a test with multiple asserts is actually testing multiple things, so it may as well be considered as multiple tests. A good "tell" sign is when it's hard to name a test because it does several things instead of just one.

There is another very important reason which I neglected to talk about in my previous posts or articles, which I have recently realized.

Even if we assume that all the asserts are run, you're essentially running multiple tests on code that has "dirty" state. For example, asserting on the result of a method call may actually change the state of the object under test so that the next call to that method may actually be skewed because of the previous asserts. that's a bad world to be in. having single assert per test means you also know exactly the state of your object before the assert.

Worst yet, you may not have the object state changed now when you're writing the current tests, but sometime in the future you may add that state changing ability to some method or property call and suddenly some of your tests will break unexpectedly, but only on the 2nd or 3rd assert call in the same test. That's a bug that would be very hard to find, because it's actually in the tests, and the behavior is expected.

Having said that, I'd like to walk through James' points one by one as to why he'd like to have multiple-assert ability in the framework and why separating tests per assert is not his cup of tea.

 

  • "I think it’s actually harder to read since the assertions are scattered around in separate methods. "

It shouldn't be harder to read if each test was really simple and was named correctly, and in fact, James' tests are pretty readable as far as I'm concerned. they just look like a lot more tests than his "simple" example. I actually think that his "simple" example may seem easier to read, but the tests are harder to understand unless you actually read through all the code in them.

  • "It would increase the number of tests. On my current project we have 1800 tests, if we followed the one assertion rule we would have over 6,000 I am sure. "

So? In essence what you really are doing is running 6000 tests that only appear to be 1800. that might mean that if one of the 1800 tests breaks it may be harder to realize what the problem is without reading through that test code and finding the actual assert that failed (aka "debugging"), and even then you may have a hard time finding the bug if you have asserts in that test that haven't even run.

 

  • "If my method breaks and starts returning null then I have 8 tests failing instead of just 2, this means I have to know the dependency tree of my tests to find the real issue. "

That is true also when you have multiple asserts in the same test, I would think, or you probably wouldn't have written them all together. so if you had two failing tests, how many failing asserts would you have? you may have had only 2. you really want to be sure that your other asserts would fail given that the first two fail, which is not always the case (it is the case in James' examples).  Imagine that you had 4 small failing tests and four small passing tests - wouldn't that be great instead of two big failing tests? 

for example your method could still return null, but a you may have an assert that shows that a specific state of the object is actually true, as you expect it to be. If that assert never runs, you can't know that and might look in the wrong place to fix the bug. if it is in a separate test you know that it succeeded. if it is run inside the test, you again face the possibility of changing the object's state as mentioned earlier.

 

  • "Any code I have to write to setup the data for my test has to be duplicated 8 times. (if I move that setup data to the setup method than I am effectively limiting my fixtures to one fixture per test) "

You can and should extract setup code for specific tests which are reusable into an external private method (XXX_init for example) that is called from each of those tests. No one forces you to use the setup method for that. the setup method is there only for global setup routines, which are used by all the tests.

 

  • "I now have over double the amount of code. I am constantly trying to reduce the amount of code in my project, whether test or production, and anything that doubles it better add a ton of value."

if the tests are concise and code is refactored correctly inside the tests, that "larger" amount of code is simply a better syntactical way to express exactly what you did before  - write all the tests you wanted, but they are well separated and solid and readable. Just like you may want to write some method calls in two lines of code instead of one for the sake of readability, this is the same idea, but for the sake of solid confined tests.

Published Wednesday, October 04, 2006 12:30 AM by RoyOsherove
Filed under: , , , ,

Comments

Tuesday, October 03, 2006 6:41 PM by Electronic Blues » Blog Archive » Multiple Assertions in One Unit Test

# Electronic Blues » Blog Archive » Multiple Assertions in One Unit Test

PingBack from http://www.electronicblues.com/index.php/2006/10/03/multiple-assertions-in-one-unit-test/

Tuesday, October 03, 2006 7:02 PM by Oran

# re: Avoid multiple asserts in a single unit test: revisited

So what if my method-under-test is responsible for coordinating and setting up multiple things? How do you verify that each thing has been set up properly by this method without doing multiple asserts?
Tuesday, October 03, 2006 7:21 PM by RoyOsherove

# re: Avoid multiple asserts in a single unit test: revisited

Oran: I'd advise a separate test for each check, each one calling the method on an preinitialized object and then doing one assert.

That way each test can also be named appropriately.

Tuesday, October 03, 2006 9:53 PM by Ben Scheirman

# re: Avoid multiple asserts in a single unit test: revisited

Roy, While I completely agree with you in theory, I find it very time consuming to do that and I feel that the benefit isn't always warranted. For example, say you create a simple class and you write some tests for it. I'd start of with making sure that the constructor satisfied the invariants (like: User u = new User("Fred", "Flinstone"); Assert.AreEqual("Fred", u.FirstName, "First name wasn't set"); Assert.AreEqual("Flinstone", u.LastName, "Last name wasn't set"); in this example I do 2 asserts, but the test is really CanCreateUserWithInvariants(). The message can tell me which assert failed. Is this wrong? would you really separate this into 2 tests? My 2nd comment (really a question) is this: I find myself wanting to create DEPENDENT tests, and before you shudder think of this: Say you have some functionality that you build upon in your tests. Your model grows and each addition is well tested. Now you want to test higher level abstractions, so these tests now DEPEND on the functionality of other tests, like making sure our User class can actually set its own properties. If that part fails now I have 2 (or likely a lot more) failing tests. If NUnit had an attribute or something that specified DependsOn("CanCreateUserWithInvariants") then the GUI can mark those as Purple or something to say that other tests must be passing before this one runs. I realize this opens the door for serious abuse of dependent testing, but maybe you could shed some light on it for me or recommend a better way of testing :) How can I ensure that my tests are atomic and singular, but also leverage the code I've written to provide even more robust tests?
Tuesday, October 03, 2006 11:24 PM by Kent Boogaart

# re: Avoid multiple asserts in a single unit test: revisited

Hi Roy, great post as per usual. I have recently trended towards smaller, more targetted unit tests myself. Whilst I find that this makes tests easier to read and failed tests easier to rectify, I find that writing them is more burdensome. When you get a chance, would you mind blogging a bit about alternatives to implementing unit tests in C# / VB / whatever. Are there any unit test-specific CLR languages out there that would make unit tests easier to write, read and maintain? Thanks, Kent
Wednesday, October 04, 2006 3:20 AM by Michal Talaga

# re: Avoid multiple asserts in a single unit test: revisited

I do not agree with the part of having to know which one of the asserts failed. As long as the whole test fails it fails. Sure it would be easier to find the concrete assert just by looking at the code, but ... I preffer less test code overall than less code in one method, but that is my opinion. And for your second argument. If you expect your method to run differently after each call you should also test that it does it in a correct, intended way and write your tests with this in mind.
Wednesday, October 04, 2006 4:46 AM by Andrea Raimondi

# re: Avoid multiple asserts in a single unit test: revisited

Hi Ray. When I'm using UT, I usually have more than one assert per test. Why? Simple: when an assert is successful you have a perfect consistency of the object state and using messages you can easily determine what was going wrong. Plus, sometimes you need to test more "aspects" of a feature and doing several tests instead of one covering more aspects is dispersive in the task switching. That said, I'm not advocating one test method per class, rather I'm advocating one test method per logical block inside the class. For example: if I have a class that renders an ul HTML tag and various li's and then I add a property to the li class like "useAnchor" and an "AnchorUrl" one, then it's likely I'll want to test those in the li tag generation rather than in another test which would inherently be burdensome to write. Cheers, Andrew
Wednesday, October 04, 2006 9:55 PM by James Avery

# re: Avoid multiple asserts in a single unit test: revisited

I think the main reason (that the first assert fails and the others don't run) is a valid reason to have single assert per test, but I think if that was mitigated it would largely become a matter of preference. You mention that you would extract the setup code to a private method, but you now run this on every test. So if you setup code creates 5 new records in the DB, you know have to do that 10 times instead of one. This is a problem we have on our current project, we have so many tests and so much startup code that it takes forever to run all of our unit tests. I tried to think of an instance of where I have written an assert that changes the state of an object and I couldnt think of one... could you provide some examples? -James
Thursday, October 05, 2006 8:47 AM by Michael Dorfman

# re: Avoid multiple asserts in a single unit test: revisited

Once again, you hit the nail right on the head, Roy. Nice job. How are things going, by the way? Drop me a line when you have a chance....
Saturday, October 07, 2006 1:40 PM by Oran

# re: Avoid multiple asserts in a single unit test: revisited

One test I've written recently that has two Asserts is for a method that is responsible for taking a single object and creating a reference to it from two different places. So I pass in my object and verify that the two locations have references to it when I'm done. If you need to know which Assert failed in a failed test, put in a string description for your Assert that tells you which Assert failed. I think of this kind of a test as an all-or-nothing transaction. Sure, I could break it down further, but if you think of your unit test as being your specification, all you really want to specify is that the entire transaction succeeds. Anything more than that is simply a debugging aid, and there's more than one way to accomplish that, with my preference being meaningful messages for my Asserts. Another thing to keep in mind with the single-assert mantra is that any verification of mock object usage is essentially another Assert. So if you have an Expect.Call(obj.MyMethod()).Return(true) and later have a mocks.VerifyAll(), you are essentially asserting that MyMethod was called once. You can omit the VerifyAll() call to eliminate the assert and make sure you have something like a Repeat.Any() on each mock expectation to eliminate the implicit assert on the number of times the mock method is allowed to be called, but it's very very easy to have lots of implicit asserts in tests that use mock objects.
Wednesday, October 11, 2006 4:49 PM by Jeremy Harning

# re: Avoid multiple asserts in a single unit test: revisited

"So? In essence what you really are doing is running 6000 tests that only appear to be 1800. that might mean that if one of the 1800 tests breaks it may be harder to realize what the problem is without reading through that test code and finding the actual assert that failed (aka "debugging"), and even then you may have a hard time finding the bug if you have asserts in that test that haven't even run." The problem with this is that it takes too long. Consider 15 developers on a team checking code in all day in a continuous integration evironment. Say 1800 tests takes 10 minutes to run. Also assume that you would follow an atomic testing strategy (each test setting up and tearing down the data that it needs), most of the 10 min that it takes for your test harness to run is spent on setting up and tearing down the data. So all the sudden you have to do that 6000 times instead of 1800, and your test harness takes 45 min run instead of 10. 45 Min X 15 Developers X 3 Times a day = a boatload of wasted time. For what? So that you would instantly know that your data layer is omitting 1 row as opposed to not returning any rows without having to look at or debug through any code? You are still only testing one method, which is the thing you need to debug anyways, at least this way you have all the asserts right in front of you to quickly figure out the problem.
Friday, October 13, 2006 8:29 PM by .Avery Blog

# Multiple Assertions per Unit Test

Multiple Assertions per Unit Test

Sunday, October 15, 2006 1:05 AM by Upchurch

# re: Avoid multiple asserts in a single unit test: revisited

I hate it when I get a Nullexcetion with Assert.IsTrue(addressList[0] != null,) because my list is null, I've much rather have an Assert.IsNotNull(addressList) fail on the line before, to give me a better idea what went wrong. I realize the idea is you would have 2 test, and they both would fail, but the problem is one would fail because of the assert, and the other would fail because of a null exception, thus hiding the real error.
Friday, November 03, 2006 5:16 AM by Avi

# re: Avoid multiple asserts in a single unit test: revisited

We can always use a higher-level assertion (such as CollectionAssert.AreEqual, CollectionAssert.AreEquivilent) or custom assertion (such as an AssertPortIsOpened, AssertPictureDimentionsAreEqual) to reduce the number of assertions per test. This usually also improved the readability of the tests and conveys their meaning better (not to mention code-reuse of the asserions). As for one assertion causing state-alterations that would disrupt later assertions - that's a tough one.. It could happen in any number of situations when asserting on an object instance. You don't really know what happens inside the assertXXX() methods of xUnit, so you might be calling CollectionAssert.Equals() just once on your collection, but inside it makes several calls to the collection's members. On the other hand, if the collection should be changing it's state when you call it's Count() method, this state-alteration would be seperately tested elsewhere, and is probably not supposed to affect any other calls anyway, so I don't find this to be a serious argument neither for nor against multiple-assertions. On the contrary - if one call alters the state of the object-under-test so as to alter the result of another call, you *should* make sure your tests invokes the calls in the proper order.
Wednesday, November 29, 2006 5:00 AM by Chris Hedgate

# re: Avoid multiple asserts in a single unit test: revisited

Great post. As I recently wrote (http://www.hedgate.net/articles/2006/11/17/one-assertion-per-test-should-come-natural), I have come to think that multiple asserts in a test method is often a smell of badly organized tests. If tests are organized by fixture (as in having a common setup) it is more natural to have a single assert per test method.

For the examples in James blog post there would be a fixture called ListOfBooksReturnedForAuthor1 (or similar). This fixture would then have six test methods, each with a single assert.

(Sorry if this was double posted, my first comment did not seem to show up).