ISerializable - Roy Osherove's Blog

PingBack from http://www.electronicblues.com/index.php/2006/10/03/multiple-assertions-in-one-unit-test/

So what if my method-under-test is responsible for coordinating and setting up multiple things? How do you verify that each thing has been set up properly by this method without doing multiple asserts?

Roy, While I completely agree with you in theory, I find it very time consuming to do that and I feel that the benefit isn't always warranted. For example, say you create a simple class and you write some tests for it. I'd start of with making sure that the constructor satisfied the invariants (like: User u = new User("Fred", "Flinstone"); Assert.AreEqual("Fred", u.FirstName, "First name wasn't set"); Assert.AreEqual("Flinstone", u.LastName, "Last name wasn't set"); in this example I do 2 asserts, but the test is really CanCreateUserWithInvariants(). The message can tell me which assert failed. Is this wrong? would you really separate this into 2 tests? My 2nd comment (really a question) is this: I find myself wanting to create DEPENDENT tests, and before you shudder think of this: Say you have some functionality that you build upon in your tests. Your model grows and each addition is well tested. Now you want to test higher level abstractions, so these tests now DEPEND on the functionality of other tests, like making sure our User class can actually set its own properties. If that part fails now I have 2 (or likely a lot more) failing tests. If NUnit had an attribute or something that specified DependsOn("CanCreateUserWithInvariants") then the GUI can mark those as Purple or something to say that other tests must be passing before this one runs. I realize this opens the door for serious abuse of dependent testing, but maybe you could shed some light on it for me or recommend a better way of testing :) How can I ensure that my tests are atomic and singular, but also leverage the code I've written to provide even more robust tests?

Hi Roy, great post as per usual. I have recently trended towards smaller, more targetted unit tests myself. Whilst I find that this makes tests easier to read and failed tests easier to rectify, I find that writing them is more burdensome. When you get a chance, would you mind blogging a bit about alternatives to implementing unit tests in C# / VB / whatever. Are there any unit test-specific CLR languages out there that would make unit tests easier to write, read and maintain? Thanks, Kent

I do not agree with the part of having to know which one of the asserts failed. As long as the whole test fails it fails. Sure it would be easier to find the concrete assert just by looking at the code, but ... I preffer less test code overall than less code in one method, but that is my opinion. And for your second argument. If you expect your method to run differently after each call you should also test that it does it in a correct, intended way and write your tests with this in mind.

Hi Ray. When I'm using UT, I usually have more than one assert per test. Why? Simple: when an assert is successful you have a perfect consistency of the object state and using messages you can easily determine what was going wrong. Plus, sometimes you need to test more "aspects" of a feature and doing several tests instead of one covering more aspects is dispersive in the task switching. That said, I'm not advocating one test method per class, rather I'm advocating one test method per logical block inside the class. For example: if I have a class that renders an ul HTML tag and various li's and then I add a property to the li class like "useAnchor" and an "AnchorUrl" one, then it's likely I'll want to test those in the li tag generation rather than in another test which would inherently be burdensome to write. Cheers, Andrew

I think the main reason (that the first assert fails and the others don't run) is a valid reason to have single assert per test, but I think if that was mitigated it would largely become a matter of preference. You mention that you would extract the setup code to a private method, but you now run this on every test. So if you setup code creates 5 new records in the DB, you know have to do that 10 times instead of one. This is a problem we have on our current project, we have so many tests and so much startup code that it takes forever to run all of our unit tests. I tried to think of an instance of where I have written an assert that changes the state of an object and I couldnt think of one... could you provide some examples? -James

Once again, you hit the nail right on the head, Roy. Nice job. How are things going, by the way? Drop me a line when you have a chance....

One test I've written recently that has two Asserts is for a method that is responsible for taking a single object and creating a reference to it from two different places. So I pass in my object and verify that the two locations have references to it when I'm done. If you need to know which Assert failed in a failed test, put in a string description for your Assert that tells you which Assert failed. I think of this kind of a test as an all-or-nothing transaction. Sure, I could break it down further, but if you think of your unit test as being your specification, all you really want to specify is that the entire transaction succeeds. Anything more than that is simply a debugging aid, and there's more than one way to accomplish that, with my preference being meaningful messages for my Asserts. Another thing to keep in mind with the single-assert mantra is that any verification of mock object usage is essentially another Assert. So if you have an Expect.Call(obj.MyMethod()).Return(true) and later have a mocks.VerifyAll(), you are essentially asserting that MyMethod was called once. You can omit the VerifyAll() call to eliminate the assert and make sure you have something like a Repeat.Any() on each mock expectation to eliminate the implicit assert on the number of times the mock method is allowed to be called, but it's very very easy to have lots of implicit asserts in tests that use mock objects.

"So? In essence what you really are doing is running 6000 tests that only appear to be 1800. that might mean that if one of the 1800 tests breaks it may be harder to realize what the problem is without reading through that test code and finding the actual assert that failed (aka "debugging"), and even then you may have a hard time finding the bug if you have asserts in that test that haven't even run." The problem with this is that it takes too long. Consider 15 developers on a team checking code in all day in a continuous integration evironment. Say 1800 tests takes 10 minutes to run. Also assume that you would follow an atomic testing strategy (each test setting up and tearing down the data that it needs), most of the 10 min that it takes for your test harness to run is spent on setting up and tearing down the data. So all the sudden you have to do that 6000 times instead of 1800, and your test harness takes 45 min run instead of 10. 45 Min X 15 Developers X 3 Times a day = a boatload of wasted time. For what? So that you would instantly know that your data layer is omitting 1 row as opposed to not returning any rows without having to look at or debug through any code? You are still only testing one method, which is the thing you need to debug anyways, at least this way you have all the asserts right in front of you to quickly figure out the problem.

Multiple Assertions per Unit Test

I hate it when I get a Nullexcetion with Assert.IsTrue(addressList[0] != null,) because my list is null, I've much rather have an Assert.IsNotNull(addressList) fail on the line before, to give me a better idea what went wrong. I realize the idea is you would have 2 test, and they both would fail, but the problem is one would fail because of the assert, and the other would fail because of a null exception, thus hiding the real error.

We can always use a higher-level assertion (such as CollectionAssert.AreEqual, CollectionAssert.AreEquivilent) or custom assertion (such as an AssertPortIsOpened, AssertPictureDimentionsAreEqual) to reduce the number of assertions per test. This usually also improved the readability of the tests and conveys their meaning better (not to mention code-reuse of the asserions). As for one assertion causing state-alterations that would disrupt later assertions - that's a tough one.. It could happen in any number of situations when asserting on an object instance. You don't really know what happens inside the assertXXX() methods of xUnit, so you might be calling CollectionAssert.Equals() just once on your collection, but inside it makes several calls to the collection's members. On the other hand, if the collection should be changing it's state when you call it's Count() method, this state-alteration would be seperately tested elsewhere, and is probably not supposed to affect any other calls anyway, so I don't find this to be a serious argument neither for nor against multiple-assertions. On the contrary - if one call alters the state of the object-under-test so as to alter the result of another call, you *should* make sure your tests invokes the calls in the proper order.

Great post. As I recently wrote (http://www.hedgate.net/articles/2006/11/17/one-assertion-per-test-should-come-natural), I have come to think that multiple asserts in a test method is often a smell of badly organized tests. If tests are organized by fixture (as in having a common setup) it is more natural to have a single assert per test method.

For the examples in James blog post there would be a fixture called ListOfBooksReturnedForAuthor1 (or similar). This fixture would then have six test methods, each with a single assert.

(Sorry if this was double posted, my first comment did not seem to show up).

Pingback from  Give Me An Assertion Vasily.  One Assertion Only, Please « An Experiment in Scotch

Pingback from  One Assert vs. Multiple Asserts « Clackwell’s Weblog

Pingback from  Give Me An Assertion Vasily.  One Assertion Only, Please « Mental Pandiculation