“NUnit is being problematic again”, they told me when I came to visit the project. “When running unattended it’s not catching assertions properly and the test is coming up green, but when stepping through in the debugger, it works fine.”. It’s nice, when getting a passing test is acknowledged as a bad thing, at least when you don’t expect it to be. In this case, though, the fault wasn’t really with NUnit.

  1: [Test]
  2: [ExpectedException]
  3: public void DoTheTest()
  4: {
  5:     _myComponent.RunMethod();
  6:     Assert.IsFalse(_myComponent.EverythingIsFine);
  7: }

“It’s simple. Either the method throws an exception, or at the very least – the EverythingIsFine property won’t be set to “True”, so the assert will catch the problem. But in their case, no exception was thrown and Everything wasn’t Fine,  but the Assert call wasn’t raising a red flag – unless they stepped through, in which case it did. What’s going on?

The basic problem is that to many developers, NUnit is a kind of magic. You write a self-contained little bit of code, the [Test] method, but you don’t call it yourself, you don’t get a feel for the whole execution flow. The result – developers don’t exercise the same sort of judgement they do on their own application code.

The root of the problem here is that the [ExpectedException] attribute told NUnit to pass the test if an exception is thrown. NUnit’s Assertion utilities, however, use exceptions as the mechanism for failing tests – when an assertion is hit, it raises an exception – it can be an AssertionException. For various mock frameworks, it can be an ExpectationException. It doesn’t matter – it’s these exceptions that make the test fail, and not some behind-the-scenes magic. Because the test had an open-ended [ExpectedException] attribute, these exceptions were caught, fulfilling the condition, and NUnit was happy.

What can we do to avoid this?

• Be explicit. Don’t try to catch ALL exceptions with [ExpectedException]. If you’re expecting an exception, you’re probably expecting a specific exception. Specify it.
• Be aware of how your tools work. If NUnit works by throwing an exception, don’t wrap it with a try/catch. Your tests are C# code too, as is the plumbing to enable it. It plays by the same rules.

No Comments