ISerializable - Roy Osherove's Blog

"Isolation" seems like a good idea.  Did you think about just "Double"? Still too overused, or too specific?

Interesting notes, Roy. Lots of good lessons learned in there.

This one in particular, though, sounds a little dubious... like marketing material that is only telling half the story:

"Isolator was the only one where people didn’t ask me a single question about what API to use. they did put calls in the wrong place in the test though (AAA is hard to get used to)"

What order are you teaching the frameworks in? If you're teaching Isolator last, then you are stacking the deck in favor of the Isolator syntax because everyone will already be familiar with other mocking framework syntax and APIs. Since all mocking framework APIs are well within the same realm of language, it's relatively easy to switch between them and understand what's going on.

if you are teaching Isolator last, then that claim in irresponsible at best. You would need to change up the order in which you teach the frameworks, across many different groups of students, to get an accurate picture of which syntax is picked up easier.

Thanks for sharing! Very interresting.

> i think Isolator is the only one that will re throw swallowed exceptions).

Rhino Mocks will rethrow exceptions as well

Excellent notes, Roy. This should help folks designing unit testing and mocking frameworks.

Few things I wanted to comment on. You said,

The word “Expect” confuses because it makes you think about “expectations” as in interaction testing, when in fact it can be used for stubbing out things.

After using RhinoMocks for about a year and a half now, I tend to agree. There is a call in Rhino that makes a distiction between stubbing out things and expecting a call:

// Stubbing out a value:

SetupResult.For(foo.Bar).Return(42);

// Setting up an exptectation:

Expect.Cal(() => baz.Shabam());

We tend to use this distinction wherever we can. Helps clarify intent of the unit test.

Also, you mentioned that unit test names really help flesh-out the spec. I now name all my unit tests something like:

[Test]

public void ClickingFobnicatorLaunchesNewBuzzTap() { ... }

I've also found this to be useful for maintenance purposes -- if you break some code, your unit test will fail -- you can hop right to the unit test and find that the code you broke is the launching of new buzz taps when clicking a fobnicator.

One last thing: I have tried to move to AAA syntax. Honestly, I find record/replay cleaner and more structured!

Pingback from  Reflective Perspective - Chris Alcock  » The Morning Brew #237

Very interesting notes, and thank's for sharing. I have to tell you that you're one of the VERY few that has not overlooked VB.Net for using mocking frameworks. Ofcourse the next version of VB is going to support Lambda-statements as well as Lambda-expressions so that will help a lot. I really like what you were doing for the VB-friendly API for Typemock, it looked really cool.

Pingback from  Dew Drop - December 4, 2008 | Alvin Ashcraft's Morning Dew

Quite interesting to read the reactions of the course participants, I do recognize many of the reactions from my own experiences. I was just interested as to why you or the participants feel that TestDriven.NET is soo much better than resharper as a unit test runner?

Only TypeMock can be considered an "isolator framework".  The other frameworks are just mock frameworks.  The isolation is coming out of explicit design considerations.  The source of isolation is design, not tooling.

You should drop NMock from your course Roy, se muerte:

blog.robbowley.net/.../rip-nmock

>> You should drop NMock from your course Roy, se muerte:

NMock2 is NOT dead (see sourceforge.net/.../nmock2)

>> Only TypeMock can be considered an "isolator framework"

I agree with that, TypeMock has a completely different approach to "isolation" than all other mocking frameworks.

>> The verification error messages that all three big frameworks (rhino, typemock and mock) throw could be much clearer (expected X but only got Y)

If you tell us what is unclear, we will see to clear things up (honestly, we did a lot this year to improve this topic)

>> The AAA (Arrange act assert) model for isolation framework, while much shorter, is still not as easy to grasp as you might think for newcomers. In fact, I’m not sure people found it easier than record-replay.

I agree, with NMock2 it is not easy to understand that how to setup a unit test: expectations can be scattered through unit test.

>> The word “Expect” confuses because it makes you think about “expectations” as in interaction testing, when in fact it can be used for stubbing out things.

Although there exists the Stub entry point in NMock2, it is not really stubbing (it's equivalent to Expect.AtLeast(0)). A clearer distinction between expectations (things that have to happen) and stubs (things needed that the operation can succeed) would be better.

However, we made the experience that using Stubs can result in unused Stub declarations. Therefore, we normally use the Expect.On(mock) syntax, which is a shortcut to Expect.AtLeast(1).

>> Verification Exceptions can be too easily “swallowed” by inner code so the test seems to pass (false positive). i think Isolator is the only one that will re throw swallowed exceptions).

I'm not sure I understand what you mean. If the tested code swallows exceptions then the test should detect that anyway.

>> People really likes the fluent API design of NMock2, but disliked its strictness

That's the main reason we use NMock2. Can you please be more specific about "dislike its strictness".

Cheers,

Urs

Thanks for the clarification.

Currently there is no direct way for nonstrict mocks. Only way would be to define a Stub for all Methods and all Properties and so on: Stub.On(mock).Method(new AlwaysMatcher()); But it should be an easy extension to allow stubbing of whole interfaces.

Exception swallowing: we could rethrow a swallowed exception in the VerifyAllExpectationsHaveBeenMet method.

We'll think about that.

And thanks again for the very useful feedback!

>> The verification error messages that all three big frameworks (rhino, typemock and mock) throw could be much clearer (expected X but only got Y)

I found that to be more the case with Rhino Mocks than with NMock2. NMock2 tends to have much more understandable error messages than Rhinos. I have written about the contrast in error messages between those two mocking frameworks:(lexicalclosures.blogspot.com/.../cryptic-rhino-mock-exception-messages.html)

>> "...The verification error messages that all three big frameworks (rhino, typemock and mock) throw could be much clearer (expected X but only got Y)..."

I have experienced this more with Rhino Mocks rather than with NMock2. NMock2 exception messages tend to be much more understandable than Rhinos. I have written about the contract in messaging for these two testing frameworks:

lexicalclosures.blogspot.com/.../cryptic-rhino-mock-exception-messages.html

Just wanted to stop by to say that the next version of NMock2 will

- rethrow swallowed exceptions in the Verify.. method

- be able to mock classes (virtual and abstract methods)

- stubs are supported (no need for expectations for simple stubbing)

Happy mocking

Urs

Hi Roy,

just wanted to point a couple inaccuracies:

> # One of the hardest points was remebering the APIs in rhino and Moq, becuase there is more than one entry point to using the API (Constraints alone in rhino have 4 different “root” classes)

Moq has ONE entry point: Mock<T>.Expect. Constraints also have just one, It. Did people give any specifics on what could possibly be simpler?

> # The word “Expect” confuses because it makes you think about “expectations” as in interaction testing, when in fact it can be used for stubbing out things.

Ha! That would only confuse someone who knows the difference between interaction testing and stubbing things, which in my experience is a minority by far.

> Verification Exceptions can be too easily “swallowed” by inner code so the test seems to pass (false positive).

Would you mind giving me a repro of this with Moq?

> Given an interface IFoo with a boolean property,if people want to make the property return a false value they instinctively write “myFake.Foo=false”. All frameworks should support default “property like behavior” to make this possible. Rhino mocks has this ability.

Moq does too. I agree that this should be turned on by default, and it will in Moq v3.

> # the Moq syntax for constraints is just too complicated (It.IsAny<string> is OK, but It.Is<string>(s=>s.contains”a”) is just too much code noise and complexity

Moq also offers a very straightforward way to define your own constraints without creating new classes, etc. It's explained in the Advanced Features in the quickstart (http://moq.me/wiki/QuickStart) and Brian's blog (weblogs.manas.com.ar/.../improoved-argument-matchers-in-moq). It basically allows you to have your own matchers like:

// custom matchers

mock.Expect(foo => foo.Submit(IsTooLarge())).Throws<ArgumentException>();

where the definition is simply the following pair of methods:

[Matcher]

public string IsTooLarge() { return null; }

public bool IsTooLarge(string arg)

{

 return !String.IsNullOrEmpty(arg) && arg.Length > 100;

}

> # If you use moth Moq and rhino in the same test class, you will see all the extension methods mixed up when you open intellisense on any object. *shiver*

Moq does NOT provide any extension methods on System.Object. That's why you set expectations on the mock and use mock.Object to retrieve the instance. We did not want to polute users' intellisense experience.

> # AssertWasNotCalled feels like over-specification.

Cool! Was just about to add that to Moq :). Less code!