in continuation to my challenge (which no one had bothered answering, lazy web!)
One of the things that inhibits unit tests in organizations is the idea of fragile tests. a Fragile test is a test that can easily break when the production code changes. That’s not to say that tests should never break as you change production code, but there are ways to minimize this effect so that when tests do break, they break for the right reason.
the right reason is when a feature in production code isn’t working.
the wrong reasons can be many and varied. Of the most common ones is the idea of over specification in tests. The more your test expects of your code’s internal implementation, the more “specified” it is, the more “expectations” it has on the internals of the code. since internal code is more prone to change, so will the test break more often. so the situation can occur that the production code changed, but still gets the job done, but since it’s doing it differently internally, the test breaks even though the end result is still OK.
One of the main ideas in the new Isolator API was to reduce the fragility of tests by default. there are several features in Typemock Isolator that remove lots of fragility from tests by default, compared to other tools:
- All Fakes are non strict by default.
- Recursive Fakes
- Method stubs ignore parameters by default
- “From now on” Method behavior as a default
- Isolate.Swap.AllInstances
- Isolated.NonPublic APIs will not break when refactoring from private to public
- Ignore one or all static methods on a type
- True Properties
- All Method overloads are faked by default
All Fakes are non strict by default. when a fake object is strict, then it will throw an exception when someone calls one of its methods that wasn’t specifically “expected” by the test (or a different number of times than expected). since internal code can change and start interacting with other code in different ways, strict fakes will almost always fail your test on the lightest nuance of change. Most frameworks used to have strict fakes by default.
Isolator has all fakes non strict by default. in fact, there is no “strict” mode at all.
var fake = Isolate.Fake.Instance<SomeType>();
//code under test can now call any method without the test needing to specify it. the test only needs to specify behavior of methods it cares about.
Recursive Fakes This is a feature first introduced by Isolator more than a couple of years ago and then adopted by Moq and Rhino Mocks. the idea is simple : allow chained calls in your code without the test needing to specify fakes at each hierarchy level. for example, calling MyFake.SomeProperty.DoSomething() will, by default, do nothing since its root is a fake object. the “SomeProperty” will be initialized by default to a fake object as well, recursively down the line.
With other frameworks this will only work with interfaces, methods which are virtual and classes that are non sealed. Isolator can make this work with anything, including static methods and non sealed classes.
var fake = Isolate.Fake.Instance<SomeType>();
//code under test can
fake.SomeProperty.DoSomething()
//or
var propObj = fake.SomeProperty;
propObj.DoSomething();
As your internal code is refactored, or changed to call a method down the hierarchy, this should not break a test (as long as the method does not matter to the test)
Method stubs ignore parameters by default all other frameworks except Isolator will use the values being sent to methods of fake objects during the “setup” phase, as expectations of these actual values. Isolator by default ignores values being sent in, and just returns the fake result we tell it to, no matter what was being sent in. Of course, you *could* tell it to use the exact argument values, but that is not the default behavior.
other frameworks will require that you put in special constraints on each parameter value to say that you don’t care about the actual value.
var o = Isolate.Fake.Instance<SomeType>();
const int IGNORED_VALUE=10;
object IGNORED_OBJECT= null;
Isolate.WhenCalled(()=> o.SomeMethodWithParams(“ignored string”,IGNORED_VALUE,IGNORED_OBJECT) ).WillReturn(3);
this method will return 3 no matter what parameters are being sent to it. so internal code can change parameters slightly and it will still work.
by specifying only on specify param values, we could be over specifying the test.
“From now on” Method behavior as a default with Isolator, once you set behavior on a method (“Ignore”, WillReturn(..), WillThrowException(..) ), it will use that as its default behavior for all subsequent calls. That way, if the internal code ends up calling a method more than once, the test should not care about it (as long as the result does not hurt the test). This is great for ignore calls to “Log” objects and such, where you don’t know or care how many times it will be called. you just want to ignore them without having to worry if the code will use them or not.
for example, the method from the code above will *always* return 3, no matter how many times it will be called.
if we specified it twice with different return values, it will return the *last* behavior as the default, after it was called once with the previous behavior.
Isolate.Swap.AllInstances This beauty will replace all existing and future instances of a specific type in the code under test with the fake type. that way you don’t care how many times some object might be created, or how many instances of it are used. for “util” type objects, this makes things a breeze to setup for a test that could be very complicated or impossible in other frameworks.
as the internal code changes with usage of these types, the test will still pass.
var fake = Isolate.Fake.Instance<SomeType>();
Isolate.WhenCalled( ()=> fake.Foo() ).IgnoreCall();
Isolate.Swap.AllInstances<SomeType>().With(fake);
//now production code can do this:
var x = new SomeType();//<—this will behave like the fake object we created
Isolated.NonPublic APIs will not break when refactoring from private to public Isolator allows setting method behavior on non public methods as well:
var x = new SomeObject();
Isolate.NonPublic.WhenCalled(x,”SomeMethod”).WillReturn(3);
The beauty is that if you do refactor the method to make it public, the test won’t nudge you or throw an exception. It will just work with public methods as well. You can then refactor the test without being a slave to it.
Ignore one or all static methods on a type Isolate.Fake.StaticMethods<SomeType>() will ignore all void methods on that type, and return recursive fakes on all functions. That way, if you add more methods to that type, the test won’t care about it. great for utility style classes with lots of static methods.
True Properties “True properties” is a feature that Isolator picked up from Rhino Mocks – you can set properties on a fake object as if they were real properties.
The nice thing is that if internal code then sets these properties to values you don’t care about, the test will work just fine. it’s also very readable.
var fake = Isolate.Fake.Instance<SomeType>();
fake.SomeProperty=3; //will behave like a real property and return 3;
Default Faked method overloads Once you set a method’s behavior, if that method has any overloads, they will have the same behavior as well. So if you refactor your code under test to use a different overload of the same method the test will “just work” and won’t need to be changed to reflect this change.
One of the things that inhibits unit tests in organizations is the idea of fragile tests. a Fragile test is a test that can easily break when the production code changes. That’s not to say that tests should never break as you change production code, but there are ways to minimize this effect so that when tests do break, they break for the right reason.
the right reason is when a feature in production code isn’t working.
the wrong reasons can be many and varied. Of the most common ones is the idea of over specification in tests. The more your test expects of your code’s internal implementation, the more “specified” it is, the more “expectations” it has on the internals of the code. since internal code is more proone to change, so will the test break more often. so the situation can occur that the production code changed, but still gets the job done, but since it’s doing it differently internally, the test breaks even though the end result is still OK.
One of the main ideas in the new Isolator API was to reduce the fragility of tests by default. there are several features in Typemock Isolator that remove lots of fragility from tests by default, compared to other tools.
What do you think they are? (look here for hints) I found at least 8 separate things.
what do other tools like Rhino Mocks, or Moq do to help or hinder the fragility of your tests? *here is a search that will help*
I’ll post my own answers tomorrow. but I’d like to see if people actually “get” how important the API of an isolation framework is to this cause.
I’m happy to announce that I’ll be doing a 5 day TDD master class in the UK, September 21-25. Full course details are here.
If you read my blog or seen my twitter, you also have the chance for an early bird pricing.
but, if you use the following VIP code [VIPTDD09], you get an additional discount over that as well.
IF you want to get a glimpse of what are the kind of topics we cover in the first two days alone, you can listen in to the latest Hanselminutes episode about the art of unit testing, with me.
Seats are limited to 20. you’ll be pair programming most of the time, and I want to make sure people get as much personal attention and guidance from me as possible.
I think I heard Jerry Seinfeld say that the life of an actor are:
Who’s Jerry Seinfeld? Get me Jerry Seinfeld Get me a young Jerry Seinfeld Who’s Jerry Seinfeld?
Which really feels these days like:
- What’s “CMM”
- Let’s do “CMM”
- Let’s do an adapted version of “CMM”
- Here’s a better,lighter “CMM”
- What’s “CMM”?
- What’s “Agile”
- Let’s do “Agile”
- Let’s do an adapted version of “Agile”
- Here’s a better, lighter “Agile”
- What’s “Agile”?
- What’s “Scrum”
- Let’s do “Scrum”
- Let’s do an adapted version of “Scrum”
- Here’s a better, lighter “Scrum”
- What’s “Scrum”?
- What’s “Lean”
- Let’s do “Lean”
- Let’s do an adapted version of “Lean”
- Here’s a better, lighter “Lean”
- What’s “Lean”?
I’m half way through JQuery in Action and have to say I was pleasantly surprised to read a lot of things about the “why'” things are the way they are, and not just simple facts about JQuery.
Lots of useful information, especially for a web newbie like me (I used to hate html related activities until $() came along)
Way to go, great book!
My upcoming TDD Master class in Oslo, Norway, during august, has only a few seats left. I heard from several people at NDC that they are planning to sign up, so I’d suggest they hurry up.
This class is a 5 day pair programming fest, where people work together on Unit Testing and TDD related excercises with my coaching.
It’s the class I wish I had done 10 years ago.
- If you’re interested in me doing this class in other countries, drop me a line.
- If you’ve already been to my class, I’d love it if you left your thoughts, comments or recommendations to it on the comments to this post. I will later use them on other places that talk about this class (with your name next to it!)
here are the questions that teams and team leads should be asking themselves on a daily\weekly basis.
There are more, but these are the basics, to me. It’s part of the summary for the talk “Beautiful teams I am giving at SEConf and NDC. we do a lot of this stuff over at work, and it’s proving itself on a daily basis.
Whole team
- What can we automate?
- where are we "Reinventing the wheel"?
- what are the tools that slow us down?
- what tools can we use better?
- are there bugs that I could have found earlier? how do I make sure I find them earlier?
- when do we find out we built the right thing?
- when do we find out our code\design sucks? how can we make that earlier?
- How do we show progress at the team level? at the management level?
- How many meetings does each dev have every week? how can we remove them?
- Are we building by feature or by layer?
- can we make all our team sit in the same place?
Team Lead
- daily: what bottlenecks exist in the team? what have I solved?
- will my devs be better in a month or two than they were before? if not, how do I make that happen?
- what prevents my devs from working? what am I doing about this?