Here’s another feature Typemock Isolator has that no one else currently has: the ability to return fake objects from properties recursively (without the use of an auto mocking container for such a feat). this saved a lot of reperitive test code of creating stubs that return stubs that return some custom result.
For example, an object person has a manager which is also a person (which of course also has a manager etc..)
here’s how you can use this feature:
[Test,Isolated]
public void test()
{
Person person = Isolate.Fake.Instance<Person>(Members.ReturnRecursiveFakes);
Assert.IsNotNull(person.Manager.Manager.Manager.Manager);
}
This test will pass (Manager is null by default. look at the class in question below:
class Person
{
public string Name
{
get { return name; }
}
public Person Manager
{
get { return manager; }
}
private string name;
private Person manager;
}
Here is how you can return stub results from properties of the stubbed objects:
[Test,Isolated]
public void test()
{
Person person = Isolate.Fake.Instance<Person>(Members.ReturnRecursiveFakes);
Isolate.WhenCalled(() => person.Manager.Manager.Name)
.WillReturn("roy");
Assert.AreEqual("roy",person.Manager.Manager.Name);
}
Notice that:
-
By saying “ReturnRecursiveFakes” we are actually saying “I don’t care who uses this object and how from now on. It will return a fake when needed from all properties and the properties of the stubs as well.
-
We can still set stub results on the fakes on any level
If we wanted to only fake the Manager name and leave everything else intact we could just use a live object in the test like this:
[Test,Isolated]
public void test()
{
Person person = new Person();
Isolate.WhenCalled(() => person.Manager.Manager.Name)
.WillReturn("roy");
Assert.AreEqual("roy",person.Manager.Manager.Name);
}
In my search to understand a little more about BDD and BDD frameworks, one gem that I gleaned is that there is an important distinction between “Spec” and “Behavior” frameworks. “Spec” is intended for the unit level granularity (like unit tests) and “Behavior” is intended to relay user stories at the application level.
This makes sense to me because in all the BDD discussions I’ve seen I had a hard time understanding why you want to put an application level story on top of a unit level story. it just didn’t make sense.
So here’s what I get so far (correct me):
I can use Behavior Frameworks to create stories that use the same shared language that my customer would understand, at the application level use cases. Most frameworks that support this allow you to “code” this up which still has a lot of friction for most people to read so it’s a problem.
I can use Spec Frameworks to replace the language I use in my current unit tests so that instead of “testing” I am defining “behavior” of a component at the unit level. this is usually almost a purely syntactical change in nature.
Some Frameworks have APIs for both Specs and Behaviors (like RSpec). Some only have APIs for one of them (which ones?)
Update: A full list, with links to the tools, an editable wiki page and more, is now located on the book wiki site. Feel free to add your own tools in there.
Working on an appendix for my book, with a list of tools and frameworks you should care about. Tell me if I missed anything:
· Mock Frameworks
o Moq
o Rhino Mocks
o Typemock Isolator
o NMock
o NUnit.Mocks
· Test Frameworks
o MS Test
o NUnit
o MbUnit
o XUnit
o Gallio
· Dependency Injection and Ioc Containers
o StructureMap
o Microsoft Unity
o Castle Windsor
o Common IServiceLocator Framework
o Managed Extensibility Framework
o Spring.NET
o AutoFac
· Database testing
o Use Integration testing
o MbUnit,NUnit,XUnit Rollback attributes
o TransactionScope
· Web Testing
o Ivonna and Typemock
o Team System Web Test
o NUnitASP
o Watin
o Watir
o Selenium
· UI Testing
o NUnitForms
o Project White
o Team System UI Tests
o Typemock Isolator
· Threading Related Testing
o Microsoft Chess
o Typemock Racer
o Osherove.ThreadTester
· General Testing
o Pex
· Acceptance Testing
o Fit & Fitnesse
o Watin and Watir and selenium
Here is one feature that sets Typemock Isolator apart from all the other frameworks, that has nothing to do with legacy code: Live Objects.
This feature allows you to just “new” up an instance of an object, if you can, and then fake method results on it, and send it somewhere into production code:
[TestMethod, Isolated]
public void DoSomething_LoggerHasFullDisk_NothingHappens()
{
RealLogger logger = new RealLogger();
Isolate.WhenCalled(()=> logger.IsDiskFull).WillReturn(true);
new MyClass(logger).DoSomething();..
}
With any other framework you are forced to “mock” or “stub” the RealLogger class to begin using it. Removing this need makes for clearer more elegant tests. This feature is already available in Isolator 5.1 that was released this week.
The discussion on the future of unit testing for the masses has shifted from the standard “if they are too stupid to learn it, we don’t want them” to “TDD without good design will make really bad tests”. and this is a good thing. it’s a good thing because it secretly implies that if you could write Good Unit Tests, regardless of design, then teaching design as a separate thing should not pose a problem.
Discussion so far:
But first, let’s make one thing clear:
TDD is NOT Unit Testing.
My previous post was not “TDD decoupled from design==adoption”, it was “Unit testing decoupled from design == adoption
TDD is (technically) about when you write the unit test.
You can still write crappy tests in TDD mode. With good design. Unit Tests deserve a good book about them without TDD in it because the current books about TDD suffer from lack of SRP.
On the other hand you can still write good tests, without TDD. The design might make them a little longer, but if you learn to write unit tests well, this should not pose a problem.
Learn Good Unit Testing regardless of TDD and Design
people can start learning unit testing before learning TDD. they learn the silly stuff (the kind udi looks down upon) in an hour, and then they get to the hard stuff : making the tests readable, maintainable and trust worthy. these are separate properties that can be learned to be done regardless of design, since test naming convention, refactoring and correctness can be learned in separate.
The only reason it’s really hard to teach in separate today is because as those devs get to test real world code they have to start dealing with real world dependencies and that it the barrier that usually stops most people. having “intermediate” frameworks in place such as JMockit and Typemock Isolator can help in that regard, keeping people focused on the task at hand and deferring the need to learn other types of design for a time that is better. The idea is to let the developer choose when to learn which technique:
- Learn how to write good unit tests (book)
- Learn Good Design technique (give me a link to a good starter book!)
- Learn Test Driven Development (book)
I want a developer to grok each one of those things to a very deep level, but today’s “market” wants them to learn it all in one go. Since each one is its own little art form, are you really surprised to learn that most of the mass population is not there yet?
If people think that the only way to write good unit tests that are not worthless is by doing them TDD style with SOLID design, they are showing several things to me:
- they don’t get what “good” unit testing is
- They don’t “get” what value good tests have outside the context of TDD
- they guide a community who reads them down a path of “all or nothing”. which for most people begets “nothing”.
Field notes:
Udi writes:
“If logic is duplicated, if the code is tightly coupled, if there is no separation of concerns, the unit tests will be useless - even if they “test” the class in isolation.”
That is wrong in many levels. Let’s start with the fact that a unit test can have a specific value in testing a specific class in isolation, even if the bugs in that class are duplicated on other classes. If Udi does not agree on that, I don’t know what else to say about that. I wish that what he meant was that all these problems were repeating in the tests, since this is usually why people fail when they start out unit testing: they write crappy tests with bugs, code duplication and lack of isolation (this can all be learned). But he’s not. It feels like a “purist” view that does not reflect the real world. It must be nice living in an echo chamber!
Let’s see udi contradict himself in the same post:
“Managers shouldn’t necessarily sacrifice their projects on this altar of learning. Organizations need to find ways for developers to safely practice these techniques as a part of developing their “human resources”. First of all, this needs to be communicated to everyone - that the organization understands the importance of these techniques, the desires of developers to adopt them, and the projects that need to be delivered.”
“…It’s a gradual process.”
That’s the problem Udi. It’s not possible to do it gradually today, because people keep saying that you can’t do things in little chunks. you have to learn it all together. If we were to help developers learn what good unit testing means they could much more easily learn that stuff, and then learn the other stuff.
Casey writes:
“It is not a technical challenge to write a unit test, any half trained monkey could do it. A few attributes, a couple of classes, a few new() statements, and a few Assert statements. We don't need magic mocking frameworks, we don't need any mocking frameworks for 90% of our tests, we don't need application of 12 GoF principles before we can write a test, honestly.
Testing and writing automated tests is hard because it requires a strong understanding of the user stories that sit behind your code and project, because it requires you to think as a user and not as a developer, and because it requires you to be able to think of multiple conflicting axioms, test cases and class interactions, and how to cover the weaknesses of each individual test with other complementary tests.”
Casey preached about what he sees as the “end game” for unit testing, TDD and Design. where everything is in one great ball of goodness. Or did he?
Let’s separate these points into better chunks:
Design related stuff:
- to think of multiple… class interactions
Good Test related stuff:
- Understanding of the user stories
- Think as a user not a developer
- think of multiple conflicting axioms (..?) test cases..
- how to cover the weaknesses of each individual test with other complementation tests.
TDD related stuff
So, what I gather from casey’s words is that you want your tests to be :
- testing the right thing
- be readable and show intent
- have good coverage
- test or be aware of class interactions (the design related stuff)
except the last point which is where most of today’s mock frameworks force you to redesign, there is nothing here you can’t learn separated from TDD and SOLID techniques. the interaction stuff is where those other isolation frameworks I mentioned come into place.
We need to decouple Good Unit Testing from TDD and SOLID. And it seems that not only “noobs”, as a stupid idiot commented thoughtfully on my last post, can benefit from learning this distinction.
In my previous post I started talking in more coherent words about feelings I’ve had lurking in the past year. This post is an expansion of that, after having some great twitter conversations in the past few days with some great people. Treat this as a stream of consciousness.
before that, here’s a short highlight summary of responses to my post in other blogs:
there are also plenty of people who picked up the ball (some of them have already done that before I started mumbling on my blog) and continue to make better attempts at bringing knowledge to the masses, in more coherent, newbie friendly manner.
and the comments on that post are… interesting (some are very expected, though).
Let’s make another interesting statement, and this one is indeed product related.
To reach the masses, we need to decouple S.O.L.I.D Design Principles from Unit Testing and TDD
I’m not saying throw out design. I’m saying – it’s important enough to be learnt on its own, and not in the context of tests. in fact, in that context, it inhibits too many people to be practical as a learning tool. so we need to separate forces.
But why?
The wall to start unit testing is too high (for most devs) and will always be too high if you have to know your SOLID in order to even start writing your first test or refactor your code for testability. This problem only exists in static languages like Java, C# and VB.NET, where in order to replace something you need to have a way to do polymorphism on that class (interface, or inheritance with virtual methods).
The reason people love Test Driven design so much is because the technique, aided by most of today’s leading mock frameworks, squeezes out all the problematic static coupling that exists in the current design, before it will even let you test something. Can’t decouple that web service dependency from your class? you’re screwed. Redesign and only then you can test it. Have a static method that gets called? you’ll need to jump hoops to override the calls to that method, let alone the method itself. Feel the burn, baby!
TDD existed before mocking frameworks existed. what did people do then? SmallTalk was one of the first languages that had people doing TDD with it. SmallTalk is testable by default in that you can re-define anything to do anything else at runtime. People just thought about their design as they wrote the tests first, and they had no real constraint on what specific design to follow. It still holds today in Ruby, for example.
Now, the reason today’s “hardcore” Test Driven Design is even possible is because current mocking frameworks rely on static language (and compiler) to tell the developer what can and cannot be done in terms of replacing objects and implementations. That forces a specific design methodology that closely matches what SOLID principles stand for, and it’s great. you get great code out of it, and nice designs.
I repeat – I love the idea of SOLID design and I practice it.
So, why do I think that these things should be decoupled? At the end of the day it is a constraint that is put upon each developer that wants to start unit testing (not even talking about legacy code right now). That means that each developer that starts tests will hit a brick wall and will have to start reading up on all things SOLID, patterns, or just buy my book. There is no “easy” way. and that’s fine and dandy for most of the people reading this blog. And for the past four or five years, it’s been just as fine and dandy for me too.
BUT, The adoption rate for TDD and unit testing at all is low, and I suspect that it is indeed this learning curve that makes that possible. And it will continue to be low as long as doing great (not OK, great!) designs is a prerequisite to doing proper unit testing (or at all). The current mock frameworks (most) enable this notion and embrace it.
Today we live in our own little echo chamber where everyone is happy with their “pure” designs and consultant fees, because we are part of a very elite group: those you need to pay in order to learn the “hard stuff” to do something which many people want to do but just can’t. It’s almost like knowing Regular Expressions, except a much larger scope. We’re making it harder on purpose to start doing unit testing, because we believe that it’s someone’s best interest (and in a way, in our best interest) to learn SOLID first.
The people I know from the mocking community are amazing, passionate and smart people. They would never do anything they don’t fully believe in, and their value system is almost radically pure. And I love that. but we cannot play judge here. people learn in different ways, and the low adoption rate tells us that it’s not working as well as expected.
Why would decoupling work?
If we were able to decouple the requirement of knowing SOLID and just get people to start testing today and learn SOLID later or in parallel, adoption would be much higher. Why?
- People who get test infected soon want to learn SOLID anyway
- People who succeed doing one thing, will be more open to adopting other things from the same source of knowledge
- It’s less scary, and less time consuming to start doing things. We reduce the barrier to entry into the agile world.
- It allows developers in a non-agile environment to do little,incremental steps on the road to a better way of working, without needing to chow down the whole meal in one big bite.
- Sucky tests are better than no tests. They also lead to better tests and better understanding of design as people learn and get better at it.
What’s stopping us?
But with the current set of mock frameworks, you can’t do all those things. you technically are not allowed to do that. and that’s why solutions like Rhino, Moq, and NMock will not appeal to the masses, and why solutions that support other types of design decisions (JMockit, Typemock Isolator) will continue to gain popularity as people strive to become more agile under non-agile conditions. they provide that needed stepping stone to learn unit testing without needing to learn SOLID as a requirement. That’s the difference. But the community is actively depressing thoughts about using such tools today because of the belief that TDD and design have to go hand in hand at ALL times.
Many of them that will read this are scared that I’m even suggesting this, because they fear a return to darker ages where people didn’t have a clue, and there was nothing to stop them from screwing up the design. But as you’ll see later on, the plan is to get people on board the design stuff, just not as a prerequisite to the unit testing stuff. that’s all.
Of course, the big problem here is that I actually work at Typemock, and thus, many people feel that every word that comes out of my mouth is a sales pitch. but the reality is that if Typemock had been an open source project, those people would have read the words that I write, and not try to find a hidden meaning in them. and the truth is I’d write something like this even if I didn’t work there today. I’d quit to say these words with more credibility but I want to have an impact on how those products work and feel, so I’m not going to do that, sorry.
What does the industry need to do?
-
Direct people who are reluctant to go through that big learning curve or have a problem with the current frameworks to tools that mitigate those fears (Typemock, JMockit). One is open source, the other has an open source free license. so they are both achievable to use and play with.
-
When doing so, learning better design techniques should be on the horizon. but it’s important not to discourage use of these frameworks on existing non SOLID design notions. people have a hard time with change as it is. They need to learn one step at a time and we have to let them have the time to grok things separately if they want to.
-
Decouple the task of teaching unit testing from the task of teaching design. They are both important but are not a requirement of each other. We need more Design material out there with a simple way to reach and find it (we’re on the way today). Unit testing information is already out there and more is going out every day.
-
Realize that tools that enable different designs are not a problem, but a way to get a foot in the door with developers who wouldn’t walk through that agile door in other ways. it’s a bigger net to catch more fish. These tools allow people to have a bridge to the agile world with less pain involved.
This is something that we need to continue talking about, since it can change the future for most devs today. If we shy away from talking about this, and stay in our cozy little submarine underwater, we will sail right beneath all those people lying on the beach, wondering what’s up with all those people in the water, or afraid to take a swimming lesson. (yeah, the world’s worst analogy)
Thoughts? Comments?
About 4 years ago I was consulting for one of the largest companies in Israel (about 15,000+ employees) and helping them grok TDD and automated unit testing. I had spent there over a year and a half, on and off, and was teaching both .net, C++ and java developers about the techniques and tools they can use.
I was consulting for C++, Java and .NET developers.
The C++ devs were first.
They had plenty of objections to what I was teaching them, because the amount of code refactoring they needed to do in order to make their code testable was almost unrealistic for them. Even writing new code meant writing something on top of old-untestable code, and they had a hard time adjusting to the more interface (pure virtual) based, contract driven TDD approach for new code. C++ devs has a hard time becuase it takes more code to write good testable code, and the coding tools they used (VS on windows and VI and such on UNIX based systems) didnt’ provide a good productive way to make code “bend” easily as they refactored it or created it in the first place. every little change meant a big “search and replace”. every method refactoring meant adding it the declaration in an .h file and thus they tried to have a little to do with changing code as possible.
Looking back, today they still don’t have the right infrastructure in place or the right tooling or the willingness to do unit testing or TDD becuase it’s too hard with the current tooling to accomplish it.
I had also spent some time with the .NET devs
and thought them the same things. Becuase tooling support was better it was a bit easier for them to accept the practices, but still people had a lot of trouble grasping concepts such as dependency injection, inversion of control and designing for testability in general. I found that most people didn’t really have a big problem with the idea of exposing things they didn’t use to expose before, it was more about the concept of learning how to refactor the code in interesting OO ways to make it testable. Making a class non sealed, a method virtual, and then inheriting from that class and testing against the inherited class (“extract and override”) was very much unintuitive for them, and other concepts like containers (spring at the time) didn’t help them get into the groove either. For them it was like drinking water from a fire hose.
Consulting for the java devs
Java has lots going for it in terms of unit testing, automation and TDD
- Java devs have a much longer “heritage” of tooling and unit testing than in the .NET world, and so you’d expect them to be more inclined to do these things.
- JUnit is part of any decent java idea so the barrier for entry should be lower, right?
- Refactoring was also in most decent java IDE's, so you’d expect them to grok that too. but that wasn’t the case.
- class methods in java are virtual by default so you can override what you want as long as the class is not sealed (final)== more testable by default.
- Java required a much higher OO learning curve to start with. Good OO knowledge is fundamental for testable designs.
Unfortunately, most developers I met didn’t even know how to use the refactoring tools they had available and had a hard time with the same dependecy injection principles that the C++ and .NET devs had. It was just too cumbersome to make the code testable, but eventually the infa team at the company had made the decision to take a couple of weeks and make all the infra stuff replaceable for testing purposes. they did not regret that decision.
So the java group had the most success with testable designs, but they still had a hard time doing it. in fact, only the infra team at that group took the plunge. the other (higher level) devs in the java group didn’t grok it. and don’t use it today.
Why was this adoption been such a failure? They were willing to learn, but they didn’t like what they learned. It didn’t fit their needs in the real world. Or the way they worked, or the way they knew how to do things.
Think what you will ,but 99% of the dev worlds aren’t ready for the “testability” message. and that is a fact that is hard to realize for most of us, which just can’t grasp why isn’t that any sensible human being isn’t willing to do everything much better than they are doing today.
Spending the past week running around the Microsoft Campus in Redmond had made several things clearer for me. I’ve been feeling them for a while, but this week these abstract feelings materialized into something that is more coherent that I can talk about.
Why do most developers not write unit tests, still?
Reason #1: Learning curve
One of the main reasons most developers still today don’t really take to unit testing (and TDD) is the really high learning curve that is forced upon them. That is not to say that learning about
mocks, stubs, dependency injection, IoC containers, Extract & override technique, Record\Replay, AAA and more
-- is not useful, but it is a big obstacle to get people into the habit of doing something which they know is good for them – test their code, verify their assumptions, automate, integrate, be confident, get feedback, define behavior – all those and many other benefits are being thrown away by developers all over the world because the entry fee to this world is too high, for all the wrong reasons.
It’s time to simplify. Get back to basics. Take a good hard look at the huge spiral that we are leading people into and asking whether removing some things today can lead to those things being accepted and being easier tomorrow. or maybe they weren’t needed at all.
Say Goodbye to “mock” and “stub”
“mock” and “stub” have played their part in us trying to explain to ourselves what it is we are trying to do, to start learning the minute and important differences in the patterns of our test code. They helped us define what it is we are looking to do, in terms that, for lack of anything better, are now the de-facto pattern names for something that should never have been named in the first place for the casual programmer.
That’s why in the new version of Isolator we’ve chosen to completely get rid of the words mock and stub, and get back into “fake”ing. You can fake a method or a class instance, and you can verify that something was called. You should not care whether this is a stub or a mock.
We need to clean our language: mock, stub, fake, test double: They all mean something in the context of us trying to identify patterns in our test code. But as users mature and start using the frameworks, those same words increase the entry barrier, and effectively block people out. If I’m showing a simple test that fakes some dependency that a class uses to a new dev on my team, I should not have to send them over to read fowler’s “mocks arn’t stubs” article (which is enlightening, but should not be relevant to them) because it misses the point.
Less interaction testing, more state based testing
Most crappy unit tests I’ve seen were over specified in that they asserted on internal interaction with an object way too heavily, which made the tests very fragile. Getting back to basics means making it easy to enable state based testing (assert style checks) and only enabling interaction testing where it makes sense: (when your app does one-way messaging to an external component). verifying that something was called should be the exception to the unit testing rule. The AAA style APis make that a more clear distinction.
Record-replay APIs make it too easy to make interaction based tests when you really want a state based one (getting a result vs. checking that something was called)
Use AAA style Apis
As an industry we are learning and changing all the time. The “Arrange-act-assert” pattern in the field of mock frameworks represents us trying to return to basics. Moq did that nicely by showing a way that leads there, and it is being taken gladly by Rhino and Typemock Isolator. It is taking off because we are starting to look beyond our own little echo chamber and seeing what is is that all those other 99% of devs are expecting as guidance, and in what form.
AAA is guidance because there was an outcry (a silent, vote of feet) that record-replay is not cutting it – it’s too confusing for non-experts. It’s too technical. and it does not lend itself to the way people want to work. So most people didn’t use it.
Visual Basic Language Support
Everyone screwed up on this one.most VB developers who will try to use one of the current frameworks will not succeed since they required anonymous methods or lambda expressions. Some frameworks use VB specific keywords which makes them VERY awkward to use. If we want broad acceptance of our practices we need to make sure whoever wants to can use the tooling we make. VB is huge and its amazing how we all pretend it does not exist. But I think it’s the key to getting to those people who want to be better but all they see is a high wall instead of an entrance.
Use “Isolate” and “fake” instead of “mock”
Don’t say ‘mock’ frameworks, say “isolation” frameworks. Because ‘Mock’ is an overloaded word it is confusing. Explaining to a developer that you can “isolate” something from its dependencies rather than “mock” its dependencies feels more natural.
Simplify,Simplify,Simplify
Remove APIs instead of adding to them. Have single point of entries to your frameworks because discoverability is the key to good framework use. That’s why extension methods are problematic, they mean multiple points of entry.
Be open: Realize that other tools and techniques can act as bridges to the agile world
Design for testability is just one way to get where you want to go. it is not the only way. Sometimes it’s impossible. Discouraging the use of other ways of working (using tools like Typemock Isolator or JMockit) will only make the current echo chamber more closed and feel more un-achievable for those who need to get to the same point with a very different set of constraints than what most of the “loud” voices need to face.
By “allowing” other forms of working you enable those who want to, to get a bridge to help them instead of walking by foot the whole way. Any other way would mean that we will all fail in delivering what we believe in to those who have no idea what we’re talking about.
Here’s what you can find on the latest typemock racer build. I think it’s pretty cool.
This feature was just added to the Racer Alpha.
Given the following method:
_thumb.jpg "WindowClipping (14)")
Can you see a potential for deadlock in this method? Here’s the answer.
I wonder if it will find this one as well..
After finishing a wonderful week and a half in Norway, I can’t wait to get back.
Thanks for the great memories!
I’ll be coming back in December for another TDD Master Class Course.
More Posts
« Previous page -
Next page »