ISerializable - Roy Osherove's Blog : Agile Related

Art of unit Testing goes to print in April

RoyOsherove — Mon, 16 Mar 2009 23:33:45 GMT

The book of never ending production is now actually near the end.

the projected print date of “Art Of Unit Testing” is now April 30 and I’m happy to see the Amazon page for the book already has one review of the early access version (though he totally destroys my spelling and book formatting abilities, he likes the content, so I’m happy)

What Isolation\Mocking and unit testing framework are you using?

RoyOsherove — Mon, 09 Mar 2009 20:11:06 GMT

It’s time to re-examine where things stand in our little niche.

This should be interesting, if enough people answer – so please answer (you need flash).

Unit Testing in VB.NET – with Typemock Isolator (with a free license offer)

RoyOsherove — Tue, 13 Jan 2009 18:24:18 GMT

A move I’ve been pushing for a while now has finally come through today – Helping to bring unit testing into VB.NET. This is something I strongly believe in: VB has been kind of left behind with the range of latest alt.net related frameworks – mostly because they are C# specific. That means that some of them if not most, that use mambda expressions or anonymous delegates, are less approachable to the larger VB community (VB is missing some key functionality in that space, but even if it did have that functionality, I think the new API we are bringing is easier to grok).

We're announcing today a new version of Typemock Isolator (5.2). One of the coolest features in the new version is the inclusion of a new VB .NET unit testing API. This 'VB friendly' API was designed and written in VB.NET, for VB.NET developers. It uses the strengths of the VB.NET language to allow easy and intuitive use of the isolation API, for easier unit tests on any piece of .NET code.
To launch this enhanced VB support we are offering you a FREE full Typemock Isolator 5.2 personal license (worth $139 !!!).

How do I get a free license?

Post the text below on your blog (as is, with links) and tell us about it . If you are in the first 25 to mail us after posting, your license will be on its way!

If you have a VB.NET dedicated blog, you'll get a license automatically (even if more than 25 submit) during the first week of this announcement.

A few simple requirements:
(1) You must own a website, blog space or internet column, older than 2 months, or get permission from the owner of such a site.

(2) Your post must include the text below (including the links) between *Begin* and *End*.

*Begin*

Programming Visual Basic applications?

Typemock have released a new version of their unit testing tool, Typemock Isolator 5.2.
This version includes a new friendly VB.NET API which makes Isolator the best Isolation tool for unit testing A Visual Basic (VB) .NET application.

Isolator now allows unit testing in VB or C# for many ‘hard to test’ technologies such as SharePoint, ASP.NET MVC, partial support for Silverlight, WPF, LINQ, WF, Entity Framework, WCF unit testing and more.

Note that the first 25 bloggers who blog this text in their blog and tell us about it, will get a Free Full Isolator license (worth $139). If you post this in a VB.NET dedicated blog, you'll get a license automatically (even if more than 25 submit) during the first week of this announcement.

Go ahead, click the following link for more information on how to get your free license.

*End*

Unit Testing decoupled from TDD as well== Adoption

RoyOsherove — Wed, 01 Oct 2008 13:27:22 GMT

The discussion on the future of unit testing for the masses has shifted from the standard “if they are too stupid to learn it, we don’t want them” to “TDD without good design will make really bad tests”. and this is a good thing. it’s a good thing because it secretly implies that if you could write Good Unit Tests, regardless of design, then teaching design as a separate thing should not pose a problem.

Discussion so far:

But first, let’s make one thing clear:

TDD is NOT Unit Testing.

My previous post was not “TDD decoupled from design==adoption”, it was “Unit testing decoupled from design == adoption

TDD is (technically) about when you write the unit test.

You can still write crappy tests in TDD mode. With good design. Unit Tests deserve a good book about them without TDD in it because the current books about TDD suffer from lack of SRP.

On the other hand you can still write good tests, without TDD. The design might make them a little longer, but if you learn to write unit tests well, this should not pose a problem.

Learn Good Unit Testing regardless of TDD and Design

people can start learning unit testing before learning TDD. they learn the silly stuff (the kind udi looks down upon) in an hour, and then they get to the hard stuff : making the tests readable, maintainable and trust worthy. these are separate properties that can be learned to be done regardless of design, since test naming convention, refactoring and correctness can be learned in separate.

The only reason it’s really hard to teach in separate today is because as those devs get to test real world code they have to start dealing with real world dependencies and that it the barrier that usually stops most people. having “intermediate” frameworks in place such as JMockit and Typemock Isolator can help in that regard, keeping people focused on the task at hand and deferring the need to learn other types of design for a time that is better. The idea is to let the developer choose when to learn which technique:

Learn how to write good unit tests (book)
Learn Good Design technique (give me a link to a good starter book!)
Learn Test Driven Development (book)

I want a developer to grok each one of those things to a very deep level, but today’s “market” wants them to learn it all in one go. Since each one is its own little art form, are you really surprised to learn that most of the mass population is not there yet?

If people think that the only way to write good unit tests that are not worthless is by doing them TDD style with SOLID design, they are showing several things to me:

they don’t get what “good” unit testing is
They don’t “get” what value good tests have outside the context of TDD
they guide a community who reads them down a path of “all or nothing”. which for most people begets “nothing”.

Field notes:

Udi writes:

“If logic is duplicated, if the code is tightly coupled, if there is no separation of concerns, the unit tests will be useless - even if they “test” the class in isolation.”

That is wrong in many levels. Let’s start with the fact that a unit test can have a specific value in testing a specific class in isolation, even if the bugs in that class are duplicated on other classes. If Udi does not agree on that, I don’t know what else to say about that. I wish that what he meant was that all these problems were repeating in the tests, since this is usually why people fail when they start out unit testing: they write crappy tests with bugs, code duplication and lack of isolation (this can all be learned). But he’s not. It feels like a “purist” view that does not reflect the real world. It must be nice living in an echo chamber!

Let’s see udi contradict himself in the same post:

“Managers shouldn’t necessarily sacrifice their projects on this altar of learning. Organizations need to find ways for developers to safely practice these techniques as a part of developing their “human resources”. First of all, this needs to be communicated to everyone - that the organization understands the importance of these techniques, the desires of developers to adopt them, and the projects that need to be delivered.”

“…It’s a gradual process.”

That’s the problem Udi. It’s not possible to do it gradually today, because people keep saying that you can’t do things in little chunks. you have to learn it all together. If we were to help developers learn what good unit testing means they could much more easily learn that stuff, and then learn the other stuff.

Casey writes:

“It is not a technical challenge to write a unit test, any half trained monkey could do it. A few attributes, a couple of classes, a few new() statements, and a few Assert statements. We don't need magic mocking frameworks, we don't need any mocking frameworks for 90% of our tests, we don't need application of 12 GoF principles before we can write a test, honestly.

Testing and writing automated tests is hard because it requires a strong understanding of the user stories that sit behind your code and project, because it requires you to think as a user and not as a developer, and because it requires you to be able to think of multiple conflicting axioms, test cases and class interactions, and how to cover the weaknesses of each individual test with other complementary tests.”

Casey preached about what he sees as the “end game” for unit testing, TDD and Design. where everything is in one great ball of goodness. Or did he?

Let’s separate these points into better chunks:

Design related stuff:

to think of multiple… class interactions

Good Test related stuff:

Understanding of the user stories
Think as a user not a developer
think of multiple conflicting axioms (..?) test cases..
how to cover the weaknesses of each individual test with other complementation tests.

TDD related stuff

none

So, what I gather from casey’s words is that you want your tests to be :

testing the right thing
be readable and show intent
have good coverage
test or be aware of class interactions (the design related stuff)

except the last point which is where most of today’s mock frameworks force you to redesign, there is nothing here you can’t learn separated from TDD and SOLID techniques. the interaction stuff is where those other isolation frameworks I mentioned come into place.

We need to decouple Good Unit Testing from TDD and SOLID. And it seems that not only “noobs”, as a stupid idiot commented thoughtfully on my last post, can benefit from learning this distinction.

Kanban, anyone?

RoyOsherove — Wed, 27 Aug 2008 21:41:49 GMT

ooh. This Kanban thing looks nice and shiney. Will it “meme” and die, or will it stay and flourish, to be the next “xp”?

Maybe it’s time to learn from our XP-Scrum experiences and evolve. Maybe Kanban is one path to that evolution.

Over Specification in Tests

RoyOsherove — Sat, 12 Jul 2008 10:23:07 GMT

I've gotten the chance to visit and consult various companies and I see this all the time: Over specification in tests, especially when doing mocks and stubs.

One main reason for that is that people use their stub as a mock object (stubs are fake objects that we will not assert on so that we can test something else). here is a small example from something posted on the alt.net mailing list:

[Test]

public void Can_Return_All_Plans()

{

IPlanDao mockPlanDao = MockRepository.GenerateMock<IPlanDao>();

mockPlanDao.Expect(x => x.AllPlans(_fakeAdmin)).Return(new List<Plan>{_fakePlan});

PlanController planController = new PlanControllerForTesting(_mockCommonDao, _mockCommonService, mockPlanDao);

ViewResult result = planController.All(false);

mockPlanDao.VerifyAllExpectations();

result.AssertViewResultNameAndViewDataType(typeof(PlanContainer), "Plans");

}

in this test the following line is used to "stub" out a return value into the system under test.

mockPlanDao.Expect(x => x.AllPlans(_fakeAdmin)).Return(new List<Plan>{_fakePlan});

no problem here. The problem lies in the last two lines of the test:

mockPlanDao.VerifyAllExpectations();

result.AssertViewResultNameAndViewDataType(typeof(PlanContainer), "Plans");

the second line (the assert) is probably the thing you really want to test. but the call to "verifyAllExpectations" is the one that makes the test very fragile. It is "asserting" that someone has actually called "GetAllPlans()" to get the results, when it clearly does not matter how one got the results. all that matters is that:

"Given a set of results returned into the application under test, the view in the controller is correct" (the translation of the last line in the test).

If by any chance the application later does several more calls to the stubbed out dao, or uses a different method to get results, the test could break, even though the application would still ultimately work with the same end result.

It's like me ordering a pizza delivery and then asking the delivery guy "did you get here by car or motorcycle?" . I realy don't care, as long as the end result: PIZZA, is correct.

So the general rules for me are:

If you see both "Verify" and "Assert" in the same test, it is usualy a smell of over specified tests
if you see "expect" and "Return" on a fake object, make sure that you name it "stubXX" or "mockXX" so that you can distinguish whether you want to call verify on it later or not(most cases should be "not")
Try to test on the end result or end state rather than verify interactions. The only time you absolutely have not choice but to test an interaction using verify is when calling void methods on external objects. that is clearly a one-way communication and is (or part of)the end result of what you are trying to accomplish.

A lot of people think that adding that extra "verify" just means it is a good thing since they are doing more assert. Well, they sure are testing more things, but they are internal things to the app's behavior and that is usually leading to brittle tests. Try to test on the end result or end state rather than verify interactions.

A sneak Preview of Typemock Racer : Thread Deadlock Finder

RoyOsherove — Tue, 08 Jul 2008 15:42:28 GMT

Here's a sneak preview of the upcoming Typemock Racer product, which I'm currently working on. We should be out in private CTP in a couple of weeks I hope.

Typemock Racer tries to solve one huge problem for developers working in multi threaded environments: Detecting code that can result in deadlocks or race conditions. It does this by providing a framework API which you can use in your tests (under NUnit for example), that allows you to execrise your code under test until it finds a deadlock or times out.

Here's an example of how you'd use it. Consider the following class which has two methods, that if run by two seperate threads, can result in a deadlock condition:

At some point in time the threads could execute such that thread 1 is waiting on resource b, which is held by thread two , which is waiting on resource a, held by thread 1 - a pure deadlock.

Given this class we can write a test that looks like this:

The ThreadTest class has an "AddThreadAction" method that receives a delegate. you can use it to invoke your own code, it will be invoked in a different thread. IN this example we are creating two threads, each one running a different method on MyClassWithLocks.

We can just right click and run the test with TestDriven.NET to recieve the following results:

The output states that there is a deadlock that was found, with the exact steps that happened (with line locations) to reproduce this.

It currently only works with lock (monitor.Enter\Exit) but will support all locking constructs in .NET.

What if the methods you point to create their own threads? Like this:

Here is the output that you'd get if you run this test:

Is Moq too poweful?

RoyOsherove — Sun, 06 Jul 2008 18:59:28 GMT

It's interesting to see the other mocking frameworks as they grow into maturity. Especially Moq which has been getting loads of new features recently. The latest one is something which makes me feel a bit weird after reading all the things people fear in Typemock Isolator:It seems that Moq now allows mocking protected fields.

What's really interesting is that Daniel has chosen to actively allow this, and hide the feature at the same time, as well as put it under a strict optional request:

you have to opt in for each code file where you'd like this ability by adding a special imports statement.

So, the questions I have are:

What happened to "if it's not testable it's a design smell"? Is this a compromise or have people figured out that good design can take many forms, with tools being left to do the hard lifting?
If you're going to add powerful "non classic" mocking ability to a framework, does adding it as a special "opt-in" feature make it OK? (this is what we are talking about doing with Typemock Isolator)
If tomorrow Rhino.Mocks were to add the same feature or something even more powerful (statics?) how would people treat that?

Two Faced Commits - how the ALT.NET Community is becoming more and more dogmatic

RoyOsherove — Mon, 19 May 2008 10:50:00 GMT

disclaimer: I work at Typemock.

There is a big trend I am starting to notice in the "Advanced" unit testing community. That trend feels more and more like hypocrisy. Take Daniel's post about mocking statics without Typemock. (Daniel is the creator of Moq, an open source mocking framework) Granted, it is important to know that you can always refactor things for testability, but just looks at what he writes:

"So, don't buy a product just because you're lazy and want to avoid creating a few wrapper classes and interfaces"

followed by

"Always use the simplest thing that could possibly work."

Which is it, community? The idea of being prgamatic is using the right tool for the job. if that tools makes your job easier than not using it "cause you're lazy otherwise" is plain old hypocrisy.

The same argument also holds for tools like FinalBuilder vs. Nant\MsBuild\Other XML Hell. Tools don't create problems if you use them correctly. The community (especially the ALT.NET community) is going in a too extreme direction. so extreme that their advice are sometimes skewed too much to be considered pragmatic, sometimes falling into "dogmatic".

Tough Questions about unit testing - Your comments required

RoyOsherove — Sun, 18 May 2008 17:30:00 GMT

As part of My Book I have a chapter on the challenges of integrating unit testing practices into an organization. One of the sections deals with "Tough questions and answers". In short, what are the tough questions you will have to deal with when introducing unit testing.

here are the questions I've come across but I'd love to see if there are some that I am missing. Also, I am looking for great answers and references as responses to these questions. Your comments are appreciated:

1 How much time will this add to the current process?
2 Will my QA job be at risk because of this?
3 How do we know this is actually working?
4 How much time will this add to the current process?
5 Is there proof that unit testing helps?

- any other tough questions?

- Got good answers?

Comments:

I'm using a beta test of Disqus.com's comments INSIDE this post so please try to respond there:

Discuss this topic on Disqus.

blog comments powered by Disqus

Agile Rejection Letter

RoyOsherove — Mon, 12 May 2008 23:51:51 GMT

"Dear Roy,
Thank you for your submission to the Developer Jam stage, entitled "Advanced Unit Testing Patterns, Tips and Tricks with examples in .NET". We received a number of great submissions for the stage--more than could fit in the space & time available, of course.
To select sessions for inclusion, the Developer Jam review committee (see http://www.agile2008.org/stage-developers.html) used a ranking system to indicate which sessions they would most like to see as part of the stage. We averaged the numerical rankings from all of the committee members to select the proposals with the most interest.
Unfortunately, your submission was not ranked highly enough to be included in the final program."

When will I cross the glass ceiling?

NullObject.For - As simple as it gets, but no simpler

RoyOsherove — Sat, 10 May 2008 21:19:11 GMT

This NullObject Factory from Paul is very nice, simple and clean in its design. I really like it and I can see many uses for it, in parallel with working with your standard mocking framework or IoC Container Implementation.

here is a little usage example (last line):

var controller = new AccountController(         // Class I am testing
    mockCustomerRepository,                     // Class I am mocking
    NullObject.For<ILogger>());                 // Who cares about logging!

Unit Testing Denial Pattern #1: Pretending the test is no longer valid

RoyOsherove — Wed, 30 Apr 2008 20:21:26 GMT

Today was both a good day for e and a bad day for me. Good day because I managed to get a full suite of tests that were somehow failing to work again after a refactoring that got out of hand. Bad - because out of all the tests, one of them failed.

Here's the part I'm ashamed of:

I was working with a pair, and when we both looked at the failing test, I went out and said "Oh, I think this test is no longer valid. It may just need to be removed". We both knew it was still a valid test, that for some reason was failing. But I was on such a role with making everything work, that test was feeling like a thorn in my thumb that I wanted to get rid of.

It was a hard few minutes to convince ourselves to debug through the test and realize what the problem may be

So, denial pattern #1:

Forces: You just want to get on with your code, with only one or two tests that are "ruining the party"

Behavior:

Instead of fixing them debugging them or accepting that you might not have fixed everything, you try to tell yourself that the tests may no longer be valid without actually checking your assumption, which you really wish would be true.

How to avoid: work in pairs. It's harder to ignore a problem (or at least more awkward) with 4 eyes staring at it.

Testing XML serialization attributes

RoyOsherove — Fri, 25 Apr 2008 02:42:23 GMT

Not sure how I feel about this particular brand of testing syntax. still, it's good that someone thought this problem through - testing xml serialization attributes.

[via the morning brew]

Future of Mock Frameworks: AEIS - Abstract Expectations & Isolation Syntax

RoyOsherove — Thu, 14 Feb 2008 23:10:00 GMT

As I was looking back and forward on the changing syntax of testing and mock frameworks, it occurs to me that there are still things that need to be solved in the mocking space. many things.

It's hard to choose a framework and stick with it because they have different abilities which you are not sure you might need or not later on
Each framework has different syntax which is a learning curve and maintainable curve in test code
You can't change your mind easily after you choose
There is no specified set of known features to be expected. Each one has what it thinks is best.

What if we had an abstract layer on top of our mocking frameworks? Just like ADO.NET is an abstraction layer, and the Data Application block is an abstraction layer, NHibernate is an abstraction layer - Mocking frameworks are infrastructure libraries that, if looked at well enough, are pretty much trying to do the same things in different ways (like databases)

Here are the main services a mocking framework can provide(if I miss something tell me):

o Stubs

§ Create Canned Answers

o Mocks

§ Create Expectations

· Record-Replay Model

· Explicit Model

· Parameter Constraints

· Custom param constraints

§ Partially Implemented Mocks

§ Pass-through mocks

§ Strict\non strict

§ IoC- Auto mocking containers

o Mocking Abilities

§ Strict (Traditional Interface based)

§ Non Strict (Typemock,JMockit Stuff)

There are two main services here: Expectation abilities, and Creation\Isolation abilities.

what if there was a framework that would allow you to write code like this:

 [Test]
        public void CreateSimpleInterfaceBasedMock()
        {
            MockingEngine.Provider = new RhinoMocksProvider();
            IMock<ILogger> mockLogger =
                Represent.Type<ILogger>().AsMock();

        ...

        }

What would the proposed AEIS (Abstract Expectation and Isolation Syntax) look like?

I would consist of a very thin API wrapper that allowed all the different services mocking frameworks do today, but it would not do this functionality on its own. instead it would use a provider model that will allow one to write adapters for specific mocking frameworks, forwarding calls and translating the different syntaxes between AEIS and the specific framework. I will also have a rules engine (on the left) that will allow creating custom rules that will run when using various APIs (perhaps using an event driven relay) that will allow to enforce specific usage of specific features.

what would be some of the benefits in something like this?

You can change mocking frameworks without changing lots of code

There is an investment to be made when choosing a mocking framework today: You spend time learning the syntax, you write tests with that syntax and you plan on not learning a new syntax to do slightly different things. A year later mock framework X comes in and you think that you'd like to use it, but you've already invested so much time and money in the original one that you aren't going to try. You lose valuable features just to avoid the risks.

You can add rules “on top” of the mocking framework

An abstraction layer can have custom code that will allow you to write custom logic that makes sure developers are using the framework in a way that you'd like. for example, you'd like to use Typemock Isolator but you are afraid that people will abuse the ability to write tests for code that isn't "testable" by default (no interfaces etc..)

I call that"Strict" mocking policy vs. "loose" mocking policy (or "classic\traditional vs. modern..."). So the
o Allow only strict mocking syntax, for example

One syntax to remember and learn for all types of jobs

Simplify and overcome problematic mocking syntax in existing frameworks

And there are definitely some around. For example, Typemock Isolator currently has a very confusing Mock() and MockObject() set of methods which sometimes I personally forget what each one does. Sure, we'll change it, but RhinoMocks also has some confusing syntax (PartialMocks() vs. DynamicMock() Vs. Mock()...)

Having a top level API can help alleviate and standardize some of these issues.

Provide a recipe for mock framework builders

If you have to build a provider for your mocking framework, you have to have a contract of what you can and cannot provide. this will also help standardize the essential set of services that mocking frameworks provide today or at least, the set of things you need to consider when choosing among them as the underlying engine.

It will help avoid religious wars (like design for testability) by allowing you to choose easily and enforce to your standards through the common API.

What about testing frameworks? why not start there?

I think currently most test frameworks (XUnit.net aside) are pretty much interchangeable (Nunit and MbUnit and msTest) so there is less pain there. However, take a look at the next version of MbUnit , running on the Gallio automation platform) which tries to provide a generic test runner model for all test frameworks in .net. I think they are trying to solve the same set of pains and the same set of benefits.