Extreme JS

JS Greenwood's WebLog on architecture, .NET, processes, and life...

When TDD Goes Bad #1

Although Test Driven Development (TDD) is one of the greatest steps forwards in software engineering, especially when combined with modern languages and testing frameworks (i.e. xUnit), there's a definite anti-pattern lurking in there - Test Oriented Development.

At its most basic level level, writing tests is done for one reason and one reason only.  It's not about ensuring you meet requirements, it's not to have a repeatable means of knowing the code functions properly after changes, and it's not to aid design.  The key reason for writing tests is the same as everything else in software development - to make the company commisioning the software more profitable.  So, the justification for TDD is based upon the premise that, in the long run, writing tests will save the company money on down time, effort in re-workings, and so on - but these are just implementation details of the underlying principle.  Now, although most developers that do TDD understand the long-term payback view of the approach, I'd question how many actually consider the value of each test or set of tests that are being developed.

Using an agile methodology, tying the (acceptance) tests to the stories is fairly simple, seemingly giving visibility of the value of tests.  But, given that many verbal tests are quite open ended, how do you know how much testing is enough?  The test may be stated as "When I enter the correct data, but there is a connectivity issue, I will see error message X".  A verbally described test such as this can have a hundred different programmatic tests implemented - simulating standard firewall issues, XML-firewall issues, web servers being down, latency in connections, time-outs, corruption of data, etc.  There is no immediately visible answer to "how much testing is enough?" in such a scenario.  This is even worse with unit tests, where they aren't likely to be defined formally (verbally) at all, and are largely up to how meticulous a developer is.

Added to the lack of metric is the fact that the developer is concentrating on testing before "coding" - the primary function is hence the development of tests, leading to a situation of "test-oriented development".  This is where the production of tests becomes the overriding concern and output of a team, with the development of required functionality being secondary.  When this occurs, the design, quality, and scope of the tests all become greater than that of the system actaully being created.  As the development of tests follows the law of diminishing returns principle, beyond a point, each additional test you write for a certain circumstance adds only more and more marginal increases in quality.  Eventually, you will reach a point where there will never be a point in the lifetime of the system where the development of the tests achieves ROI.  Basically, the development of tests becomes counter-productive to the aims of the business.  This is compounded by the Heisinger's Uncertainty Principle-like facet of TDD - it's alteration of the design of the code it's testing.  But that's a subject for another post in the next few days...

Avoiding test-oriented development can be quite difficult - changing a developer's mindset to take on TDD is difficult enough to start with, trying to then amend that to temper the creation of tests goes back on that principle to some extent.  My current thinking on this is that, along with acceptance tests and development tasks, a separate list of requirements needs creating for each story - a list of required unit tests.  Rather than just estimate the effort required to complete a story point in its entirety, the effort required for each unit test (and its value in pound-notes/dollar-bills) should also be calculated to some extent. Each test is then subject to the same cost-benefit analysis and prioritisation as stories at a higher level.  The task of doing this should also help developers to consider the reason for creating a test; how realistic/contrived the situation it's testing is, how better the effort could be expended, and finally, help remind them what they're actually trying to achieve at a less programmatic level.  If this seems like too much effort, then an alternative may be to look at the ratio of time that is to be spent on writing tests to that of the production code itself for a given story-point to ensure that the focus isn't shifting from the system to the tests.

Posted: Nov 26 2004, 12:58 AM by jsgreenwood | with 45 comment(s)
Filed under:

Comments

TrackBack said:

Will's Blog - My.Thoughts == My.World » when TDD goes bad
# November 25, 2004 5:03 PM

Ian Cooper said:

I would be cautious here when unit testing you need to test 100% of the things that could possibly break. Why? Because that allows you not only to have confidence in your code, but also enables refactoring. Making your code low-cost to change is a key point of the XP development process which seek to flatten the cost-time curve so that changes are euqally costly at any point in the system's life i.e. software does not become brittle. If you have areas of untested software, your ability to make changes with confidence diminishes, because you do not know if you are breaking things that used to work. So the cost of change then increases over time and you have lost the key objective of 'embracing change' and making your software amenable to change

However developers do need to show caution in differentiating between unit tests and acceptance tests. TDD writes unit tests. Customers define acceptance tests. Developers might implement automation of those acceptance tests, but those end-to-end tests are acceptance tests and originate with the customer. They are the 'sign-off' agreement with the customer. Acceptance tests do not need to run at 100%. but a level that customer considers is 'good enough' to ship code. That is where your cost vs. quality balance is raised.

If you worry that your developers are over-testing it may be that they are writing acceptance tests for unit tests. Heavy collaboration of classes in the test, not using mocks to replace that, end-to-end running, going to long between running the tests because your tests are too complex are probably 'bad smells' here.

Remember XP proposes that maintenance is the norm for software and to avoid maintenance problems - the big ball of mud - you need to use TDD aggressively. Skimping on unit tests is software short-termism.

# November 26, 2004 4:38 AM

Frans Bouma said:

"At its most basic level level, writing tests is done for one reason and one reason only. It's not about ensuring you meet requirements, it's not to have a repeatable means of knowing the code functions properly after changes, and it's not to aid design. The key reason for writing tests is the same as everything else in software development - to make the company commisioning the software more profitable. "
This is the key mistake you make. Test driven development is solely to make sure that an interface IMPLEMENTATION is not suddenly broken when something is changed in the implementation, i.e. it is a key part of refactoring. Refactoring and unit tests go hand in hand.

I agree with the term: test oriented development and that it can be an antipattern, i.e.: "the unit tests I wrote prove the code is bugfree". No, the tests prove that the functionality embedded in the tests is bugfree.

Test driven development is however a step further than the refactor - unittest combination: it demands you write the tests for ALL the functionality you have to implement PRIOR to the actual code. This has the advantage that you already test the design of interfaces and hte functionality they embed before you actually implement them.
# November 26, 2004 4:57 AM

JS Greenwood said:

> This is the key mistake you make. Test driven development is
> solely to make sure that an interface IMPLEMENTATION is not
> suddenly broken when something is changed in the implementation

At a certain level, absolutely. But the key goal of every person in every (supposedly) profit making company is to increase profits. Period. Noone is hired into a role for any other reason than to make the company more successful.

> I would be cautious here when unit testing you need to test
> 100% of the things that could possibly break

I absolutely agree with this... In principle. But sticking rigidly to such a stance when the financials don't weigh up is just a fundamentalist stance. For instance, if you have a system that is only going into live for a finite (known) period of time; if you know what the maximum value that can be derived from this system is, and the (lack of) dependencies between other sub/systems, and the cost thereof, then you can easily calculate whether implementing more tests is actually in the benefit or to the detriment of the company.

If a system is to be "in live" ad infinitum, and the dependencies and interactions can't be accurately measured (but are known to be significant), then yes, 100% test coverage makes sense. But this isn't always the case, so having a rigid stance is counter-productive. Additionally, you will never know if you really have 100% test coverage - it doesn't matter what NCover says - there'll no doubt be external dependencies unaccounted for.
# November 26, 2004 7:07 AM

Curtis Olson said:

I think there is not too much to disagree with from any of the comments here, the viewpoints seem closer than might initially appear. Still, "short-term" software often lives much longer than intended, I'd tend toward testing more than testing less.

But the main thing I wanted to say, and I hope you'll forgive my pedantry, is that the physicist you should reference is Heisenberg rather than Schroedinger.

Cheers,
Curtis Olson
# December 2, 2004 2:50 PM

David Anderson said:

It's amazing how few software engineers actually get the basic fundamental of collecting a pay cheque (note bowing to British spelling on this site).

You don't get paid to write nice clean, maintainable, working code. You get paid to come to work and generate profit for the owners of the business. If you can't demonstrate that any change you make in your working practices is generating more profits then you shouldn't be doing it.

Ian Cooper's economic argument that TDD flattens the cost of change curve is unproven scientifically. No one has demonstrated how TDD avoids fundamentally flawed architecture which is recorded in the RAD and iterative literature long before agile was around. If an architecture is broken then the refactoring requires that all the tests are refactored too. How does that flatten the cost of change? The truth is it doesn't. There is no replacement for good forethought in architecture and the real asnwer is to find architectural techniques which are robust to change. Emergent design in TDD does not offer this.

There is doubtless value to TDD. I genuinely believe that it helps certain types of developers, who are good at modeling and design, to think problems through and write better quality code. It also does improve confidence in code and reduces the cost of regression testing.

However, the true economic benefits of TDD and the tradeoff in time spent of test development versus coding has not yet been demonstrated in any objective framework. The writing on TDD amounts to little more than subjective anecdotes which say "we did this and our code quality improved". Whoopie do! Can you show that profits improved in your employer's financial statements?

TDD remains an unproven subjective technique until someone in the academic world gets to grips with it and produces an objective framework for measuring its performance on large scale real world software development problems. Only then can executives in public companies make informed decisions about whether or not TDD is right for them.
# December 2, 2004 6:59 PM

Ronald E Jeffries said:

Although I've been working in and coaching agile development for about 8 years now, I have /never/ seen a case where "the production of tests becomes the overriding concern and output of a team, with the development of required functionality being secondary" as you suggest. I do know of a team which was directed by its management to spend some months catching up on tests. Since that was a directed behavior, it doesn't seem to me to be the same as the anti-pattern you're describing.

Further, I believe that the following quote shows a misunderstanding of what TDD is: 'Added to the lack of metric is the fact that the developer is concentrating on testing before "coding" - the primary function is hence the development of tests, a situation that occurs quite regularly is "test-oriented development".' In TDD we don't write many tests before coding, we write exactly one, make it run, then write another. Writing a whole batch of tests ad infinitum would be a bad idea -- but it wouldn't be TDD.

I am wondering where the original author, and some of the other posters, have gained their impression of what TDD is. It would not appear to have come from the books or from working with someone who understands it.
# December 3, 2004 2:02 PM

Thomas Eyde said:

When I switched to TDD, I discovered I spent less time in the debugger, I spent less time redoing things and I generally have fewer bugs. I can use the time I save on this to crank out more code, which means more functionallity, which is what the customer pays for. How's that about generating profit?

And don't forget maintenance cost in the profit equation. It's easy to lose all profit in bug-hunting and maintenance nightmares. Creating nice, clean code reduces maintenance efforts which also generates profit.

I have spent enough time in my career on silly bug-hunting, with TDD those days are gone. Yes, there are still bugs, but they are fewer and far less critical.
# December 3, 2004 4:34 PM

Christophe Thibaut said:

For TOD to be a definite anti-pattern it should have been encountered in several professional situation. Can you point your readers toward real stories where some developers were stuck in this anti-pattern ?

Adding several new tests to an unchanged code seems completely at odds with TDD practice. It's just as if, while counting peas, you'd count one pea several times, just to be sure..

Regards
# December 3, 2004 4:52 PM

JS Greenwood said:

Marvellous; differing opinions on this - just what I was hoping for. :) In order...

* Curtis Olson: "the physicist you should reference is Heisenberg rather than Schroedinger."
Whether I refer to Schroedinger's cat, or the Heisenberg uncertainty principle, someone ALWAYS says it's the other. Someone REALLY ought to put up a site clearly explaining when to use each analogy. Personally, I'm no physicist, so I'm happy to accept I'm wrong... but it does sound to me closer to the cat than the uncertainty principle.

* David Anderson said: Things about TDD being qualitative rather than quantitative.
Absolutely. TDD clearly makes sense in principle. But in practice, we've got to be careful that we don't take the principle as a carte-blanche to keep writing more tests, feeling that we're automatically "adding value" with each. Any given test has a finite cost of development and a finite value it delivers. The value must clearly outweigh the cost for the test to have been worthwhile. Factors such as the longevity of the system, chances of the condition being tested actually occuring, and so on all alter that cost:value ratio.

* Ronald E Jeffries said: "The following quote shows a misunderstanding of what TDD is"
Exactly my point. Anti-patterns generally emerge where people don't understand the implications of the application of a principle. TOD will occur where people understand the technical implementation details of TDD, but not the wider picture. If you've been working in Agile coaching, I'm guessing you've had the ability to guide inexperienced teams until they understand the principles. I'm sure there are numerous teams out there going down this route with little guidance, however, leaving plenty of scope for error.

* Thomas Eyde said: A perfect summary of TDD working
Spot on. And that's how TDD should be. I'd probably still question if you /categorically/ know that the up-front time spent writing tests is less than the time spent maintaining code; whether you've got proper metrics around this - the %age benefit, etc. I absolutely believe that you should be able to prove this, and it will probably come out with a large saving - that's what TDD is about, and it wouldn't have developed such a following if it didn't offer benefits. That doesn't mean people can't get it wrong, though... There are no widely accepted models for calculating the value of given tests, and so on, which leaves plenty of room for error.

* Christophe Thibaut said: Show me the proof
Having re-read my post, I was possibly a little heavy handed with stating the frequency of the anti-pattern... words like "regularly" were too emotive (but I do like to generate a reaction). Yes, I have seen this in industry, by people from at least two companies. I've seen systems where the tests were better structured than the code, where a convoluted test that will have cost the client thousands has been developed, yet the chances of their occurence are negligible at best, and will never return ROI. I don't think it would be professional to go into detailed specifics on here, though. However, the worst case of this was on a public sector system, about one month out of just over a 2 month development cycle was spent creating tests around a temporary solution that was only to be put in place for a couple of months while the separate, full system was prepared for installation. The more effort that was spent on this temporary solution, the less that could be spent on the final one, etc. I won't go into any more detail, but it was a very sorry state of affairs.

In summary: don't take this post as a bashing of TDD. It's not... as my first sentence stated "[it's] one of the greatest steps forward in software engineering". Just because something can be used to great advantage doesn't mean it can't be misused to detriment, though.
# December 4, 2004 9:03 AM

Curtis Olson said:

Re: Schroedinger v. Heisenberg

Schroedinger's Cat is a comment on the paradox of probability calculations in quantum physics. Heisenberg's Uncertainty Principle is a comment on the paradox of measurement in Newtonian physics.

Heisenberg states that you can't measure position and velocity at the same time accurately without the act of measurement changing the result. I think this is closer to what you indicated may be one result of unit testing.

Schroedinger's Cat is inside a black box, you can't measure anything at all, you can only calculate probabilities as to whether the cat is alive or dead. Schroedinger's calculations indicate that the cat is both alive and dead at the same time. Of course it can't be both, but we can never know which. This is probably closer to the "testing" most often practiced in IT.

Were you right or wrong? How about both at the same time ;-) Quantum mechanics is too weird.... By the way, my physics grades were less than stellar, so feel free to trust another source.

Cheers,
Curtis
# December 6, 2004 2:08 PM

JS Greenwood said:

Cheers Curtis! Feedback appreciated, and article edited to satisfy the pedant in both of us. :) Still not sure either's actually a direct match for this. Think I'll stick to hyperspace and multi-dimensionality based physics. All makes far more sense to me than the hack that is quantum theory. <grin>
# December 6, 2004 5:11 PM

Darrell said:

I'd say that 100% code coverage is not the goal. Ron Jeffries would disagree with me here, :) , but if there is a tradeoff between cost in money and time and the possible payoff. As Frans said, tests don't prove your code works. They just prove that in certain cases it does work. There are even times when the tests you have don't necessarily mean that the code works, especially for non-deterministic things like race conditions.

If I have a objects that basically delegate calls to objects that are tested, such as the topmost UI layer or a facade, then the payback in writing the tests may not be there, especially if the test code is difficult to get right or time consuming.
# December 6, 2004 10:17 PM

Anna Shchurova said:

This is just my opinion:
Please read this http://www.developertesting.com/archives/200411/LiveBlogFromDeveloperTestingForum.html#DTG and maybe you will find that there is a way to measure benefits from using TDD - LESS BUGS IN YOUR CODE and MORE STABLE BUILDS!!! Also by writing test first it forces you to think how you are going to use the object the functionality of you are about to test because until you try to use it you don’t really know what you need and want this object to do. I am sorry but I think this post doesn’t make sense.
# December 7, 2004 11:22 AM

JS Greenwood said:

Anna, please see http://weblogs.asp.net/jsgreenwood/archive/2004/12/07/277694.aspx, and the follow up I'm going to post shortly.

Hopefully the couple of posts I'm going to put up there will explain it better...I *do* believe that TDD, in the general case, has a measurable benefit. And that *all* professional developers should be doing it.

However, there are definitely cases where it can fail - for two human reasons (as I'll describe in those posts):
1. (Business) people aren't all that good at defining requirements
2. (Techie) people need experience at writing tests to be good at it
# December 7, 2004 12:51 PM

JS Greenwood said:

Darrell, agree completely with what you're saying. (I'd actually say that 100% test coverage is a "goal", but has to be tempered and controlled with economics)

The problem is, how do you know upfront if you're going to get payback for any test that you develop? Some tests are easy... like the ones we write to see if we have file permissions to write to log files, etc. that we know will get frigged everytime somebody patches a box. But some tests are both/either much harder to place a financial value on, or the cost/value equation is a marginal call.
# December 7, 2004 12:55 PM

Thomas Eyde said:

One of the biggest payback from TDD is that refactoring becomes second nature. You are so used to do it, that you do it all the time. That gives you a cleaner code base, hopefully with simpler code, but also with less duplicates. That pays off in maintenance or when you have to add more features. The only project I know of which doesn't need maintenance or never had more features added, is the canceled one.

But TDD takes practice to do well, so beginners will do curious things when they start. But then their skills improve. I am also not surpriced to hear of consultants who misuse a technology to generate their own profit.
# December 7, 2004 2:42 PM

Short-term systems? said:

"For instance, if you have a system that is only going into live for a finite (known) period of time; if you know what the maximum value that can be derived from this system is, and the (lack of) dependencies between other sub/systems, and the cost thereof, then you can easily calculate whether implementing more tests is actually in the benefit or to the detriment of the company. [...]"

This is quite the straw man argument, only because we /don't/ know the value of any nontrivial system. (See /Waltzing with Bears/ for a discussion of this.) On the contrary, TDD shines a big light on the fact that, in general, our clients can't even estimate the value of the features we implement for them. In this case, it's generally safest to build those features with the highest quality we can, so as to reduce their cost. If we cut corners, we do so at our own, blind risk. Sometimes it pays off and sometimes it doesn't.

Also, I wonder about the notion of a system that will be put in place for a finite period of time. I wonder about why systems die. Typically, they die because the cost of adding new features exceeds the value of adding those features; however, continuous design improvement helps us to reduce software entropy, staving off the heat death that claims most large systems. We cannot safely improve the design of existing code without tests, so the tests actually help us cheat death in our systems. The more systems are test-driven, the less the change they need to be taken out of service and superseded by something completely different. (At this point, technology change becomes the bottleneck -- we might replace the system only because J2EE is /so/ passe, and we want our system to be built on .NET.)
# December 7, 2004 4:08 PM

JS Greenwood said:

Here we go....

"we /don't/ know the value of any nontrivial system". If we don't know the value, then why are we building it? That's not agile, trivial or not. If you don't know the value, then you don't know whether it's worth implementing software full stop.

"our clients can't even estimate the value of the features we implement for them. In this case, it's generally safest to build those features with the highest quality we can, so as to reduce their cost". Sorry, but that's the most anti-agile argument I've heard in a LONG time...
1. If you don't know value, you don't know what's an acceptable level of cost
2. If you don't know value, you don't know whether it's worth implementing in software at all
3. You don't "automatically" reduce their cost by building things to the highest quality, or there'd be no such thing as over-engineering. Simply implementing TDD does NOT guarantee a positive ROI. Stating "Sometimes it pays off and sometimes it doesn't" is a clear sign of a lack of visibility/measurability, and isn't in anybody's best interests (well, maybe yours if you're getting paid regardless)

"I wonder about the notion of a system that will be put in place for a finite period of time. [...] Typically, they die because the cost of adding new features exceeds the value of adding those features". That view may be based on your experience, but there are other sides to it. In the last few months alone, we've had to develop two financial products for the bank that have well-known, finite lifespans. One had to be in live for just over a month, and another had to be in live for a few months. Both were customer facing, financial products that could be purchased. I could give dozens more examples of systems like this from numerous industrial sectors, across several companies I've worked for.

Your comments about entropy without tests are spot-on, and that's why I'd always advocate TDD, especially in systems that will be entrenched for unknown (but long) periods of time. I get that, and spend half of my time advocating TDD across the enterprise... (I don't write tools like NUnitMSI, a new GUI for NUnit, as well as reviewing articles on TDD for magazines, just to slap it down!)
# December 7, 2004 5:00 PM

TrackBack said:

# December 15, 2004 7:16 PM

payman_arab said:

Hi

thanks for your idea,but I have a problem yet, I want to upload picture on the web but my pictures are very weighty and laden so I'm finding a way that I can send thoese faster. if anyone know a good way Please show me. Thanks.

please mail to me: king_kurosh@yahoo.com

# October 3, 2008 4:28 AM

Chris said:

I am always wondering what people mean by testing all the things that might break or 100% coverage... According to which metrics? Do we have to test all possible paths? You should not test everything that might possibly break unless you are ready to see the number of tests grow exponentially. I am not obsessive about testing and it does not always help with refactoring if your unit tests have to be rewritten to fit the new design (this is less true of acceptance tests which should always reflect what the customer wants). TDD is not even about testing...

# May 24, 2011 8:12 AM

oem software said:

txMpiJ Great, thanks for sharing this post.Much thanks again. Fantastic.

# May 2, 2012 2:07 PM

Cheap Seo Services said:

dU7KA4 Great article.Thanks Again. Want more.

# July 9, 2012 9:48 PM

Fitness Trackers said:

Muchos Gracias for your article.Really looking forward to read more. Cool.

# July 11, 2012 6:03 PM

sticky tape said:

Major thankies for the blog post.Really looking forward to read more. Want more.

# July 15, 2012 4:46 AM

dymo label printer said:

Thanks for sharing, this is a fantastic blog. Really Great.

# July 15, 2012 7:52 AM

laminating rolls said:

Very neat post.Much thanks again. Will read on...

# July 15, 2012 10:59 AM

executive chairs said:

Muchos Gracias for your blog post.Much thanks again. Fantastic.

# July 15, 2012 2:31 PM

rubber band said:

A round of applause for your blog post.Much thanks again. Much obliged.

# July 15, 2012 11:53 PM

seo training in delhi said:

I truly appreciate this blog article. Awesome.

# July 16, 2012 1:50 AM

cell phone tracker said:

Appreciate you sharing, great post. Great.

# July 16, 2012 6:31 AM

Fat Burning Programs said:

I really liked your blog.Really looking forward to read more. Fantastic.

# July 16, 2012 5:47 PM

office depot laminating pouches said:

Very informative article post. Great.

# July 16, 2012 8:28 PM

gbc laminator pouches said:

Awesome blog post.Much thanks again. Much obliged.

# July 17, 2012 8:26 AM

brain guru said:

ASP.NET Application Company in india - Jobs in Brainguru

We hire ASP .NET Developers for asp.net application and portal development in Noida, India.

Keywords:-  ASP.NET Application Company in india, Jobs in Brainguru

Brainguru Technologies Pvt Ltd.

C -17, First Floor

Sector - 2, Noida

Call Us: +91-8010010000 / 9311347742

Email us: info@brainguru.in

brainguru.in/.../asp-net-application-company-noida-india.html

# September 3, 2012 9:03 AM

cheap seo services said:

32wNnb Enjoyed every bit of your blog post.Much thanks again. Keep writing.

# September 19, 2012 12:48 PM

cheap seo services said:

DeYEHt This is one awesome post.Really looking forward to read more. Really Cool.

# September 19, 2012 11:05 PM

RapSmump said:

must look at this <a href="www.cheapraidersjerseys.com/">raiders jerseys cheap</a>  for gift  

# November 5, 2012 11:14 PM

kraaotxi@facebook.com said:

Keep functioning ,terrific job!

# January 25, 2013 7:37 AM

loss weight pills said:

78kgxi I am so grateful for your article.Really looking forward to read more. Will read on...

# January 31, 2013 12:06 PM

Stabilo Highlighters said:

Really informative article post.Really looking forward to read more.

# February 2, 2013 9:00 PM

Staedtler Pencil said:

Looking forward to reading more. Great blog post.Thanks Again. Really Cool.

# February 2, 2013 10:13 PM

Small shredder said:

I think this is a real great blog post.Really looking forward to read more. Fantastic.

# February 3, 2013 12:51 PM

cysfbn@gmail.com said:

I admire the things you have conducted here. I adore the thing where most people say one does this to give back but I'd personally assume by all of the comments that is certainly helping you as well. Do you may have any advise on this?

# February 28, 2013 12:11 AM
Leave a Comment

(required) 

(required) 

(optional)

(required)