in

ASP.NET Weblogs

Extreme JS

JS Greenwood's WebLog on architecture, .NET, processes, and life...

When TDD Goes Bad #1.1

OK, so people don't seem to get what I was trying to say in my previous TDD post, to the point of claiming (on certain newsgroups) that I "don't get TDD".  There were two points I was trying to make. So, I'll try and lay out the first of these now, in a simpler manner:

  • Every test you write has a cost associated with it - the development time/cost: C
  • Every test you write has a value associated with it - the time/money it'll save over the lifetime of the system: V
  • At its simplest level, V must be greater than or equal to C for it to have been worth implementing the test (I'll probably post something separately on the mathematical modelling of this in the future)

The reason that V isn't always greater than C can arise from several situations from a "story" (business) perspective:

  1. Developers have been provided with open ended requirements - where there is no bounding to the tests that have to be written.  At its most extreme case, this could be something like "Test: An unexpected physical occurence takes place outside the system. Result: The data will be backed up correctly".  That's an open ended test, though.  What if someone cuts power?  Or a lightning bolt hits the server rack?  Or a tiny blackhole forms over the CPU?  "C" would become infinite if all scenarios were dealt with (just because I've used an extreme example here doesn't mean it couldn't be something much more subtle, either).  Some of these scenarios are more likely than others, and there's diminishing return on each test being implemented as the likelihood of the scenario occuring drops.  To stop this situation from occuring, requirements need to be closed and finite.  If they're not closed/finite, you arrive at the situation in my previous post where a developer can simply "judge" how much testing is enough.  Unless developers are they calculating the C:V ratio of each test in a situation with any open-endedness, then TDD can go bad.
  2. Developers have been provided with requirements with negative value - where the cost of implementing the test + code is inherently greater than the value it will deliver.  Accurate estimation of tasks and comparison against the value of any given story will avoid this situation.

[In part #1.2, I'll discuss the second part of this]

Published Dec 07 2004, 05:43 PM by jsgreenwood
Filed under:

Comments

 

TrackBack said:

December 7, 2004 7:51 AM
 

Thomas Eyde said:

The reason why we, or at least I, don't get what you try to say, is that what you say is totally alien to me. I don't recognize any of the situations you describe. It would be easier to understand if you could provide a live example when TDD actually did go bad. I am eager to know this, because it's good to know when or if TDD is not the way to go.

But most of all, I have this feeling we are not talking about the same kind of tests. In my experience, TDD pays off everytime. It's so hard to make code cleaner, or to fix bugs for that matter, without the unit tests TDD gives you.

Most important: If the requrements are so vague you can't define the next test, then you know too little to start any coding. If you still decide to code, you generate loss, not profit because you have to do it over.
December 7, 2004 2:11 PM
 

JS Greenwood said:

Thomas, I gave as much detail as I'm willing to about a real-world situation in a comment on my previous "When TDD Goes Bad" post (probably more than I should have, actually). To give any more would not only be unprofessional, but could end up with a particularly awkward situation with evious employers/clients.

Suffice to say, this isn't necessarily about "vague" requirements or TDD not being appropriate. TDD is always appropriate, but like any tool, it can be misused. And you can have absolutely specific requirements, rather than just vague ones that are open ended.

From a function point aspect (I'm tired of writing stories for now), here's a simple (made up) example:
"When an error occurs in the system, whether it's caused internally or externally to the application, it will be logged with a stack-trace to the EventLog. Should writing to the EventLog fail, a globally-configurable UNC path will specify a flat-file to log to"

So, we've got our requirement. And we can calculate a value, V, for having that correctly implemented in the system (based on the time taken to trace-faults, the saving over not having logging, etc). But, in theory, we can test for a million things; kernel errors, .NET Framework issues, application issues, caspol settings to ensure we have/can view a stack trace, the event log service not running, full event logs, event-log sources existing, event-log write permissions, network paths for flat file-logging, file permissions, full disks, network connectivity issues, viruses, file-locks being held by other applications.

Admittedly, this is a contrived (and not so great) example. Real-world situations I've seen have been much more subtle (and complex). But it's not that hard to get to the point where the cost of developing the tests has outweighed the value of the software (some things are just inherently hard to test).
December 7, 2004 3:31 PM
 

Pablo Montilla said:

What we do when using TDD is to *drive* the development via tests. We are designing by example, the tests themselves being the examples.

I have problems with what you say on two camps:

First, even with rigid, closed and not vague requirements, you cannot test for every possible occurrence in wich the code will be used. That's why you use your tests to setup a context in wich the classes you are testing will be used. So not only you need to know what your classes should be doing, but in wich context, and with tests or without them you need to know it to write the code (and with tests you'll be better able to express that contexts of use).

Second, the value V that you assign to writing the tests cannot be measured in isolation. Your tests are acting as a driver of the design *and* as a safety net should changes in other parts of the system (that context I've written above) affect what you are testing, so their value increases with the size of the system, not with the size of the tests themselves.

I think I get your point, the developer should know when to stop. But that's the case with the tests *and* with the code.
December 8, 2004 9:27 AM
 

JS Greenwood said:

Pablo you're spot on - it is about knowing when to stop. :)

To cover your other points...

* You're right, V cannot be always be accurately measured. But that doesn't mean it shouldn't be estimated, that you shouldn't calculate value where possible, or that you shouldn't state assumptions and check them further down the line. Simple calculations can regularly reveal things to be an order of magnitude out (one way or the other).

* The "rigid, closed, and non-vague requirements" isn't about testing in "contexts" as such, it's about testing against specific requirements. Technically, you can test for everything defined in a "finite" requirement (that's what makes it finite), the economics of this may not be viable, though. As for every context, you're right - you can't test all of these, that's why the requirement should be "closed" - to specify the contexts (or the default context should be assumed).

Please see http://weblogs.asp.net/jsgreenwood/archive/2004/12/08/278332.aspx - part #1.2 for a concise "developer oriented" summary of the overall point I was trying to make. #1.1 was just about where problems can occur from the business angle.
December 8, 2004 11:17 AM
 

JS Greenwood said:

"What we do when using TDD is to *drive* the development via tests. We are designing by example, the tests themselves being the examples."

Forgot to say - totally agree with Pablo here, too. Exactly how it should be.

(I'm just trying to describe an anti-pattern when things go wrong, after all...)
December 8, 2004 11:19 AM
 

TrackBack said:

^_^,Pretty Good!
April 10, 2005 12:00 AM

Leave a Comment

(required)  
(optional)
(required)  
Add
Terms of Use