Justing Etheredge posted a great article about that Design Up Front (DUF) is something else than Big Design Up Front (BDUF). He discusses the misunderstanding that just because BDUF is considered harmful in a lot of projects, it's not said that DUF is too and one should just jump in, start hammering in code and hope for the best. As an example of BDUF, Justin uses the example of a car manufacturer and asks himself why a car manufacturer uses BDUF (according to Justin) and we in software land often don't or at least try to avoid it.
One of the arguments he uses is that software is easily changeable, or at least easier changeable. I beg to differ. Sure you can change text in an editor, use some refactoring tool to get very low level refactorings on your code, but that's similar to hammering the drive train which doesn't fit till it does: it doesn't change a design. It changes the result of the design. One of the biggest problems in software land is that clients / customers think software is easy to change and that they can ask for changes right in the middle of a project or even at the end. Ever saw a large building which was almost completed to be transformed into something else with the swimming pool at the top instead of the basement right before it was done? No, because every human knows that changing a building when it's finished is very hard to do.
But software has the same restrictions.
Software parts represent functionality. This functionality is consumed by other parts so those parts can deliver their functionality and so on. This is a functional dependency, not necessarily a technical. I'm not talking about code, I'm talking about functionality and the design of the functionality elements inside the total system and subsystems, data flows between them etc. etc. Code is created from these functionality elements, so the programmer knows what to write and the architect knows what that piece of code does. That's Software Design 101, and for the hard-boiled 'code == design' visionairs: think again why you wrote class ABC the way you wrote it and not differently.
It's these functional dependencies which make it hard to change a design after code has been written. The reason is simple: changing the functional elements often leads to a lot of changes in the code (which is nothing more than the executable form of the functional elements). Changes which aren't made from the code's Point of View (POV), but from the functionality's POV. This means that refactoring code isn't a technical matter anymore, but a functional one: the basis of the code, the sole reason the code is there has changed.
So, Justin has a good point in stressing the fact that DUF is required. What's left untouched is when a design becomes 'Big'. I think what's meant with BDUF is that the design is completely done up front and not changed. DUF is more of a design where the borders are known, the general goals are known but the details in a lot of areas are filled in along the way. There are two things important to make this work:
- Know the goals of the project. A comment by Glenn Block has a good point: often the goals are unclear so what a project should become is unclear, so doing design up front is a task which can't be completed as what to design is unclear. I think that the acknowledgement of not having a clear set of goals is something which should lead to a quest to get these goals defined properly before anything is done further. Remember: this doesn't mean you've to define details. It means you have to analyze and determine what the core problems are the software is trying to solve and what the possible solutions are to these problems.
- Details shouldn't be able to change the design. To be able to fill in details during a project, you shouldn't run the risk of having a required detail forcing a significant change in the project, which for example could change a lot of the rest of the project already written.
Agilists will now say that by following Kent Beck's rules their software automatically evolves into something a customer / client wants. Though the engineers at BMW as described in Justin's example also are able to provide a car which works, drives well and a lot of customers are willing to buy it. If BDUF is so bad, why are BMW able to produce cars which work well, using BDUF? Justin uses the argument of 'costs' to illustrate that within software we simply can't do BDUF. Though I think costs have nothing to do with it. One of the reasons is that building cars also costs a lot of money, millions if not billions. Although BMW's cost a reasonable amount of money, small European cars are often very inexpensive and are build with the same principle. Having 1 billion in development cost is a big hurdle to overcome when a company wants to make money in the end.
If you consider the fact that large software projects also cost a lot of money (here in the Netherlands we have had some terrible government project failures recently which added up to more than a billion Euros), and in general are meant to bring other costs down, it's first-grade math to conclude that a software project has the same restrictions: to make a profit, costs shouldn't be too high. That costs shouldn't be too high isn't an argument that costs are relevant or a reason BDUF doesn't work: a failure is much more expensive: having to throw away a lot of the work done because an important part of the design has to change is very costly, so the path followed by BMW and other car makers is simply to avoid having to do that. But, isn't that also true in software land? Can we affort to throw away a lot of work because the design has to change? No.
So what's the secret? Why is BMW pulling it off with BDUF and we poor software developers aren't? It's simple: BMW actually doesn't use BDUF. They use roughly this route:
- Defining the goals of what the car should meet: which category it should fit it (so they know their competitors and the target price), should it be a family car, sports car etc.
- Make a rough design within boundaries which are well known. This has nothing to do with details like the thickness of the steering wheel. It has everything to do with the areas they have previously experience with: an engine up front with front wheel drive gives different characteristics than an back wheel drive, etc. etc.
- Make a prototype. Often these prototypes are build with existing, proven parts, like the base of a different model. Subsystems which are new, are filled in with surrogates
- Test the prototype in a wind tunnel and with a computer model
- Build the subsystems they need and design the assembly lines
- Test the final version and tweak here and there some things, like car height etc.. This phase, it's impossible to change a lot of the car, as that would make a lot of things change with it, effectively going back to an early stage in development. In software, this is the same thing
Don't let the word prototype make you go blind and ditch this path as stupid or not applicable to software. The lesson to learn isn't in the word 'prototype'. The lesson to learn is in the second step and the third step: design within known boundaries and re-use proven parts. This isn't at the level of code, if we project this onto software development. It's at the level of functionality elements: if you need authentication, pick the authentication subsystem which worked the last time and use that and move on, as that's already solved. After you've decided that, you can check whether the implementation you have of that functionality still works or you have to refactor it. Though it's not at the level of 'I have this class, it will work'. That class represents functionality, that functionality should fit into the design. If not, don't use that class. A simple matter of what comes first and what follows.
I know, someone will come up with 'but... BMW is creating a product, we're creating software for a client who doesn't know what he wants/needs'. Think! You're not in a different position than BMW is. In fact, you're in a better position, because you know your client, your customer, you can even ask questions. BMW can't, they can only research what a potential customer might look for. Though if you think a person who buys a car is a person who knows what s/he wants and a person who buys software is a person who has no clue, you're wrong: both think they know what they want and both turn out to have no clue what they want. That's ok, that's why they hired you to analyse what they need and give that to them.
Yes, Software Engineering is hard, deal with it. Don't use the excuse that because goals are unknown, because the client changes his mind every day the design therefore has to be done in the code editor. If those are your problems, solve them, deal with them. Because after all, software isn't written for the sake of writing software, it's written to solve a problem, to make things easier, better handleable and controllable and understandable. What your client needs therefore should fit in that criteria and it's your job to figure out if that's the case and in what form, not your client's.