On the Future of Software Development #2 – We are not so special

Let me start my reasoning about how the people-side of the future of software development might look like with two perceptions that form the basis of my argumentation.

Perception 1: Software Development is not that special

I guess we all know those storys of exceptional programmers living on just jolt coke and pizza while developing single handedly a new operating system :-) And we all now the proverbial “diva status” some programmers claim. And we all know how “ordinary people” (our customers) complain about the often introverted natures of programmers and their underdeveloped communication skills.

Well, as always there is some truth to such kind of storys. And we can speculate where the reasons lie, what maybe makes dealing with computers so attractive for introverted personalities. But I think much more important is to recognize the underlying attitude of our trade that´s still nurturing these kind of perceptions: Software developers still really think they are something quite special.

Remember Donald Knuth´s “The Art of Programming”? Although this book probably sits more on the shelf than it is studied, its title summarizes how generations of programmers view themselves: as artists. In their view software development is an art, a craft, something that is close to composing music. In the center of this picture is the programmer (possibly sitting alone in his cubicle) sculpting an ingenious algorithm out of the clay of the formal language of his choice.

Well, a nice picture and I sometime indulge in feeling like that when pondering the intricacies of parallel programming or the like. But it´s not reality! Maybe it has been like that 20 years ago, but we cannot hold up such an attitude any longer.

20 years ago a programmer or software developer or software engineer had been a highly trained professional in a narrow field of expertise: software development. This distinguished him from carpenters and aircraft engineers.

On the other hand, once highly trained in this trade, the programmer felt ready to tackle any and all problems. Regardless if he was asked to develop a compiler, a book keeping application or painting tool he never would have declined such a request. He felt competent to wield the state-of-the-art tools and materials to produce a satisfying result. He just needed complete requirements and enough time to wreck his brain – which was his most important tool.

Back than this might have been a valid attitude, but it got carried over to the present. And that´s plain wrong and makes it very difficult for developers today to produce high quality and get job satisfaction.

20 years ago software development was special in that it had few tools and materials and mostly relied on the creativity of single developers with deep knowledge in theoretical concepts (like “algorithms and data structures”).

Also software seemed and still seems to be so different from anything else humans produce. This adds to the trade´s attitude of being special and unique. Software is so abstract, so highly formal/structured, so ephemeral. Software developers were the digital guys in an analog world. Bits and Bytes made ordinary people gape in awe. Programmers were the heros able to tame the computer beast by wielding their formal language whip. Wow!

But, hey, let´s get real! Our world of 2005 is digital. Music is digital since we switched to CDs in the mid 1980s. Video has become digital. Telephoning is the last bastion about to fall when finally VoIP is taking off on the consumer market.

Computers in some form are everywhere and RFID will add to the pervasiveness. And kids get aquaintant with programming in elementary school. Software development has lost its nimbus of a secret art.

But how about our products? They are digital and can easily be mass produced. (Just copy that file.) And they are highly individual. And they are made on and for changing requirements and a long maintenance life. Isn´t that quite unique?

Well, I wouldn´t say so. Although analogies are a difficult thing and usually don´t explain everything, let me try some:

Are there no other products that need long maintenance? Sure, there are. Chemical plants, aircrafts, and even cars are build in a relevatively short time and need to be maintained over a very long time (up to decades). Or even take a city as example: You don´t build a city and then leave it to itself. Rather it is constantly changing, buildings are torn down, new ones are build, subway tunnels are dug etc. Some structures are more volatile than others. But over the decades or even the centuries a city changes massively without losing its basic character and while always fulfilling its main purpose: offer an infrastructure for the inhabitants. My conclusion: Mankind or engineering in general has a lot experience with long lasting structures that need to undergo quite some changes during their lifetime. Software is not special in that regard.

Are there no other products that are mass produced but always unique? Sure, there are. Take roller shutters for example. Each roller shutter usually is tailored to the exact measures of a window/opening of a house. Or – an analogy I even like more – take a movie as an example. Each movie is a unique product. Hollywood produces 400 movies each year, the Indian Bollywood even 800 each year. And despite its uniqueness (and often changing requirements; think of producers as divas ;-) it is an industrial product and a result of a coordinated effort of hundreds of individuals.

So I´d say, long lifetime with high maintenance and uniqueness are not really attributes of software development, which justify a special status or attitude. Also the easyness of multiplication is not unique. The music and video industry are facing the same problem. And print is struggling with is since the invention of photo copying machines.

But what about the ever changing requirements? The waterfall model failed for software development, but it works for civil engineering. Yes, the requirements seem to change much more often during the manufacturing of software compared for example to a movie. But does that make software development uncomparable to other manufacturing processes? I don´t think so. Because not the magnitude of volatility of requirements is crucial for a comparison, but whether the tools, materials, models, processes, education etc. match a given volatilty.

And therein lies the crux, I´d say! Don´t try to compare apples with pairs. Don´t try to compare a mature trade like “building skyscrapers” or “manufacturing cars” with a comparatively young (or immature trade). So, you should not take a rediculous comparison for a prove of uniqueness.

When comparing software development to some other industry/trade, first ask, if software development can match the other side´s maturity and in how far that could affect the outcome of the comparison. My opinion: Software development is still after some 50 years pretty immature – but not as unique as most people think. (The uniqueness of software development, though, might lie in its stubborn self assurance of its uniqueness. Remindes me a bit of autism.)

But, now, finally, what about the lack of natural laws in software development? Yes, that´s true. That poses a certain difficulty and accounts for some uniqueness. When planning a building or a CPU, the effects/stability of certain designs can be assessed by using mathematical formulas or by simulation based on natural laws (i.e. again mathematics). Before even digging the smallest hole you know if a bridge will be able to carry the trains and cars rolling over it later. That´s because you know the exact parameters of the ground it is build on, the stones, the steel, the concrete etc. which are going to be used in building it. For software, though, you cannot just compute its stability or correctness or quality. Contrary to most other trades you need to constantly monitor and actively asses it (that´s called “testing”) or model scenarios when in doubt (that´s called “prototyping”). And since each software is unique, you are never done with testing.

But does this uniqueness justify the software development´s trade attitude of “Oh, boy, are we special!”? No, I doubt it. It is not so important how we determine correctness of our products. It is important to have a mature methodology for doing so that is adequate for our tools and materials. And likewise it is not so important, that software development differs so much and lacks “mathematical precision”. We must not feel inferior because of that. We just do it differently. Correctness is determined in different ways when building skyscrapers, finding a cure for a disease, coming to a judgement in a trial. So why can´t we be different – and still be comparable to other trades?

So when comparing software development to other trades the question is not, whether we produce similar products. Rather we should ask, do the processes differ fundamentally? We should look at mature and established industry and see what they consist of: there are tools, materials, process phases (e.g. planning, manufacturing, maintenance), division of labor, quality assurance, professions, education etc.

Then we should compare our findings with what we have in software development. And we can only realize: all of this is present in our trade. So software development does not fundamentally differ from other industries!

But if we don´t differ fundamentally, why do we have a hard time liking software development to other trades? Because the level of maturity of our tools, materials, process phases, division of labor, quality assurance, professions, education is so different from their´s. And the level of maturity does not even always match the complexity of the problems to solve.

Don´t get me wrong, there are good tools at our hands and there are excellent programmers. But still we can and should learn a lot from other much more mature industries. UML is an approach to learn documentation and visual modelling from civil engineering or electrical engineering or processor design. Software Factories now is an approach to learn from mass manufacturers about tool specialization and manufacturing. Now my question is: What can we learn from other industries in terms of process organization, education, quality assurance, divison of labor?

And if our trade is not unique, neither is our profession as software developers. We should not deem ourselves as artists or divas or very special in general. We are ordinary hard working people in a demanding business with some very specific challenges. Not more, not less.

The only thing that maybe makes us special is, we need to deal with a lot more change than other professions. But that will be the topic of the next installment of this series.