On the Future of Software Development #3 – Feel the heat!

Ok, so now that any “attitude” is out of our way (see my previous posting for details), we can start to look at what is the real challenge for programmers today.

Perception 2: Software Development is changing at a much higher rate than 20 years ago

What makes a trade, an industry? It´s of course the products which it produces. Bakers produce bread, waiters produce a “dining experience”, directors produce movies, programmers produce software.

But to produce a quality product, skills, knowledge, tools, materials need to be of high quality. (Let´s leave methods and processes aside for the moment.) A master of his trade needs to have the rights tools and materials, be able to use them on a high level (skill), and be aquaintant with their capabilities/limitations (knowledge). Sounds like common sense, right? I´d say so.

Now, does our trade lack quality tools? No, I´d say, there are a lot of quality tools. A whole lot.

Or does it lack quality materials? No, there a lot of quality materials. A whole lot.

But what about skills and knowledge? Does our trade lack quality in this regard? Yes, I think so! And I think this is an inevitable outcome given how we view our trade. So I don´t really blame todays software developers – but if they don´t see the writing on the wall and don´t change, there will be no happy end for many of them.

We are like a frog sitting in a pot of water. If you heat up the water slowly, the frog won´t realize it and gets boiled alive. (But if you put a frog into hot water, it jumps out immediately.)

Like the frog we are in a system that was good to live in 20 years ago. But since then it has changed – but so slowly we are not conscious of that change. Maybe we just feel some diffuse pain and can´t lay our finger on it.

So what is the change in our system? Where´s the heat? The heat is in the faster and faster movement of our tools and materials. And in addition to that, the products we are requested to manufacture become more and more complex – with the tools lagging behind in their abstraction of this complexity.

20 years ago a master programmer´s toolchest just contained a couple of tools: a formal language or two, a very simple IDE (or just a text editor like vi), a debugger, … what else? Ok, on Unix systems there were those lovely command line tools like sed, awk etc. Great! But still a pretty small number all in all. Also, there was a strong agreement on what was the “canon” of tools you should now. This was fine, because programming was still an “art” and programmers where the universal heroes for solving any software problem.

What about the materials 20 years ago? Hm… there was the C library. You needed some API for character based UI, a data access API (what was your favorite ISAM library?), maybe some printing API? What else?

Or your world was even smaller: you just used Cobol or a 4GL like Zim or Amber.

Let´s face it, your choice of tools and materials was pretty small.

However today, there is no limit to the number of tools available. Even the number of tool categories is expanding all the time! 10 years ago there were no Application Servers. 5 years ago there were no Business Process Management Servers. So how do you keep up with the acronym (and code name) avalanche coming down upon you? What´s LCE? What´s QC? What´s STA? What´s MOM? What´s SOA? (I know, that´s an easy one ;-) What´s SOAP? What´s XQuery? What´s BEPL? What´s VSTO? What´s ORM? What´s MARS? What´s XAML? What´s GPL? What´s IRS? (Upps, wrong trade ;-)

Ok, I guess you got it. And I even left out the non-acronym technologies and code names. And I left out the many “instanciations” of concepts/technologies. There is not one data grid control, but many. There is not one installer tool, but many. All with subtle (or not so subtle) differences. And ever changing. A next release is always upcoming.

The rate of change and well as the breath of changes as well as the complexity of technologies has increased so much in the past 20 years (and especially in the past 10 years), that we cannot possibly think, we can cope with it without changing ourselves.

To make it even clearer: We still work around 40-60 hours a week like 20 years ago. And this is not gonna change much in the future ;-) But within the same amount of time, we now need to keep track of maybe 5 to 10 times as many tools and materials plus the increased complexity of the technologies and our products.

Is that possible if it was a specialist´s, a hero´s job 10 or 20 years ago? No!

Now, you might argue, the tools are so much more productive nowadays. To do a Windows UI with C took a long time, to do it with VB1 was a snap. Sure, of course some tasks have become easier through improved tools. Garbage collection is a godsend! And ServicedComponents are much easier to host in COM+ than ATL/VB6 COM-Objects. But my feeling is, the productivity advancements do not compensate for the overall rate of change. They just mitigate it a bit. Or new developments even add to it: Earlier I had a crude printing API to use for all my printing tasks. Today I can choose between several reporting engines, XSLT-FO, and PDF.

To come back to quality products as the goal of a trade: I think, we are at a turning point in our industry. Given our in general unchanged attitude of 20 years ago (“Once a programmer you can produce any piece of software!”) we will be less and less able to produce good quality. Because what quality needs is not just quality tools – which we certainly have (see list above) -, but skill and knowledge.

Skill and knowledge require time. Time to acquire knowledge, how tools work, what the capabilities and constraints of materials are, and time to hone your skills by actually using tools and materials.

Unless you actually build up experience in using tools/materials, you cannot wield/process them like a master, and you cannot even choose the best tool/material for a given task. Just reading an article or two about LCE does not (!) make you a master. Just seeing a presentation of Notification Services does not make you a master. But only a master can decide for a project, which technology to use, which not to use, and how to use the chosen one.

How should a host of “general purpose developers” (GPD ;-) using mainly general purpose programming languages (GPL) be able to master that many more tools/materials to produce even higher quality software? (I hope we agree, that software quality in general is not really good – at least not for version 1.)

The answer: Not – at – all! It is impossible!

So what should we do? Well, that´s the topic of my next installment.