Metrics in software and physics
Every so often, somebody points out how bad of a metric code coverage is. And of course, on its own, it doesn’t tell you much: after all, it’s a single number. How could it possibly reflect all the subtlety (or lack thereof) of your designs and of your testing artillery? Of course, within all the various *DD approaches, some better than others enable you to know whether or not your code conforms to its requirements, but I thought I’d take a moment to reflect on the general idea of a software metric and how it relates to the mothers of all metrics: physical ones, cause you know, I used to be a scientist. Proof: the lab coat on the picture.
The theory of measurement is at the center of all experimental physics. This comes from the realization that any observation of the natural world is ultimately indirect.
For example, when you look at a red ball, you don’t directly perceive it. Rather, photons hit it, some of them are absorbed by the surface of the ball (violet, blue, green and yellow ones, but not red ones) and some of them bounce back (the red ones if you’ve been following). Those red photons that bounced back then hit your eyes, where a lens distorts their paths so that all those photons that came from a specific point on the ball converge to roughly the same spot on your retina. Then, the photoreceptor cells on the retina transform the light signal into electric impulses in your optic nerve, which conveys all that information into your brain and then, only then the complex mechanisms of conscience give you the wonderful illusion of seeing a red ball in front of your eyes.
The brain reconstructs a model of the universe, but what it really ever perceives is a pattern of electric impulses. Everything in between is a rather elaborate Rube-Goldberg contraption that can be trusted most of the time but that is actually rather easy to fool. That it can be fooled at all is the simple consequence that what you observe is an indirect and partial measure of reality rather than reality itself.
When we measure anything in physics, we build our own devices that transform objective reality into perceivable quantities. For example, when physicists say they have “seen” a planet around a faraway sun, they don’t (always) mean that they put their eyes on the smaller end of a telescope and perceived the shape of that planet with their own eyes like I saw the red ball of the previous paragraph. No, what they saw is something like this on a computer monitor:
This shows the very small (1.5%) variation of the light coming from the star as the planet transits in front of it. All this really tells them is that something dark that takes about 1.5% of the area of the star passed in front of it. By repeating that observation, they can see that it happens every 3.5 days. That’s it. No image, just measures of the amount of light coming out of a powerful telescope aimed at a star against time.
But just from that minimal data and our centuries old knowledge of celestial mechanics, researchers were able to deduce that a planet 1.27 times the size of Jupiter but 0.63 times its mass and a surface gravity about the same as Earth’s was orbiting that star. That’s an impressively precise description of a big ball of gas that is 150 light years away (that’s 1.4 million billion kilometers in case you’re wondering or 880 thousand billion miles if you insist on using an archaic unit system).
The Rube Goldberg device that enables us to see that big ball of gas from so far away is a mix of optics, electronics and knowledge, the latter being the really awesome part. Science is awesome. The bottom line of all this is that although it seems less “direct” than seeing the red ball with our own eyes, it does just as well deserve to be described as “seeing” it. The only difference is that we’re not seeing with our eyes but more with our brains. How awesome is that?
Where was I?
Yes, you might be wondering what this has to do with software. Well, all that long digression was to show that little data is necessary to infer a lot about the object you’re observing. So code coverage? Sure, it’s just a number, but combined with a few other numbers, it can help get a reliable picture of software quality.
Another point I’d like to make is that a lot of resistance to software metrics comes from the illusion that we know a lot more about our own code than any tool can tell us. But as anyone who has ever tried to read code he wrote only five years ago knows, that is delusional. What you know about your code is a combination of what you remember and what you intended to write, neither of which is particularly reliably representative of what your code is doing. Tools give us a much more reliable picture. Sure, it’s a narrow projection of the code and it doesn’t capture its full reality, but that is exactly the point of a measure: to project a complex object along a scale of our choosing. What set of projections you choose to make is what determines their relevance.
The conclusion of all this is that we should assume that our code is an unknown object that needs to be measured, like that big ball of gas 150 light years away, if we want to get an objective idea of its quality without having our judgment clouded by our own assumptions.
And probably the best tool you can use to do exactly this by the way is NDepend by Patrick Smacchia.