Metrics in software and physics

A Horrible experiment Every so often, somebody points out how bad of a metric code coverage is. And of course, on its own, it doesn’t tell you much: after all, it’s a single number. How could it possibly reflect all the subtlety (or lack thereof) of your designs and of your testing artillery? Of course, within all the various *DD approaches, some better than others enable you to know whether or not your code conforms to its requirements, but I thought I’d take a moment to reflect on the general idea of a software metric and how it relates to the mothers of all metrics: physical ones, cause you know, I used to be a scientist. Proof: the lab coat on the picture.

The theory of measurement is at the center of all experimental physics. This comes from the realization that any observation of the natural world is ultimately indirect.

For example, when you look at a red ball, you don’t directly perceive it. Rather, photons hit it, some of them are absorbed by the surface of the ball (violet, blue, green and yellow ones, but not red ones) and some of them bounce back (the red ones if you’ve been following). Those red photons that bounced back then hit your eyes, where a lens distorts their paths so that all those photons that came from a specific point on the ball converge to roughly the same spot on your retina. Then, the photoreceptor cells on the retina transform the light signal into electric impulses in your optic nerve, which conveys all that information into your brain and then, only then the complex mechanisms of conscience give you the wonderful illusion of seeing a red ball in front of your eyes.

The brain reconstructs a model of the universe, but what it really ever perceives is a pattern of electric impulses. Everything in between is a rather elaborate Rube-Goldberg contraption that can be trusted most of the time but that is actually rather easy to fool. That it can be fooled at all is the simple consequence that what you observe is an indirect and partial measure of reality rather than reality itself.

When we measure anything in physics, we build our own devices that transform objective reality into perceivable quantities. For example, when physicists say they have “seen” a planet around a faraway sun, they don’t (always) mean that they put their eyes on the smaller end of a telescope and perceived the shape of that planet with their own eyes like I saw the red ball of the previous paragraph. No, what they saw is something like this on a computer monitor:What a beautiful planet!This shows the very small (1.5%) variation of the light coming from the star as the planet transits in front of it. All this really tells them is that something dark that takes about 1.5% of the area of the star passed in front of it. By repeating that observation, they can see that it happens every 3.5 days. That’s it. No image, just measures of the amount of light coming out of a powerful telescope aimed at a star against time.

But just from that minimal data and our centuries old knowledge of celestial mechanics, researchers were able to deduce that a planet 1.27 times the size of Jupiter but 0.63 times its mass and a surface gravity about the same as Earth’s was orbiting that star. That’s an impressively precise description of a big ball of gas that is 150 light years away (that’s 1.4 million billion kilometers in case you’re wondering or 880 thousand billion miles if you insist on using an archaic unit system).

The Rube Goldberg device that enables us to see that big ball of gas from so far away is a mix of optics, electronics and knowledge, the latter being the really awesome part. Science is awesome. The bottom line of all this is that although it seems less “direct” than seeing the red ball with our own eyes, it does just as well deserve to be described as “seeing” it. The only difference is that we’re not seeing with our eyes but more with our brains. How awesome is that?

Where was I?

Yes, you might be wondering what this has to do with software. Well, all that long digression was to show that little data is necessary to infer a lot about the object you’re observing. So code coverage? Sure, it’s just a number, but combined with a few other numbers, it can help get a reliable picture of software quality.

Another point I’d like to make is that a lot of resistance to software metrics comes from the illusion that we know a lot more about our own code than any tool can tell us. But as anyone who has ever tried to read code he wrote only five years ago knows, that is delusional. What you know about your code is a combination of what you remember and what you intended to write, neither of which is particularly reliably representative of what your code is doing. Tools give us a much more reliable picture. Sure, it’s a narrow projection of the code and it doesn’t capture its full reality, but that is exactly the point of a measure: to project a complex object along a scale of our choosing. What set of projections you choose to make is what determines their relevance.

The conclusion of all this is that we should assume that our code is an unknown object that needs to be measured, like that big ball of gas 150 light years away, if we want to get an objective idea of its quality without having our judgment clouded by our own assumptions.

And probably the best tool you can use to do exactly this by the way is NDepend by Patrick Smacchia.


  • Excellent post Bertrand!

  • One heck of an article for an early Saturady morning. You made my brain function way before it usally does on the weekend. I especially relate to the lst few paragraphs, thanks

  • nicely said. I agree that tools can help reduce the overall maintainence costs one will eventually endure in the long run.

    Unfortunately the illusion you mention about one knowing the code they write better than any tool can is a tough challenge to overcome.

    Also the learning curve of yet another tool to master is a demotivator and will most certainly hold you back. Thanks for the tip. NDepend indeed looks good. I'll keep a look out for it.

  • Nice post! It kinda sounds like a bait and switch advertising for NDepend at the end there but that made me check it out...

  • That's the longest, most scientific, and graph-adorned info-mercial I've read in a long time, leading up to ... a sales pitch for a commercial product.

    The more I see on the internet, the more it looks like paid programming on TV. At least TV sets have a "brightness" control. The internet just keeps getting dumber and there's no knob to make it brighter.

    Now, isn't the government taking action against paid bloggers, Bernie????

  • @Billy, that's a serious accusation you're making here. No, I'm not getting paid to write this. Not by Patrick at least.

  • Excellent article! An exquisite representation of applied physics!

  • It is so annoying when I see science taking credit for good old engineering. Our God is awesome, they say. (Well, they say science, but there's really no difference in their minds.)

    Oh well. This too shall pass.

    (Oh, and the idea of a planet revolving around its Sun every 3.5 days is... mind-boggling. I'm pretty sure it would have to be within the Sun corona, which leads to all kinds of contradictions.)

  • @Marcel: WHAT??

  • "The Rube Goldberg device that enables us to see that big ball of gas from so far away is a mix of optics, electronics and knowledge, the latter being the really awesome part. Science is awesome."

    The part that actually works in that Rube Goldberg device is old-fashioned engineering. The part that invents planets revolving in a Sun's corona is science. Science is stupid, usually - it proves that rocks can't fall from the sky, airplanes can't fly and trains going faster than 30 mph will kill their passengers.

    That doesn't mean I'm not with you on NDepend, btw :P

  • @Marcel: not at all, don't assume I'm going to ignore what you have to offer. It would go against who I am. I care a lot about this sort of thing. I think this debate is relevant to this post so it's ok to have it here.
    Some people may be confused about what science really is, but that only tells you something about these people, not about science itself.
    I, and all real scientists, treat scientific methods for no more and no less than what they are: methods to find out stuff about the natural world. It is always clear that any knowledge found using these methods is only valid within a margin of error (which can be quantified) and within a set of assumptions. So actually no, I don't know any scientist who treats science as God.
    Anyone can challenge any assumption and provide a better explanation for any new or existing phenomenon, but there are conditions that proponents of pseudo-science usually forget: the new explanation has to work better than the previous one in that it needs to explain more observations or to explain existing ones with better precision or in a simpler way. It also has to be falsifiable which is oh so important and so often overlooked by enemies of science.
    So there is no dogma in science, nothing that can't be challenged, as long as you do so within the boundaries of scientific methods. If you want to challenge a scientific idea *outside* of the boundaries of scientific methods by the way, that is perfectly fine. What you can't do is call it science or expect the scientific community to adopt it. Or teach it in science class.
    The current consensus is challenged all the time with great enthusiasm: Darwin challenged the consensus, as did Gallileo, Newton, Einstein, Heisenberg, Dirac, de Broglie, Zurek and so many others. In my own time and without the genius of those, I published a few articles in scientific journals that were proposing a different way of understanding quark confinement that was going against the relatively complex current consensus which relies on quantum chromodynamics. It may not have been very successful as far as scientific ideas go, but I can tell you I didn't face any dogmatic opposition from any scientific "establishment" because of it. Getting those paper published in major peer reviewed journals was absolutely easy.

    Sooo, can you please provide specifics:
    - "patent absurdity of some of [science's] constructs": what constructs?
    - "[science's] failure to fulfill many of its extravagant promises of health and life": what promises?
    - "the tolerance of the scientific community for unsubstantiated just-so stories": what stories?

    And for the punchline: "we are forced by our a priori adherence to material causes to create an apparatus of investigation and a set of concepts that produce material explanations, no matter how counter-intuitive, no matter how mystifying to the uninitiated". Well, nobody is forcing you, but what Lewontin is omitting to mention is the predictive power and the unparalleled precision of scientific results. We don't just accept a scientific discovery because it's flattering to whatever view of the world we have (scientists actually have all sorts of those when they don't do science). We accept it because it matches verifiable observations.

    Also, you haven't answered my previous questions.

  • Coincidentally, Ars has an excellent article today on how scientific consensus emerges:

  • "Coincidentally, Ars has an excellent article today on how scientific consensus emerges:"

    Yep, "lies for children" really sums it up :)

  • Just found some fun ammo for you, @Marcel ;)

  • Heh, cool, thanks :)

  • Just to check... you did receive my *previous* post - the huge one :) right? I ask 'cause I posted it before you wrote the "some ammo" post above, and it hasn't shown up. (If you decided to stop the discussion and delete it, that's fine, I just want to make sure it wasn't stopped by some spam filter or something.)

  • @Marcel: mmh, no I did not. That sucks. The spam filter has done that in the past, so I'll try to find it... You wouldn't have a copy by any chance?Found it. It was marked as spam.

  • Hmm... something weird is happening when I try to post an answer (I get redirected to the home page). I can email it to you, or we can call it a draw :)

  • @Marcel: feel free to send me your comment in e-mail (bleroy you know where). I have no idea what happened here: redirection to home is a new one.

  • @Marcel: that is truly epic! Thank you so much for that link.

Comments have been disabled for this content.