What is the value of comparing companies based upon development technology & methodologies?
I was recently talking to a company about their development team’s adoption of new technology, frameworks, and the overall sophistication of their projects. During the conversation, the IT manager made several comments about wanting to become a more “complete” development shop.
This made me start thinking, how do you measure and compare one development shop with another? What does it mean to become a “complete” development shop?
I hear a lot of attempts to characterize one company as being “behind”, “ahead of the curve”, or on the “bleeding edge” but often times each person’s assessment of each company is so subjective to their own experience and skill set. Furthermore, the types of technologies each company use and the business needs they serve often vary dramatically, therefore so do their goals. So its hard to truly know the real comparative value of one development team or company to another without segmenting them into categories and/or coming up with some standardized tools for measurement.
In the past I have passively explored the concepts around the Capability Maturity Model (CMM), which is a topic seems to really fit this discussion. CMM focuses upon the stages of growth every IT shop goes through during its evolution;
I think this is a good start in assessing an organizations “completeness”, however, it primarily focuses upon the process of software development not the technology, team-effectiveness, or rate-of-completion. CMM is incredibly useful, yet the aforementioned IT manager was really referring to all areas of their development. Yes, there was discussion of using Agile/Scrum and having a well built development environment using Team Foundation Server, but he also mentioned a number of other things such as what version of .NET they were using, the types of patterns & frameworks they had adopted, their ability to turn-on-a dime, and the relative skill level of developers implementing their applications.
Many of the details we discussed assumed an adoption of Microsoft’s development technologies, approaches, and general paradigm. So, on the one hand, you could easily take this methodology/technology stack and create a specific set of measurements for how the stages of growth of each company and compare on this bases. This would be good if all development shops thought the Microsoft Solutions Framework (MSF) was the ultimate goal and end-point of an organization, but the reality is that many do not.
Every shop is different, and many value ALT.NET technologies such as Mono, MVC, NHibernate, NUnit, Rhino Mocks, Subversion, Cruise Control, and other non-Microsoft tools much higher than those within the Microsoft development paradigm. Of course, there are also many shades of grey out there in the industry between MSF and ALT.NET. So, this brings us back to attempting to group together companies based upon their chosen, or trying to compare apples and oranges based upon other more platform agnostic factors such as flexibility, velocity, and quality.
But, at some point, I have to stop and ask myself “is this effort even important?”.
I know that its a great source of pride for some companies to say they are the “best and brightest” or are the “movers and shakers” in the industry, and undoubtedly if wielded properly, it can help enhance their recruiting of the best and brightest in the industry.
However, are there other values to a company beyond recruiting?
I guess it depends upon the type of company. For example; if you are a consulting firm or a product development company its a huge asset to sales of your products and services to customers who might value such a comparison of technologies & proficiencies.
But, what about traditional enterprise business where their development effort is not to deliver a product or service directly via technology, but instead they sell some tangible widget and simply use technology to perform internal logistics, supply chain, or customer-service functions? What is the value within these companies to using the newest technologies? What is the true ROI of continuing to adopt the latest & greatest technologies in these companies as compared to some other company who decided long ago that COBOL or FoxPro were sufficient for their business?
I think there is plenty of anecdotal evidence and case studies of how certain new technology and processes have been effective at reducing time-to-market, or producing greater results. But where are the numbers to qualify those examples?
Rather than just assuming newer technology is automatically better, I would like to see a more quantitative set of tools to measure these results and share them with the world so those companies who truly shine and benefit from these processes & technologies and shine the way for others in an objective and meaningful way.