There’s long been a debate about the role of metrics in software development organizations. There was the old adage that went “If its important, then its important enough to measure.” More recently, one of the proponents of that mantra, Tom DeMarco has questioned his line of thinking, observing that “The book for me is a curious combination of generally true things written on every page but combined into an overall message that’s wrong” This can be found in its full glory at http://www2.computer.org/cms/Computer.org/ComputingNow/homepage/2009/0709/rW_SO_Viewpoints.pdf. It presents some interesting and very thoughtful perspectives.

I recently ran into another interesting article, referenced from the freakonomics blog, pointing to a (very loosely) related study of measurement and accountability in schools in England. In this article “A natural experiment in school accountability: the impact of school performance information on pupil progress and sorting”, (http://www.bristol.ac.uk/cmpo/publications/papers/2010/wp246.pdf) the authors observed that the ceasing of measurements and rankings of schools in Wales had a profound effect on the school systems’ effectiveness.

So, like many things in life, there are multiple sides to a story. In this case, I suspect that the right answer (at least as pertains to software organizations) is measure what’s important, and on a macro scale, as a sort of score card as a means to set goals. But maybe don’t get so carried away measuring the details that one completely redirects behaviors towards miniscule observations and end up with a badly unintended corollary to  Heisenberg (that measuring some things has unintended consequences rendering related measurements useless).

Perhaps the takeaway is that its best to hold groups accountable, but only to those things that truly matter.