Why IT really needs an Archibald Cochrane

Introduction

Software development has a number of open and unresolved questions: What is the best programming language to use for any given scenario?  Should I use an object-oreinted or functional approach?  Do coding standards make software easier and cheaper to maintain?

Most people who have worked for any length of time in IT would have an intuitive personal opinion on these and many other matters.  However if we have to justify these opinions we are unable to refer back to any unbiased source of "proof" which - I would suggest - is harmful to the development¹ of our profession.

Background

In 1972 a doctor by the name of Archibald Cochrane published a book with the innocuous title of "Effectiveness and Efficiency: Random Reflections on Health Services".  In it he argued for controlled randomized trials to discover, scientifically and incontovertibly, what did and did not work in medecine.

At that time medecine was in a very similar place to where software development is today - procedures were largely based on the opinion of long-standing experts in the field who passed on their expertise and preconception through direct mentoring of those on the starting rungs of the ladder. 

Does this matter?

Medicine is a more important field than software development - it is often literally a matter of life and death.  However that does not mean that software development is too unimportant to be subjected to this type of process.  Software development is a very large and ever growing factor in the economy and is becoming a significant component of safety critical systems. 

Methodology

We are not going to be able to set up randomized trials for any of our big questions - companies will not pay us to "experiment" while we work and the open source community has more than enough work to do creating great software for free to undertake this as well.  In addition, any such work would suffer from the tragedy of the commons in that everyone would stand to gain from this knowledge whether or not they contribute to finding it out.

Fortunately there is a source of information that can be mined retrospectively to give us some of this information: the various open source code repositories (GitHub, Codeplex, Google code etc.)

Step 1: Identify your proxy

Since we cannot directly ask a question like "Is java code more easily maintained than C#" we need to identify some proxy question that we can ask.  For example you might ask "For any given project how many developers who join more than a month after the start of the project are actively contributing code another month after that?"  This is not exactly equivalent but could serve as a proxy for our underlying question.

Step 2: Log your intended analysis

It is human nature that when we ask a question like the above we have an answer we would expect.  It is also human nature that if we do not get the answer we expect we (mostly) do not publicise this.  Therefore we need to know what analysis is intended to be performed before it actually is performed, so these negative results don't go missing.

We also need the methodology used to be repeatable and open.

Step 3: Perform the analysis

Most public source code repositories are searchable so the analysis can be performed by a number of searches and the results collated in table form.

Step 4: Publish the results

The data need to be shared for the common good, and this needs to be in a searchable manner.  Fortunately this is something that we already do very very well in IT.

Points of Interest

The seed from this idea is work (and writing) being done by Ben Goldacre, Tim Harford and Hans Rosling into the use of randomized trials in education and public policy. 

History

• 2014-10-29 Initial ideas