The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures, and the more apt it will be to distort and corrupt the social processes it is intended to monitor.Campbell’s Law | Campbell, Donald T (1979). “Assessing the impact of planned social change”. Evaluation and Program Planning. 2 (1): 67–90. doi:10.1016/0149-7189(79)90048-X
Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.Goodhart’s Law | Goodhart, Charles (1981). “Problems of Monetary Management: The U.K. Experience“. In Courakis, Anthony S. (ed.). Inflation, Depression, and Economic Policy in the West. pp. 111–146. ISBN 978-0-389-20144-1.
Society is drowning in data. From businesses, to schools, to governmental policy, more data to make better decisions has become the widely accepted panacea. A quick look around both the institutions and the headlines will confirm the ever increasing bias towards metric driven future outcomes.
I remember back when I was first starting out in my career. I started working on Classification and Regression Tree software for one of my college professors. I was young and completely enthralled with the “power” of this early machine learning tool. It had enabled powerful decision making tools and was much better able to deal with situations where you had low signal, high noise, and minimal data as compared to the previous tools.
I was with a group that had created this awesome software and used by US Navy doctor Lee Goldman and my co-worker Richard Olshen to diagnose emergency room patients with acute chest pain. Their work would go on to be implemented at Cook County ER in Chicago in 1996. It would outperform all the “experts.”
In fact, the CART model won hands down:
- It was 70% better at recognizing false alarms than doctors
- It also predicted real cases 95% of the time, compared to doctors who were right between 75% and 89% of the time.
Even though I had moved on to NeXT Computer by then, I was still in touch with Dr. Steinberg, and I followed CART’s wins with great enthusiasm. It taught me a very important lesson that I didn’t fully absorb back then. I was too impressed with the computational power. I didn’t really absorb that the CART model’s success was tightly linked to the realization that more information does not necessarily lead to better decisions.
Psychologists and others have investigated this issue, testing how different levels of information affected the accuracy of a doctor’s diagnoses. Counterintuitively, what was found was that more information did, in fact, make doctors more confident in a diagnosis. On the surface, this seems like a good thing. The problem was that more information did not actually help them make better diagnoses. Doctors continued to exhibit only a 30% accuracy level when it comes to determining patient ailments. Thus, it was clear: more information does not necessarily lead to better outcomes.
Let’s be clear, we approach data and metrics with the best of intentions. There are a whole class of problems, mostly from the realm of Mediocristan, for which metrics can help with solutions. Or, at least the right data and metrics can contribute to solving problems.
Yet, too often ignored are the negative effects of metrics. Given the repetitive nature of our mistakes in the data/metrics department, I find it very curious that we don’t seem to be able to anticipate the recurrent flaws. Well most people don’t. I think Nassim Nicholas Taleb has a good grasp on things, and hopefully with his increasing popularity others will catch on as well.
The problem with real-world problems is that they are, by definition, complicated. When we set out to collect data to solve them, we too often fall into the trap of collecting data by measuring the data that is most easily measurable. We know this is a natural human tendency. We try to simplify problems by focusing on the most easily measurable inputs to assemble our models.
It is also true, as it turns out, what is most easily measured is not necessarily what is most important. Sometimes what we end up measuring is not important at all. This is probably the first leg of our data/metric dysfunction.
The second leg of our data/metric dysfunction is probably the oversimplification of complex problems into data models. Most real-world problems have way more interconnected factors that we can possibly measure. This measurement of a subset of the true factors often leads to deceptive results.
Think about the obsession we see in companies when it comes to trying to measure performance. Almost all jobs have multiple responsibilities and almost all organizations have multiple goals. However the permutation of sets of elements and the mathematical relations that characterize their properties is often too difficult to tackle. As a result, we are driven to collect data to feed metrics on just one responsibility or goal. This approach often leads to deceptive results. But we still do it.
Many times the metric driven folks in an organization will measure inputs rather than outcomes. The simple truth is that it is way easier to measure the amount of money spent or the total number resources consumed by a project, than it is to measure the results of the efforts. Organizations also often measure their processes rather than the product those processes produce.
The third leg of our data/metric disfunction is our quest for standardization. We’ve all seen how quantification can seduce us. It is seductive because it allows us to organize real world complexity into neat little buckets that our brains can process more easily. It alleviated our discomfort with the thought that something might not be truly knowable.
Numerical data makes for easy comparisons between and among people and organizations. However, to make data consumable, it must be simplified and reduced. The process of simplification and reduction strips the data points of their context, their history, and their original meaning. Now, this reduced information appears more certain and authoritative than could possibly be the case absent the original context. Lost to the other consumers of the data and metrics are all the original caveats, the ambiguities, and other uncertainties. Additionally, because people are suckers for data, nothing does more to create the appearance of certain knowledge than expressing it in numerical form.
But wait. If that’s not bad enough, enter the folks whose lives are being ruled by the sales results, the performance metrics, and the standardized test scores. Enter gaming and cheating to manipulate the data to drive the metrics in their favor.
More on that in another post …