Goodhart and Good Hearts

Defining metrics and measuring your results is essential to sustained success. But actively trying to optimise these metrics is a path that leads to ruin.

What are you trying to do?

If your answer to this question is “increase conversions” or “reduce latency” or “double sales” or any other description that relates to changing a number without any other qualifier, then you are likely not creating value in the world, you are extracting value from the world. I say that with caution, perhaps you are still creating some value, but if your focus is only on numbers then it’s by accident. Sorry to say.

In order for us, as humans, to ensure we are creating value for the world rather than extracting value from it, we must have a goal that goes beyond numbers. In order for us to have good hearts, we must understand Goodhart’s law.

What is Goodhart’s Law?

Well, you can read about all the nuance here, but I’m going to choose the most common definition:

When a measure becomes a target, it ceases to be a good measure.

What this means in more commonly used business terms is that as soon as you try to optimise a metric toward a numerical target, it loses its value as a metric. In his context, he’s talking about designing a government’s monetary policy. But I think this law is a little more fundamental.

What it means for humanity, I think, is that optimising numerical values is a sure path to poor outcomes and unintended consequences. And I’m not the only one to think this way - if you read Goodhart’s Wikipedia article, you will find mention of Jerome Ravetz’s 1971 book Scientific Knowledge and Its Social Problems, in which he explores the essential limitations of science for delivering value within the context of 20th century human society. The traditional philosophies of science that hold the process as the pursuit of truth no longer apply in the modern world, where science is “industrialised” in the service of non-scientific goals, and grants have become the goal, rather than any measure of quality or value of the inquiry at hand.

Most importantly, I think, is the choosing of values, without which no amount of science can be applied in service of humanity. Science, and more generally numbers, can help you understand what and how, but they can’t tell you good and bad - that is a moral evaluation, or value judgement. This might seem obvious - but if it is so obvious then why do so many of us obsess over numbers in business without a clear moral understanding of what those numbers mean?

Gabe Newell, co-founder of Valve Corporation, recently spoke about the negative aspects of startup culture and pitching to VCs in the tech industry. Think what you like, but I think he’s pointing at the same issue, where the goal seems to have become more about securing funding, rather than, to paraphrase and editorialise him, more immeasurable goals such as customer, partner or employee happiness. In this way we seek to extract value from VCs, rather than create value for customers, partners or employees¹.

I would extend Goodhart’s Law into the Law of Good Hearts:

Optimising numbers alone can only extract value from the world; in order to create value you must have goals and values that are fundamentally immeasurable.

Alright, enough philosophising, let’s go back to one of those bad examples I mentioned in the introduction and I’ll show you what I mean - so, what are you trying to do?

Increase conversions

Ok, so you’re trying to get more people to buy, signup, whatever. But there are many ways to do this, how do you choose? You could reduce the price by paying your vendors less. You could shove a signup wall on every page of your website. You could implement dark patterns that trick people into clicking on things they didn’t really want to. But what are you trying to do?

An example of a Good Hearted approach to this might be:

I am trying to improve people’s lives by simplifying the process of booking a holiday.

If I am successful in that goal, more people will signup and buy holidays through my service.

We have a goal first, and then we defined metrics second, as a way to tell us if we’re getting closer or further away from that goal.

So the goal is to do something of moral value - by that I mean it is something you have decided to try and change in the world. It is entirely subjective, and it lives solely within the realm of immeasurable human experience.

That said, there are some great metrics you could be keeping an eye on to help you achieve success in that goal. Number of signups is obvious, perhaps time or money saved by your clients, number of five star reviews, a growing list of tourism vendors willing to work with you, and so on.

As long as your goal remains to make some moral, human change in the world then you are on the right track to create real value. If your goal slips down into just trying to increase signups, or drive down prices, or whatever numerical thing, then you will lose your way very quickly. More likely than not you’ll simply be extracting value from the world, rather than creating it.

Perverse Incentives

The problem of optimising numerical values is tightly related to perverse inventives. You can read about this all day, there’s so many documented examples. I’m going to cherry pick one to illustrate the concept.

Standardised test scores are arguably a pretty good indication of how well students are acquiring their education. It might seem natural to want to increase these scores, somehow. But this is where we fall into the trap - if the metric of test scores becomes a target, it ceases to be a good measure of education. Indeed, it would seem that trying to increase these test scores directly through incentives seems to have resulted in driving teachers to cheat, meaning these numbers are no longer a good measure of education.

This might work in some sense - the numbers go up, politicians look good, teachers get rewarded, kids get better placement at universities, more and better paying jobs, etc. But, as we can see, this is people extracting value from the world through artificially inflated test scores, not creating it through better education.

In order to abide by our Law of Good Hearts, the focus must remain on the higher, immeasurable goal of “education”. We can imagine any number of ways to influence this: more funding for schools, more teachers, more training, etc. Many of these methods will create genuine value in the world by improving education for students, and the standardised test scores can still be used as a way to measure that impact².

If we think of our goal as making a car go faster, then these value creating ideas are putting our foot on the gas pedal / accelerator, whereas the perverse incentives are simply grabbing the speedometer and turning it to the right. The speedometer is designed to measure our speed - if we directly try to manipulate it, it no longer tells us anything useful about how fast we’re going.

Money

One of the key ideas of capitalism is that competitive markets lead to innovation that then drives economic growth through improved products or reduced cost. If you are making the best product or delivering the best service then people will give you their money in exchange for it, and you will become rich. This system turns self-interest into innovation and improved quality of life, right? Capitalism creates value… right?

The problem is that money is the metric. In the ideal, money is the measure of value creation. But when that metric, money, becomes a target (which it always does) then, as per our law, it ceases to be a good measure.

If your goal is to maximise profit, then you might make sure your products don’t last - because otherwise people wouldn’t need to buy as much. Planned obsolescence is a very straightforward demonstration of this principle, but if you think on this for a while I’m sure you will begin to see it in everything - big studio films, AAA video games, streaming services, anything where the organisation’s goal is simply to maximise shareholder value then there is no real need to innovate. It becomes a race to get things out as fast as possible, dress them up in the trailers and marketing materials, sell tickets, move units. What, it’s boring; incomplete; a joyless cashgrab? Who cares - no refunds - money go up! This is why consumer protection laws exist.

As with the other examples, the goal of competition in capitalism is to drive innovation and economic growth; to create value. But this goal is readily undermined by taking the way we measure that goal and turning it into a target. Instead of money being representative of our success in achieving our goal, it’s become the material of success, and now our purpose is to extract as much of that value as possible.

So, how do we have a Good Heart when it comes to money? We set a clear goal to see some change in the world, and if we’re creating value with that change we’ll get paid. But the focus must be on the goal, and not on the money. It’s not something we can ignore, money stagnating or falling is telling us something important, but if the goal of your organisation’s next venture is simply to “double growth” or “10x revenue” or “make more sales than last year”, without that being a way simply to track your progress against some higher mission, then your goal has become to extract value, not create it.

If money becomes your target, then it’s no longer a good measure of the value you create or the work you do. You’re more likely to be a parasite than an entrepreneur, and probably far worse than any dole bludger.

What can we do with this information?

When designing organisations and teams, when choosing missions and metrics, ensure you have a clear goal - one that is not directly measurable and is intended to create a positive value change in the world. Derive your metrics from that - it would be silly to fly blind and hope you were doing what you wanted to do with no way of understanding if you’re getting closer or further away. But ensure that always the goal remains the goal, and the metrics remain measures, only.

Remember - to make the car go faster you manipulate the pedal, not the speedometer.

It’s hard to feel sorry for the poor VCs when their goal is also (usually) to extract money from your work. ↩
The point here is not that standardised test scores are a perfect measure of education, or even a good one - just that they are one that can be used if we’re careful not to try and manipulate them directly. ↩