xkcd #2899: Goodhart's Law
xkcd #2899: Goodhart's Law
Alt text:
[later] I'm pleased to report we're now identifying and replacing hundreds of outdated metrics per hour.
I’d not heard of this before, but this explains a lot of why my call center jobs were such BS.
We were expected to resolve networking, MS Exchange and VoIP issues in 20 minutes or less on average, which just resulted in a lot more customers having to call back because all the agents had to try and rush to a solution without time to test.
It's wildly under-taught. It explains like half of all problems in the world. Education: "teaching to the test." Economics: optimizing GDP at the expense of non-material well-being. Maximizing shareholder value by selling out employees and enshittifying your product. Software: "data-driven decision making" optimizing short -term gains over long-term because they are more measurable. That's just off the top of my head.
Yep. That, and cops arrest and ticket quotas…
We don't test for false convictions, which are as good as true ones for furthering careers in prosecution and law enforcement.
We don't know if our prison population is 10% innocent or 75% despite Blackstone's ratio.
In fact, when someone isn't successfully convicted, it's assumed the suspect got off on a technicality rather than continuing the investigation to find other suspects.
There's never time to do it right, but there's always time to do it over.
It's even better when these metrics explicitly become your yearly goals. Or department-wide metrics you have very little influence over. I sure hope all these people I don't manage happen to achieve a specific error rate this year so I get a good bonus.
God this is too real. One place I worked loved to pick odd company wide metrics too. Instead of just like "profit", 50% of our bonus would be determined by how low a % held stock was against revenue, globally... I worked in marketing. Needless to say the "bonus" did not motivate anyone.
What's even better is when your metrics are based on last year's metrics so you better achieve exponential growth or start looking for another job.
The year before last we achieved 1% test error rate in an area, and the bosses were seriously considering having the following year's goal be 0%. Someone had to point out that if anyone had 1 error on Jan 1, we literally couldn't do anything to achieve the goal the rest of the year and may as well give up entirely.
How to turn telemetry into OKRs.
Cute, but it really should be the black hat with the punch line.