
The danger of surrogate metrics (2007) - Jtsummers
https://blogs.msdn.microsoft.com/ericgu/2007/05/10/the-danger-of-surrogate-metrics/
======
Normal_gaussian
I observed a (non-engineering) team successfully use this argument to avoid
being measured at all except through the customer feedback that they were a
primary channel for.

As such, they were oft quoted as being amazing. I can't say that they weren't
- my interactions with them were tangential to their main purpose - but the
work that my team had that depended on them or their input was often not done
or missed the point completely (and criticism dismissed as they are the
posterchild team).

I think you _should_ use surrogate metrics. You should just know them for what
they are, and be able to argue as to why they aren't representative when they
are bad and when they are good.

This means you can't make somebodies job, bonus, or promotion depend on the
metric.

But it means you do get a heads up when something might be going wrong. And it
means you get a heads up when something might be going right.

It also means you have to deeply understand what a metric really is, and what
each particular metric is to the mission of your company and your job. This is
hard.

~~~
abhayb
Yes! I've seen these called guardrail metrics. Having a single metric be the
one to optimize works pretty well if that optimization is what you want. But
optimizing a teams work is messy and generates organizational instabilities.
Tuning the guardrails feels like the best way to make sure that you get enough
instability for interesting things to happen but not enough that other teams
start deciding to take over whole functions just to get work done.

------
sokoloff
See also Goodhart’s Law: When a measure becomes a target, it ceases to be a
good measure.

[https://en.m.wikipedia.org/wiki/Goodhart%27s_law](https://en.m.wikipedia.org/wiki/Goodhart%27s_law)

------
vmateixeira
I know this is from 2007, but it's funny this is coming from Microsoft... with
all the W10 metrics, telemetry and spyware bloat that replaced the old and
effective "Send error report" button. Kind of seems like they didn't follow
their own advice - provided user satisfaction is actually what's on the line
here.

------
hprotagonist
In the bio/pharma scene, this is most cruelly illustrated (in recent times)
with this example failure:
[https://blogs.sciencemag.org/pipeline/archives/2017/05/18/no...](https://blogs.sciencemag.org/pipeline/archives/2017/05/18/now-
this-is-a-drug-that-does-not-work)

tldr:

\- statins have a measurable effect on HDL and LDL levels that correlates
really well with how much they reduce your risk of dying, and that risk
reduction is real and big.

\- therefore HDL/LDL levels shall be a great surrogate metric for new drugs;
let's go find some.

\- we found this great drug that does even better than statins do at HDL/LDL
level changes

\- two years and 12k patients in trials later: this new drug doesn't affect
mortality literally at all.

\- ...crap.

------
Ozzie_osman
Every consumer internet company needs to heed this warning. Instead of digging
deeper and deeper into metrics like "time spent in feed" or "upvotes per
session" or "minutely active users" (because daily and monthly active users is
so 2015)... It's pretty simple. Find a way to ask your users what they
actually think of your product (then, yeah, by all means use data to
validate/explain that)

~~~
cortesoft
But in the end, isn't "what a customer thinks about your product" also a
surrogate metric? There are a lot of products that were loved by the users
that still failed to be a sustainable business.

If you give a valuable product away for free, for example, your users will
love you.... but it you will soon run out of money and go out of business if
you don't also have a way to generate revenue.

~~~
a1369209993
"What a customer thinks about your product" is a _input_ , not a metric. You
can still fail, just like a chair factory can fail to produce chairs if
provided with unlimited amounts of wood but no nails or glue to hold the
pieces together.

