
Don’t Let Metrics Undermine Your Business - prostoalex
https://hbr.org/2019/09/dont-let-metrics-undermine-your-business
======
6gvONxR4sf7o
I work as a data scientist on an extremely data driven team at an extremely
data driven company, and I couldn't agree more strongly. Everyone I work with
would agree too. "Numbers aren't everything!" says the person whose job it is
to deal with business problems via numbers.

But this advice is so hard to implement. It's like the whole null hypothesis
p-value debacle going on in academic publishing right now. Sure we can agree
it's no good, but there's no _standard_ alternative.

Standards are powerful. They break lots of ties. All else equal, if you have a
hunch and I have a measurement tied to objective reality, people will side
with me. Nobody ever got fired for choosing Java. Nobody ever got fired for
doing the thing that made The Metrics go up.

It's so reassuring to just say "We still disagree, so let's just test it."
We're submitting our dispute to adjudication via Science by resolving via The
Metrics! It's much harder to be objective without these kinds of numbers, and
I like being objective! I want the scientific method to tell me I did the
right thing and I'm a good person.

Because without that, when we're trying to figure out what to do, it's just
us, you know? It feels like guessing, and that's hard.

~~~
jbob2000
This is a recipe for mediocrity. Great things are created by people following
their gut, not their metrics.

Jobs didn’t have a data scientist telling him to make the iPhone. A data
scientist would have told Notch to make a Call of Duty clone instead of
Minecraft. There was no data scientist behind Gates, Warren, Zuckerberg, Musk,
or anyone of their ilk.

Looking back in history, some great discoveries and inventions were held back
because the “data” said it wasn’t possible. Everybody thought ulcers were
caused by stress and bad food, then Barry Marshall came along and _literally
trusted his gut_ by eating a bunch of bacteria to prove that’s how ulcers were
formed.

At best, data science helps you squeeze a couple extra bucks from someone
else’s ingenuity. Once Jobs died, the data scientists got their hands on the
iPhone and chopped it up into different models. And it was the data scientists
who made your Facebook feed ad-ridden junk. And at worst, data science is
destroying our society. It’s the data scientists who are driving this insane
data vacuum that is plaguing the internet and our daily lives. You thwarted
the 2016 election by weaponizing data against America. You did it again with
brexit. No doubt you will be out in full force for every major political event
for the next few decades, sullying the discourse of the electorate.

Bit of a rant... sorry. I hate this over-reliance on data and the smugness
it’s captors carry.

~~~
mdorazio
I would say it's a recipe for _continuation_ rather than _innovation_. You're
unlikely to invent anything revolutionary just by looking at metrics, but
you're also unlikely to keep a business going without them. It's easy to look
at something like the iPhone and say, "metrics didn't invent that!" but
metrics absolutely put millions of phones in the hands of customers, built an
effective production and supply chain, optimized the whole experience, and
turned Apple into a trillion dollar company. As with most things, there's a
balance and a place for each.

~~~
jbob2000
I disagree that metrics helped Apple scale, but we’d have to wade in to a
semantic debate to settle this.

When you have a killer idea, scaling is _obvious_ ; just do more. More people,
more machines, more materials, etc. You don’t need a number to tell you how
much more to scale, because you always want the max anyways. Tim Cook was told
there weren’t enough CNC machines in the world to manufacture the unibody
MacBook. So he bought the company that makes CNC machines. What’s the point of
metrics when you can shift the entire world to meet your needs? Great things
are like that.

Metrics are for mediocre products and services. We use them in banking because
people fucking hate the banking system and we have to pull teeth to get
customers to spend money. McDonald’s uses metrics to gauge how shitty they can
make their beef without people noticing. Hollywood uses metrics to decide
which movie franchise to reboot or make a sequel of. Metrics are a sign that
you are done innovating and you are squeezing the sponge before someone else’s
innovation dethrones yours.

~~~
nitwit005
> You don’t need a number to tell you how much more to scale, because you
> always want the max anyways.

And you bankrupted the company building factories you didn't need, because
iPhone demand has its limits.

------
tryitnow
This article represents one of the problems I have with HBR and "business
school" literature in general.

It's a rehash of tried and true principles that have been described numerous
times before almost always without attribution to original sources.

What's wrong with that?

Well, for one the authors are passing off work that is clearly not their own.
Fine, whatever, it's a b-school publication, nobody takes their intellectual
output seriously anyways.

But the more important problem is that you lose a sense of where this
knowledge has come from, where it's been applied, how it's been applied, and
other important context.

In other words, reading this article is basically a waste of time. It's better
to stick to a handful of classic business texts (Drucker's a good place to
start and yes, he goes over all the recommendations these authors make in
Effective Executive). Reading Kaplan on the Balanced Scorecard is another good
place to start.

~~~
heymijo
> _But the more important problem is that you lose a sense of where this
> knowledge has come from, where it 's been applied, how it's been applied,
> and other important context._

Thank you for saying this!

A thought I have had in the past few years is "there's been nothing new in
management since Drucker and Deming."

I go back and forth because some people have done a nice job of updating their
ideas/applying them to modern disciplines, which is valuable.

However, what you said about where that knowledge has come from/been
applied/how/context is sorely missing.

That these ideas are still not common knowledge leads to so many unnecessary
mistakes.

------
tomrandle
Seems to me Surrogation is very similar to The Cobra Effect / Perverse
Incentives ([https://www.geckoboard.com/learn/data-
literacy/statistical-f...](https://www.geckoboard.com/learn/data-
literacy/statistical-fallacies/cobra-effect/)) , or the McNamara fallacy:
[https://www.geckoboard.com/learn/data-
literacy/statistical-f...](https://www.geckoboard.com/learn/data-
literacy/statistical-fallacies/mcnamara-fallacy/)

Both very real problems that have existed forever. The solution, as the
article suggests, is not relying on just a single metric, and not losing sight
of your wider strategy. There’s a really good article on Reforge about the
problems with the recent trend of a single “North Star Metric”:
[https://www.reforge.com/blog/north-star-metric-
growth](https://www.reforge.com/blog/north-star-metric-growth). Stacey Barr’s
another thought leader on the subject: [https://www.staceybarr.com/measure-
up/a-single-kpi-that-meas...](https://www.staceybarr.com/measure-up/a-single-
kpi-that-measures-overall-business-performance/).

Metrics absolutely cannot replace strategy and vision. It needs to be a core
value of the company that you’d rather do the right thing by the wider mission
than drive a metric at any cost. Managers at every level need to live up to
that. This is harder the bigger the company. It’s one of the things OKRs, love
or hate them, try and help with. Like with everything it’s whether you comply
with the spirit or the letter of the law. And whether people get rewarded for
latter!

------
olau
It's interesting to ponder how a metric, through its design, is a
psychological brain hack that trumps rational thought and common sense.
Perhaps it's the gamification aspect that gets us?

An example I think everyone can relate to is in education where grading is a
rich source of perverse incentives. In developing minds, no less. Yet the
metric persists.

(Just to be clear, the alternative to grading is not no feedback, but
personalized, situation-specific feedback. Freeing yourself from a bad metric
can open doors.)

------
666lumberjack
This reminds me of Goodhart's Law: "When a measure becomes a target, it ceases
to be a good measure."

It seems like in the case of employees at a company, just knowing that a
parameter is being measured can result in it implicitly becoming a target even
when there is no explicit focus on it (especially if it is the only or one of
a small number of metrics that is discussed).

I have observed that in general most humans are very bad at considering the
incentives created by rules and rewards they impose or seek to impose.

~~~
dragonwriter
Goodhart’s Law really only applies to measures that aren't measuring the
actual figure of merit but a proxy that is incorrectly assumed to be
inextricably tied to it.

This is very common, though, and a real problem associated with it is people
pick proxies without clearly documenting (or even thinking through) what they
want to measure, why they pick a particular proxy, and how the proxy might
fall. As a result, people remote from the decision (including the original
decision maker) tend to incautiously apply the proxy because they aren't even
aware that it is not the actual figure of merit.

~~~
mikeash
The point of the law is that virtually every measure is a proxy to some
extent. Finding a measure that exactly matches what you really want to
optimize is almost impossible.

------
ThomPete
Metrics are great for optimizing your business not for deciding which one to
build.

~~~
TeMPOraL
The problem talked about in the article is that it's not easy to express your
optimization goals in terms of concrete metrics, and your optimization process
will always optimize those metrics, not your real goals.

~~~
dragonwriter
> The problem talked about in the article is that it's not easy to express
> your optimization goals in terms of concrete metrics,

Largely, because what you are wanting to optimize is something like current
value of the future income stream of the firm. So a proxy is taken, which
inevitably has bias, and you need to understand why hat that big is likely to
be a continuously sanity check your results for reason to think that you may
being caught in that bias. Which you probably can't measure well (though maybe
something has improved since you adopted the proxy), because if you could
you'd have a better proxy to start with.

~~~
TeMPOraL
True, though I was thinking about something else still: unless you're a
completely amoral alien mind, even your for-profit goals have some hidden
value judgements and preferences attached. Those extra constraints are
incredibly hard to capture as explicit goals and thus impossible to properly
measure in a quantified way. So if you now turn the optimization knob up to 11
and just let it work for a while, you might be appalled by what it comes up
with. Or if not you, the people on the receiving end of your optimizations
definitely will.

