
Is there data on the quality of management decisions? - luu
http://danluu.com/bad-decisions/
======
monkeypizza
This is the same topic of Eliezer Yudkowsky's recent book "Inadequate
Equilibria" [0], reviewed by Scott Aaronson here [1]. That book does a good
job explaining why not every inefficiency can be eliminated - because getting
out of a bad nash equilibrium requires coordinated changes, and for large
enough groups you can't convince enough people to make a change at the same
time.

0: [https://equilibriabook.com/toc/](https://equilibriabook.com/toc/) 1:
[https://www.scottaaronson.com/blog/?p=3535](https://www.scottaaronson.com/blog/?p=3535)

~~~
madhadron
Or you could read short book on game theory, think for ten minutes about
perverse incentives, and get to the same results without the conceptual
diarrhea that Yudkowsky insists on. Why he had to try to drag a concept of
free energy into it with no justification, I don't know.

~~~
daveguy
"Read a short book on..." Is a pretty high bar for conceptual simplicity.

Why wouldn't those who have had exposure to the concept of free energy prefer
that metaphor to yours? You may be biased having exposure to more game theory
than free energy. One person's conceptual diarrhea is another person's
cleanest explanation.

~~~
alasdair_
>"Read a short book on..." Is a pretty high bar for conceptual simplicity.

Take five minutes and read the wikipedia page on the prisoner's dilemma then.

Even in a two player game, it's clear why individually optimizing for personal
gain leads to poor outcomes for everyone, yet if there was a way to force
people to act in a particular way, the total outcome for everyone would be
better for all.

~~~
ucaetano
Except that the one forcing people will inevitably force people to act in a
way that maximizes his/her personal gain.

~~~
taneq
Which is fine if that is also a way which maximizes those peoples' personal
gain as well.

~~~
ucaetano
But is rarely or never the case. See pretty much every dictatorship in
history.

------
jerf
"A statement I commonly hear in tech-utopian circles is that some seeming
inefficiency can’t actually be inefficient because the market is efficient and
inefficiencies will quickly be eliminated."

There's equivocation in that statement; what market efficiency is and what
people generally mean by efficiency are not the same thing. This is not an
accusation of danluu, though, as I believe it is an accurate statement of an
error made by a lot of people.

Market efficiency is really a rather boring statement about the difficulty of
performing arbitrage in the long term. Plus, given that the market is not
absolutely efficient we should expect errors to be made.

But we can do some numerical analysis here. Consider the range of
possibilities. Suppose population X is paid 0% of what everyone else is paid
in the market, for the same output. It would take an astonishingly-large, an
implausibly large, countervailing force to prevent some company from swooping
in and claiming that free labor. (Remember, even if 99.9% of the firms on the
market choose not to, it just means that the .1% is going have that much
easier of a time.) However, for group X to be paid 99.9% of what everyone else
is paid on average doesn't take much at all; indeed, mere noise could produce
that result. Rather than it being a question of "is or is not discrimination",
this sort of analysis can put numbers on _how much_ , from which you can start
analyzing things like the value of certain changes vs. their costs.

No conclusion; I'm just musing here. Except that I think that people generally
do a lot of binary thinking here when quantitative might be more useful.

~~~
rossdavidh
"There's equivocation in that statement; what market efficiency is and what
people generally mean by efficiency are not the same thing"

True, but: it's no coincidence that the terms are used for different things.
What we really care about as humans is "market efficiency" in the ordinary
English sense of the word. That's really hard to define, though, so economists
redefine it in a very precise, technical, and yes, 'boring' statement. Now it
is easier to see if we have it, but also not very useful. However, it can
still be used to argue that any intervention in the market will be a mistake.
The confusion about what the term "market efficiency" means is not just a
coincidence. Redefine it as something easily proven, and then let people
mistake that for a proof of what they actually care about, and you get the
same political/lobbying result as the (rather more difficult, perhaps
impossible) task of proving that markets are efficient in the way we actually
care about as humans, rather than just economists.

~~~
notgoodenuff88
The definition is easy: by market efficiency, people mean time (labor market
being the only market Adam Smith concerned himself and invisible hand being a
reference to how the people themselves will tacitly protect their own
interests).

And right now, the sense is that we're spending a lot of our time in jobs that
overwhelmingly benefit a rich minority, which is not time spent well it.

When our ability to focus on self-preservation is being expropriated, people
revolt

This interpretation isn't even up for debate, IMO. It's been investigated by
biologists, neuroscientists, psychologists, medical doctors, philosophers,
logicians, mathematicians, physicists, computer scientists.

A curious omission: economists. Very few of re-known willfully admit their own
field is being twisted into undermining the majority for the minority.

------
thisisit
> A statement I commonly hear in tech-utopian circles is that some seeming
> inefficiency can’t actually be inefficient because the market is efficient
> and inefficiencies will quickly be eliminated.

At the risk of sounding like an old fogey, I believe a similar statement can
be made for stats-utopian circles and their belief in ability to quantify
every thing.

As the post says:

> One nice thing about sports is that they often have detailed play-by-play
> data and well-defined win criteria which lets us tell, on average, what the
> expected value of a decision is.

So, using sports as a proxy is not the same as management decisions where
there are no well-defined criteria or expected value.

~~~
0xcde4c3db
> So, using sports as a proxy is not the same as management decisions where
> there are no well-defined criteria or expected value.

Exactly. The biggest problem with quantifying efficiency in particular is that
you have to define the inputs and outputs. Your biases are embedded in that
definition rather than in the calculation _per se_. You can make some pretty
awful processes "efficient" if you assign zero value to some of the inputs,
and conversely some pretty great processes "inefficient" if you assign zero
value to some of the outputs.

~~~
Bartweiss
> _You can make some pretty awful processes "efficient" if you assign zero
> value to some of the inputs, and conversely some pretty great processes
> "inefficient" if you assign zero value to some of the outputs._

I'm halfway convinced that time-motion studies are some kind of demonic
invention for exactly this reason. We can get 3% more productivity out of an
assembly line, just by doubling the accident rate and giving every worker RSI!

Goodhart's Law is dangerous even when we're making management and business
decisions in good faith, but when ideas like "long term injuries are
irrelevant" are implicitly added it becomes monstrous.

------
jackgolding
This article is quite interesting for the isolated baseball analysis but is
ultimately a long-winded discussion (which goes down a sports rabbit hole
which ultimately doesn't contribute to the argument.) The TLDR is:

> If we want to look at the quality of decision making, it’s too simplistic to
> say that we expect a firm to make good decisions because they’re exposed to
> markets and there’s economic value in making good decisions and people
> within the firm will probably be rewarded greatly if they make good
> decisions.

Is there actually anything a reader here can take from this writing other than
the assertion I quoted above (which is the author's point of view and isn't
proven in nearly as much rigour as the baseball analysis is.)

~~~
geofft
I think the argument holds up: it is that, even in a field where management
decisions _can_ be clearly quantified, the data is collected, and people are
doing the analysis, bad decisions persist (management makes bad decisions when
they could make good decisions, and the market does not value good management
decisions). So in a field where it is much less clearly quantified, why would
we expect the market to ensure high-quality management decisions?

I don't think the post proves that the market fails to incentivize good
decisions. That's a hard thing to prove. It does certainly give us serious
reason to doubt the common claim that the market succeeds in incentivizing
good decisions on a short timescale. (It might still be true, but we should
have much less confidence in that statement being true than before we read
this post.)

~~~
jackgolding
> It does certainly give us serious reason to doubt the common claim that the
> market succeeds in incentivizing good decisions on a short timescale.

How does the article suggest this?

~~~
geofft
The claim "the market succeeds in incentivizing good decisions on a short
timescale" is at odds with the article's demonstrated conclusion "the market
fails to incentivize good decisions on a short timescale in the field of
baseball". So you're left with two possible ways to resolve this: either
baseball is unique and the market doesn't work properly there for reasons that
don't generalize (but the article gives some reasons to believe that this
isn't the right way to resolve the contradiction), or baseball isn't special
and non-baseball markets very likely do similar things.

------
ggambetta
This looks super interesting. Unfortunately, I couldn't follow the explanation
beyond the few first paragraphs because of so much baseball terminology and
logic, which is completely alien to me :( The intro is confusing enough, but
then there are terms that aren't defined, such as "stealing" \- are readers
supposed to be able to infer what this means just by context?

~~~
csours
So the whole intro is to make the point that a certain type of play (base
stealing) doesn't make sense, even for teams that do it well - and it took
until 2012 to figure that out and get their players to stop doing that play.

~~~
AskewEgret
Ceteris paribus, base stealing doesn't make sense. But baseball is a game
played by humans and even if it weren't, there are a large number of variables
that don't go into the simple analysis.

For example: The members of the defending team adopt fielding positions based
on a maximum potential for fielding a hit ball. But if there is even a threat
of a stolen base attempt, the defense must change their positioning
accordingly, affecting their ability to field a ball put into play.

If a player does make an attempt to steal a base, at least one defender must
begin to change their positioning before the hitter has even had a chance to
hit the ball. If they do not, they will not make it to the base in time to
receive the throw from the catcher and either the throw will not occur or it
will go out into the outfield and the runner may advance another base.
Alternatively, the hitter may actually hit the ball and the ball may go to an
area that it is statistically likely to go, where a defender would have been
but for the stolen base attempt. What was to be an out is now a hit and the
runner, having a head start because of their stolen base attempt, is able to
advance further than they normally would have been able to. Because the hitter
hit the ball, this situation does not get recorded in the statistics as a
stolen base attempt!

Baseball is really a fascinating game to think about!

~~~
ashark
> For example: The members of the defending team adopt fielding positions
> based on a maximum potential for fielding a hit ball. But if there is even a
> threat of a stolen base attempt, the defense must change their positioning
> accordingly, affecting their ability to field a ball put into play.

Yeah, that was the biggest flaw I spotted in the analysis of base-stealing: it
doesn't seem to capture effect of the _threat_ of base stealing on the
defense's behavior, which is very probably non-zero and in favor of the
offense. If you never, _ever_ steal, and the defense knows that (someone'll
notice before long), they can play better defense against the batter. It may
still work out that a never-steal rule still provides better outcomes (by a
smaller margin), but I'm guessing "rarely steal" ends up being the better
guideline, overall.

~~~
hyperpape
If that's true, then I believe it means that the defenders are
overcompensating for stealing. As I understand it, in equilibrium you'll have
the feature that the defense cannot benefit by guarding steals less closely in
exchange for better fielding position.

If trying to steal loses bases, then the defense could spend less effort
preventing steals, and more effort fielding.

This principle comes up a lot: if you get into every college/job you applied
to, then you probably didn't apply to enough schools/jobs, unless 1) you
didn't want to go to Harvard/AppAmaFaceGooSoft, 2) you got in to
Harvard/AppAmaFaceGooSoft, 3) you couldn't afford the applications.

------
scott_s
An aside to the main point: the author claims that one can do a similar
analysis to other sports. My intuition is that you can't do quite as accurate
an analysis of, say, soccer, than you can baseball. As a sport, baseball is
unusually discretized when compared to a much more continuous sport like
soccer or hockey. Being discrete, it makes it much easier to represent as a
state machine, which makes it easier to simulate, and easier to suss out cause
and effect. American football, I think, may be somewhere inbetween. I know, of
course, that other sports do significant statistical analysis, but my
intuition is that it is harder to attribute cause and effect at the fine-grain
level that one can with baseball.

------
Joeri
From what I can gather making good management decisions is a lot like making a
good estimate for a software project. There are two ways to go about it: (A)
do a lot of work to break down the problem to manageable parts, put a well-
reasoned number on each part, then sum up the parts with the necessary
tolerances and risk factors, or (B) just go with your gut.

I see a lot of (B). Hilarity ensues.

~~~
kinkrtyavimoodh
What is gut if not an accumulation of experiences? If an engineering-manager-
with-10-years-experience's gut produces the same accuracy as a fresh grad's,
then it's not of much use, and there's a good chance that the manager would
generate a bad estimate even following (A). That's not to say that people
shouldn't analyze. Just saying that gut is not random guesswork.

~~~
mcguire
Mostly, it's answering a simpler question than was asked. See _Thinking, Fast
and Slow._

------
Retric
One issue with analysis like this is micro optimizations have more information
than simplified abstractions. Consider, in the bottom of the 9th 1 or 2 runs
may have identical value. So, reducing the total expected value of runs to
increase your odds of a single run may be a very good trade off.

And that's assuming you want to optimize total wins. Increasing your odds of
winning the playoffs may reduce your odds of getting into them by having more
higher paid people, but a lower average player. Further, maximizing making
money over time is yet another option, which may promote showmanship over
success.

~~~
lodi
This is addressed in the article:

"But there are situations where the approximation isn’t very good, such as
when it’s the 9th inning and the game is tied. In that case, a decision that
increases the probability of scoring 1 run but decreases the probability of
scoring multiple runs is actually the right choice."

I understood that the "full" model includes these considerations.

~~~
Retric
I was bringing up that edge case specifically because 'winning' is the obvious
optimization, but not necessarily the correct one.

Bottom of the 9th bases are loaded and the guy at bat hit's a home run, that's
the kind of thing that sticks with people and makes more money in the long run
than a bunt that get 1 run.

------
bluetwo
All companies make both good and bad decisions. They don't fall apart when
they make a single bad decision. They fall apart when he sum total of the
impact of their bad decisions outweighs the sum total of the impact of their
good decisions.

When this happens, the last bad decision usually takes the blame, however it
is rarely the true cause.

------
ksk
>A less contentious example is that when you see a big company doing something
that seems bizarrely inefficient, __maybe it’s not inefficient and you just
lack the information necessary to understand why the decision was efficient
__.

This pretty much sums up a lot of tech news and the associated comments on
such news.

~~~
analog31
Indeed, and likewise when someone does something that seems irrational.

~~~
taneq
I've long said that if the way someone's behaving doesn't make sense, you
don't understand their motivations.

Now, it may be that their motivations are all messed up and they're acting the
way they are because sabotaging their relationships reaffirms their core
belief that they're unlovable. But it does make sense once you understand the
framework they're working in.

------
ihsw2
The problem is externalities -- both positive and negative externalities
affect the outcome and it is basically impossible to determine whether an
outcome occurred due to management decision or due to an externality (eg:
revenue growth due to successful execution or revenue growth due to changing
business cycles).

~~~
bweis
Couldn't agree more. It's really hard to pinpoint the exact cause of such an
outcome.

------
inputcoffee
There are three issues I have with this:

1\. "Quality" for management decisions are notoriously ill defined for the
same reason that we never know if we did the right thing with the economy: the
experiments can never be repeated.

2\. The example serves to explain what a decision is and what a good decision
would be, but that is not the concept that needs explanation!

3\. "some seeming inefficiency can’t actually be inefficient because the
market is efficient and inefficiencies will quickly be eliminated." is not the
right statement of efficient markets. Rather, the act of trying to make it
more efficient is exactly what makes it efficient. So if you see an
opportunity, someone exploits it and thus makes it more efficient.

~~~
philipov
4\. Markets aren't actually efficient because they are stacked in favor of
incumbents through their ability to lobby for anti-competitive legislation, as
can be seen with the perversion of the copyright and patent systems.

~~~
mulmen
The mistake you made in this example is attributing an action in the
legislative system as a failure in the market.

~~~
philipov
I am saying the opposite: the market becomes less efficient as a result of
legislative capture. But it is a feedback loop, so attributing the dynamics to
simple cause and effect is not appropriate.

------
sokoloff
> we see that the average stolen base is worth -.467 runs and the average
> successful steal is worth .175 runs.

First clause should be "average caught stealing"

------
mcguire
Tl;dr: Many organizations make decisions like a drunken madman whose seeing-
eye dog just ate his hearing aid.

For reasons why, see _Thinking, Fast and Slow_ (just reading it now; it's
quite good) and Thaler's _Misbehaving_.

------
marcus_holmes
Betteridge's Law strikes again [0].

You could replace that entire post with "No. Unlike baseball."

Not that the post wasn't interesting, of course. To me, it leads to the
inevitable conclusion that we need to record failures as well as wins in order
to get accurate data on management decisions.

0:
[https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...](https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines)

------
hatcherdogg
Outstanding read.

