
Worst-Case Thinking - yan
http://www.schneier.com/blog/archives/2010/05/worst-case_thin.html
======
grellas
Worst-case thinking among business lawyers:

One of the worst traits of a _bad_ business lawyer is to fixate on remote
risks when advising a client and driving a transaction. Negotiations along
such lines rapidly enter into a surreal mode that leaves entrepreneurs
scratching their heads and wondering, "why can't we just cut through this
garbage and get to the heart of the deal?"

This sort of fixation normally adds significant costs to any business deal
("we'll need to thoroughly research all these possibilities"; "we'll need a
3-page indemnity clause instead of the 1-paragraph version"), tends to
paralyze deal momentum and cause the parties to shift their focus onto side
issues in their deal ("this deal can't possibly move forward until we can
absolutely eliminate these risks"), and winds up causing parties to negotiate
50-page contracts where a 10-pager might have done the same thing without
necessarily addressing every remote contingency that might occur in
hypothetical scenarios where things can go wrong.

A good business lawyer should be neither a deal killer nor a CYA type but
should be sharp in identifying risks and have good judgment in weighing them
against benefits so as to strike a practical balance that moves deals forward
while keeping them honest, with the client making the final decisions based on
such guidance. Of course, some deals _are_ incredibly complex, or have very
large amounts at stake, and this does warrant more fastidiousness about risk
issues. The key is to know which is which and to handle each accordingly.

~~~
geoffc
Very timely ... Amen on business lawyers! I just signed the papers on a seed
round and what drove me nuts was all the expensive back and forth on possible
IPO provisions (caps on legal fees etc.) let's see if we can get user one
without using up a significant chunk of capital on legal fees first!!

------
Gormo
This reminds me of a political philosophy course I had in college. There were
a large number of libertarian-minded students in the class, and there was a
lot of vigorous debate and discussion about the proper basis of law, the role
of the state, etc.

In one informal discussion session, the instructor introduced his concept of
"libertarian socialism", intended to design a political system that maximized
individual liberty with constraints designed to mitigate the iniquities that
might arise in a system of absolute, inviolate property rights.

The example that he used to justify his constraints on property rights was
this: What if X owned and lived happily on his private plot of land, but his
enemy, Y, decided to buy up all of the land surrounding X in a 360-degree
circle, and then build an impenetrable stone wall, completely sealing X
within? X would not be able to leave his own land without violating Mr. Y's
property rights, and Y could not be stopped from building a wall on his own
land. Y would have effectively imprisoned X, and possibly indirectly cause X's
death. Since this is clearly an undesirable result, we are justified in
adopting a system of law that allows Y's wall-construction project to be
preempted.

Most of the libertarian-minded students began attacking this model by
construing Y's wall-building as a violation of X's natural rights, and started
constructing a list of explicit abstract rights that could be asserted by X to
prevent Y from sealing him behind a stone wall a la Edgar Allen Poe.

But in retrospect, it looks like everyone, the instructor and the students
alike, was making the same mistake: in attempting to universalize their model,
they take on the challenge of having to accommodate _every_ imaginable
contingency.

Perhaps it's not merely the tendency to focus on the worst case at the expense
of the general case that causes these problems, but also the tendency we have
to think things through in an abstract, categorical way, and rationalize
universal solutions instead of defining the particular scope of our solutions
from the outset.

------
ErrantX
I jotted this down from a speaker at a security conference last year:

 _"My nightmare scenario is that I don't know my nightmare scenario. We can
obsess about all the extreme things that could happen; but the truth is when,
or if, it does happen the chance of it being one of them is tiny. In fact we
probably haven't thought of the very worst sort of scenarios; those tiny
little things that someone suddenly figures out they can exploit and turn
against us. We cannot predict or offset these scenarios. All we can, all we
should, do is make security relevant and robust - and then be ready to meet
threats quickly when they occur."_

(pretty much what Schneier says too; but I thought worth sharing)

~~~
megablast
What I take away from this is that we can waste too much time focusing on lots
of WCS, and a WCS we didn't think of comes along.

So we would be better on focusing on the core, rather than the edges.

------
j_baker
At the same time, it's important to keep Murphy's law in mind. That doesn't
mean you should let yourself be immobalized by the worst case, but you need to
make a conscious decision that the worst case isn't worth worrying about.
Don't write something off because you feel it will never happen. Assume that
it _will_ go wrong and decide if the cost of dealing with it is greater than
the cost of preventing it.

~~~
ryanelkins
>decide if the cost of dealing with it is greater than the cost of preventing
it

Yes. I build small apps for various tasks at my current job and am always
surprised at how bullet proof they want the thing to be. I spend the majority
of my time trying to safely handle the remotest of issues for these apps which
are mostly fairly trivial (not operationally critical) if they stop working. I
find it especially amusing because we host our entire operation off a single
box in our own network closet.

In other words, make sure you take care of all the simple things that have a
much higher chance for disruptive error before you start worrying about the
once in a lifetime catastrophic ones.

------
lotharbot
Whether we're talking about WMD, Global Warming, overpopulation, terrorism,
hackers, or any other issue, people have this bad habit of falling into
"Pascal's Wager" type thinking: identify the worst case, and then avoid it at
all costs. This leads to expensive and often unnecessary solutions, and it
often results in more cost-effective solutions being tabled because they don't
particularly touch on the worst case.

A better approach to risk management: create a table with likelihood on the
vertical axis, and severity of outcome on the horizontal. Then look at the
costs to move any given risk either down or to the left. Do some cost-benefit
analysis on reducing your overall risk profile, and focus your spending
accordingly.

~~~
idoh
That's not a good approach to risk management because we don't know all of the
specific risks. Even if we knew that, we don't know the likelihood of it
occurring. Even if we knew that, we don't know the harm that it would cause.
Even if we knew that, we don't know how much it would cost to make that risk
go away.

I think a better approach to risk management is to flip it and think of the
key parts of the system, and figure out ways to make the system more robust,
have redundancy, and some strategic slack built in.

~~~
Gormo
The fact that we will never know all of the specific risks is an argument
against worst-case thinking. No matter how many contingencies you try to
account for in your baseline system, there will always be risks that you
cannot account for, which will become the new worst case. So, by definition,
you can _never_ actually avoid the worst case. (Even if you may end up with a
worst-case situation that isn't as damaging as the ones you've accounted for.)

But the real solution is to distinguish risk avoidance against risk response.
Prescriptive solutions tend to focus on avoiding risks, and building those
avoidance methods into the baseline system that applies generally. But if we
isolate the unlikely risks in the way that lotharbot suggests, we can draw a
line across the likelihood axis and say "we'll avoid the risks on this side of
line and deal with the outliers on a case-by-case basis."

The best strategic slack is to have a robust way of handling unexpected
occurrences after the fact, so you optimize the prescriptive system to the
expected general case.

BTW, I'd bet there's a pretty strong correlation between "risks that we can't
account for" and "risks that are unlikely to occur".

------
huherto
Very important when applied to personal problems. Worst-case thinking can get
you in a lot of trouble. I cannot articulate how it happens but it does.
Perhaps you become paranoid and blow things out of proportion. Fear
immobilizes you. Or perhaps, you just accept the worst and stop looking for
answers.

------
edw519
As a single proprietor/founder/programmer, I've heard this "worse-case"
question far too many times:

"What happens if you get hit by a bus?"

I have no good answer, but I have lots of bad answers:

"Can't happen. There are no buses where I live."

"I work out. The bus will get damaged."

"If I don't log in 3 days in a row, the server will automatically ftp all the
source code to you."

"No problem. I have Blue Cross."

"No problem. All the code is self-documented."

"I guess I won't care anymore."

~~~
CodeMage
Heh, now I feel weird. I work for a big corporation and I've been trying to
make them worry about the "hit by a bus problem". Imagine having a very
critical part of your application, without which your business logic can't
run. Imagine that this part could bring down the production database server.
Imagine that there's only one guy who knows the heart of that part of the
system and almost no documentation at all.

Now imagine the situation in which that guy is trying -- and failing -- to
make his bosses care about that...

~~~
yellowbkpk
Sounds like you should be threatening to leave the company unless better
compensated. While you're out, you could write the documentation that doesn't
exist and sell it back to them :).

~~~
techiferous
Threatening can erode the relationship.

------
percept
Here's a good example:

[http://wtop.com/?nid=25&sid=1956201](http://wtop.com/?nid=25&sid=1956201)

Even if the people paid to think about these problems mean well, sometimes the
cure is worse than the disease.

------
godDLL
Funny, I used to joke that "democracy" really comes from Deimos, brother of
Phobos, and thus should be spelt differently.

