

Decisions Without Data are Guesses - jfarmer
http://20bits.com/2008/04/05/decisions-without-data/

======
notauser
Rough summary of the argument: When you need to make a decision, use the data
you have available to make the one most consistent with achieving your goal.
Also, the plural of anecdote is not data.

...however...

He misses the fact that it can be very hard to gather a sufficient sample of
data for a complex set of variables. If you are designing a web site there are
literally hundreds of choices to make. You can't reasonably expect to test all
of your decisions before making them, or else you will be paralyzed by delay.

This leaves you with a second option of mocking up a couple of choices and see
which ones are liked best, leaving you potentially with the best of a bad
bunch.

Or, my preference, option three: Pick a couple of design champions who will
tear any design to shreds and make you really justify your choices in terms of
work flow and user interaction.

Generally you can assume that you _aren't_ stupid, and you can figure out the
right choices to make, however you won't always notice you need to do so
unless you are being forced to justify your decisions by a grumpy critic.

~~~
jfarmer
Hrm, do you think my article was too long?

Anyhow, yes, you can make arbitrarily complex measurements.

The point was more human than technical: some data is better than none, even
if it's not fine-grained. It results in fewer pointless arguments and better
decisions overall.

~~~
apathy
_some data is better than none, even if it's not fine-grained._

Not necessarily. If the features of your small data sample are
unrepresentative or poorly analyzed, you run the risk of distorting your
priors and making a poorer decision than that expected by chance alone.

We can chew the fat on this one for a very, very long time :-)

~~~
jfarmer
Ok, you're right. I'm a mathematician by training so these technical concerns
don't escape me.

Still, I tried to keep my point constrained to those situations where you
don't run into decisions not easily resolved in the manner I described.

Sure, if you're wading into a new market, old models or assumptions might be
incorrect. Or if you need to make a decision now and don't have time to gather
data your intuition and savvy are going to be instrumental.

But for most day-to-day, pedestrian decisions around software, web
development, and business there's plenty of data that will tell you whether
you should zig or zag.

------
FleursDuMal
This article presents a very narrow view of decision-making that only applies
in certain situations.

Being able to make good decisions based on conflicting or incomplete
information is critically important.

~~~
jfarmer
Fleurs,

It was intended to be a small point. The psychology and effectiveness of
decision-making strategies could fill up books (and they have).

The scenario I had in mind is one where you have some limited set of options,
a consensus is necessary, and the overhead of measuring the effectiveness of
the alternatives isn't too high.

In those situations you're going to argue until someone gives in, someone
higher up tells you to knock it off and makes a decision for you, or you
decide to see which alternative actually does best.

------
aneesh
I agree that data-driven decisions are less common than they should be.
However, there is such a thing as over-dependence on data, and this piece
might be toying with the line. You don't have enough resources to test every
little thing - sometimes you have to use your intuition and move on.

~~~
jfarmer
aneesh,

I understand that, actually, and left it out of the blog post only for
brevity's sake.

When I make decisions about my websites and other projects I have in mind a
picture of my audience and what they're doing. This sort of experience is the
basis for most intuition. Some people are really, really good at it, too.

The right kind of experiences can be data, too, but there are two dangers: (1)
not being wiling to change your mind and (2) acting off too little experience.

I just expect that any "expert" whose intuition I'm supposed to trust will be
able to justify their opinion when push comes to shove.

~~~
apathy
_I just expect that any "expert" whose intuition I'm supposed to trust will be
able to justify their opinion when push comes to shove._

Don't expect them to justify it solely on your terms, however. If you're
breaking into a new market or operating under asymmetric information pressure
with significant uncertainty -- or even just regular old data starvation --
appealing to similar but difficult-to-quantify means of supporting one's
intuition may be crucial.

It would be better for all concerned if the person making a claim could always
articulate their reasoning _in toto_ , but under the pressures of time, you
may occasionally need to rely upon a person's track record, and trust them.
Because their track record under similar conditions is a prior -- that, too,
is data.

Extraordinary claims require extraordinary evidence -- but people with an
extraordinary track record can sometimes be trusted to make extraordinary
judgments with ordinary data.

