
The Psychology of Tackling Hard Problems - dmvaldman
http://davidvaldman.com/post/20027940591/psychofhardproblems
======
gruseom
This post is spot on. It's encouraging to see one's experience reflected back
from someone else. I agree with the author that the psychology of this is
critical and that it doesn't get discussed much.

One point I'd like to add is that history gives a very misleading picture.
History is divided into people who worked through hard problems and triumphed
(these are called "geniuses") and people who either gave up or died trying
(these are called "failures"). Those seem like opposites, but it's only
hindsight that makes them so. Psychologically, when you're in the middle of
it, which is when it really matters, there's no way to know. All you know for
sure is that you've been walking down a dark tunnel for a really long time.

An interesting question, more interesting than it appears, is: why would
someone do that? That is, at some point, having walked down the tunnel for
quite some time, the rational thing would appear to be to turn back. Why would
someone not turn back?

~~~
ArekDymalski
In psychology this problem is called auto-handicap (or
<http://en.wikipedia.org/wiki/Self-handicapping>) and generally is a strong
force behind any form of procrastination.

~~~
gruseom
No, that's a reason _to_ turn back. We're talking about the opposite - why
someone would not only try, but keep going even after countless data points
indicate failure.

------
cottonseed
I have to agree with this. I'm also in math grad school after numerous
startups. The hardest part about transitioning from math undergrad to graduate
school is that you know assign problems in undergrad have a solution, you just
have to go out and find it. Research problems just might not have a solution
in reach of existing technology.

However, I find the difficulty and uncertainty of research mathematics much
worse than anything I encountered even in research-heavy startups. Depends on
what you're working on, I guess.

~~~
cottonseed
The idea of fear in research is not a new one. I'm reminded of the classic
quotes by Dirac, "Scientific progress advances in units of courage, not
intelligence." and of the classic essay by Ivan Sutherland, "Courage and
Technology."

I think fear can be a valuable signal. If your success rate is a discouraging
1%, fear might be trying to tell you that you need to find a smarter way to
solve the problem than blind (or even moderately educated) guessing. Often in
mathematics, at least, you can discover things about the structure of a
solution before you find it, things which in turn can help you find the
solution. Often the discouragement of a few failed attempts pushes me to step
back and trying to think about the problem in a deeper and smarter way.

------
capkutay
"There’s a simple reason why tackling a hard problem can lead to depressive
symptoms: you’re necessarily wrong 99% of the time."

This immediately rang out to me...any time I'm having problems outside of
working (relationship/depression), it kills my ability to be resilient and
work through a problem until it's solved..I think working through emotional
pain is an excellent skill on its own.

~~~
coffeemug
_One can not do right in one area of life whilst he is occupied in doing wrong
in another. Life is one indivisible whole._ \- Gandhi

The more I live, the more I realize that this statement isn't about moralizing
and is about a fundamental truth about our psychology.

~~~
nick_urban
Studying Zen made me realize that most of the teachings which people interpret
moralistically (or theologically) actually point to a psychological reality.

~~~
EvilTerran
At least with Buddhist teachings, I'd definitely agree. Similarly with some of
the Ancient Greek schools' teachings (Epicureans, Stoics, etc), even in parts
of their mysticism. I don't know enough about, say, Hinduism, Taoism, or
Shinto to comment on those, but I get the impression they're fairly similar in
that regard, too.

However, I can't help but feel that a lot of Abrahamic teachings seem to be
mostly about increasing the mimetic fitness of the faith. For instance: "go
forth and multiply" is always good for keeping the size of your congregation
up; evangelism likewise; "you shall not boil a kid in its mother's milk" reads
to me as "don't eat your future dairy animals", a handy survival tip in the
bronze age; and so forth.

------
nick_urban
Hackers frequently have an idea, build 90% of an MVP and then never launch.
Why would they get so close and never cross the finish line?

It seems obvious: if you don't launch, you can't fail.

A subtle fear of rejection is understandably common in a population that
experienced it frequently in their formative years.

------
patrickd
I think the key is in not seeing "being wrong" as a "failure"

"I've failed" != "I'm a failure"

No one gets everything right all of the time. Accepting that and seeing hard
problems as a challenge to overcome rather than a potential failure waiting to
happen is, in my opinion, the way forward.

~~~
derefr
It's not so much that you see yourself as a failure, in my experience--it's
more that, without frequent successes (however tiny) to push you forward, you
run out of steam--that is, dopamine--and find it hard to _want_ to try the
next thing. It's the opposite of getting addicted to something: with no bells
and dings and coins falling out, you become apathetic.

~~~
chairman_meow

        >>..you run out of steam--that is, dopamine
    
    

Try amphetamines to plow through the occasional grinding. A colleague of mine
the other day (half?) joked that Sillion Valley runs on caffeine, amphetamines
(Adderal, Ritalin, Vyvanse, etc) and cocaine. I think the same statement
applies to Academia (at least the first two substances, not sure about the
last one..).

From: <http://en.wikipedia.org/wiki/Paul_Erd%C5%91s>

"His colleague Alfréd Rényi said, "a mathematician is a machine for turning
coffee into theorems", and Erdős drank copious quantities. (This quotation is
often attributed incorrectly to Erdős himself. The German original, "Ein
Mathematiker ist eine Maschine, die Kaffee in Sätze verwandelt"[11] of the
sentence is a wordplay on the double meaning of "Satz": "theorem" or "residue
of coffee", lost in the English translation)[12]

After 1971 he also took amphetamines, despite the concern of his friends, one
of whom (Ron Graham) bet him $500 that he could not stop taking the drug for a
month.[13] Erdős won the bet, but complained that during his abstinence
mathematics had been set back by a month: "Before, when I looked at a piece of
blank paper my mind was filled with ideas. Now all I see is a blank piece of
paper." After he won the bet, he promptly resumed his amphetamine use."

As they say in The Men Who Stare at Goats, "Amphetamines- Not to be abused,
but very fucking handy."

EDIT / Disclaimer: If you (the reader) decide to try this, do your own,
thorough research. It IS risky and it can end very badly for some people.
You've been warned.

~~~
salemh
Lighter:

Nootropcis, which may be interesting to the hackers aboard.
<http://en.wikipedia.org/wiki/Nootropic>

Focus XT has been nice for me (like a steady pot of coffee (which is
inefficient with tolerance building) at a steady rate without any jittery
crashes.

[https://www.google.com/search?sourceid=chrome&ie=UTF-8&#...</a>

------
astrofinch
One idea: positively reinforce the behavior of taking a solid swing at your
problem regardless of whether you successfully solve it.

In other words, cheer yourself on for _ruling out_ apparently promising
solutions as well as successfully solving problems.

~~~
EvilTerran
Sounds like what I've read of the quest for the Higgs Boson -- "We know it's
not _here_ , I'd say that's progress! Now let's look over _there_ instead!"

------
amcintyre
Good post--I'm in the midst of doing a PhD on the side while working, and I've
noticed that "I'm afraid it won't work" behavior in myself during both my
research and working on hard problems in my day job. Glad to know I'm not
alone!

------
NadaAldahleh
Too many comments here are a re-phrase of "learn from your failure" quote. I
hate to sound too negative here, but people learn from their success a whole
lot than they learn from their failure. It sounds like something we just tell
ourselves after we've failed to make ourselves feel better.

Though I agree that keep trying (and obviously failing many times along the
way) is often necessary to get to where you want to go.

~~~
gruseom
I'm afraid you've missed the point, which is about the psychology of failing
99% of the time. That's not, like, a little hard. It is excruciating.

We use words like "persistence" or "learn from your failures" so much that we
think we know what they mean; so much, in fact, that they become boring. That
confuses repetition with understanding. What's valuable about the OP (and the
comments are just reflecting this) is that he put into words a tiny bit of
that experience in a way that _isn't_ just repetitive.

------
the_cat_kittles
Makes me think about how, in college, I used to put down a math problem as
soon as I thought I knew how to do it. I would put off doing it, and I never
understood why. This was totally the reason! It seems obvious since, as you
said, when I went about actually fleshing out the answer, I would usually
discover something that was wrong. Great observation.

~~~
dmvaldman
I think the effect is quite general. I definitely notice what you mention in
the students I teach/tutor. Many times there are gaps in our knowledge as
well, and instead of filling them, we get frustrated and "put the book down."

------
chaseideas
Strongly agreed, and great insight on the everyday struggle of a research
heavy life or career!

It can be a heavy burden at times, but being prepared mentally helps a lot.

As my dad always taught me growing up, "attitude is everything."

Plus, tackling a large stack of problems, and iterating until successful feels
so rewarding. Both during and after the fact.

 _Failure is key to success._

~~~
dmvaldman
I remember reading the first page of a juggling book was: Pick up a ball. Drop
it on the floor. Be prepared to do this thousands of times.

Kind of takes the fun out of it eh?

~~~
chaseideas
Haha! Excellent quote. Are we talking about life or juggling, or wait,
business? ;)

Never gave much thought to my early interest in juggling or the juggling books
I read when I was young and how it could possibly be later related to my
approach to life and career.

I'm a modest juggler at best, and still young in my professional career in the
tech world, but that quote and general sentiment gives me new perspective on
how my opinion of failure was shaped growing up.

------
akkartik
To expose yourself to positive black swans you have to take on this psychic
toll. This is why it's human nature to prefer to bias for small regular wins
and the occasional catastrophe.

<http://www.gladwell.com/2002/2002_04_29_a_blowingup.htm>

------
mituljain
Great read! This really ties up together with the idea of "keep shipping" -
while working on a startup recently - I found myself spending a lot of time
thinking about what if this or what if that. The kind of logic that paralyzes
you. This is specifically a problem for perfectionists and very intelligent
people - you're either afraid it wont be perfect or spend so much time
thinking of the most efficient or beautiful solution that in the mean time
someone else - may have already solved it. That experience really taught me to
keep shipping - stuff wont work - even if it isn't one of those massively hard
problems - sometimes things just dont work. The trick is to keep shipping and
that 1% will go up to much more!

------
lbraasch
In the scientific community, things are never a failure. Instead, it's
something we like to call 'negative data'.

You learn to embrace the negative data. At the very least, you've just ticked
off one thing that assuredly does not resolve the issue.

------
daenz
I like this paraphrased quote: "An expert is a person that has made every
possible mistake within his or her field." -Niels Bohr

That way, each mistake I make (and realize/correct) is me leveling up

------
hvass
"I'm afraid that it won't work." I know this may sound silly, but I honestly
just realized how wrong that statement is. I have actually used it to dismiss
a potential solution...

------
nimrody
There are actually two difficulties here:

1\. The problem in hand _may not have a good solution_. This is especially
true of research type problems. You have to pick the right one, otherwise it
can be a frustrating experience. This is where a broad knowledge of the
subject and previous research help.

2\. Some potential solutions may be very difficult to implement or explore, so
one tends to explore the simpler options first. When these are not enough, you
tend to loose faith.

------
6ren
iteration = trial-and-error = try-and-fail

For me, the hard part is that an experiment can be a lot of work... all that
effort, for _NOTHING_. :(

Though Tesla has more geek cred these days, Edison also did some stuff. He
said: (<http://en.wikiquote.org/wiki/Thomas_Edison>)

    
    
      I have not failed. I've just found 10,000 ways that won't work.

~~~
Dortar
"...all that effort, for NOTHING" can be an easy belief to fall into, but it
really wasn't for nothing. If you think about the work you put in before this
point in time as a sunk cost, and look at it as "well, where can I start this
problem from now?" you'll see that your starting point is a WHOLE lot
different. You find yourself with a much more informed opinion from which to
climb up from.

------
omilu
Anybody else think of that scene from Home Alone when Mcauley Culkin says
"Hey, I'm not afraid any more! I said I'm not afraid any more! Do you hear me?
I'm not afraid any more! [Old Man Marley approaches Kevin and stares at him -
Kevin runs back inside, screaming like a maniac]

~~~
WiseWeasel
Nope, only you.

------
Salmaun
I can't agree with this more...this especially rings true here in the valley.
Its a tightrope walk to synthesize feedback from failures efficiently and
effectively while not breaking pace and letting your emotions have you pause
on the undesired outcome. Persistence is magical.

------
wtvanhest
I would add that a second difficulty insolving tough problems is being able to
create solutions.

~~~
dmvaldman
There's definitely that too. But to get at the creative solution, you usually
have to wade through tons of wrong options. The key is persistence.

~~~
gcheong
I think part of the fear for me is that if the solution in my head doesn't
work I won't be able to come up with another one, but in reality even if the
initial solution is not correct it usually leads to ideas of how to change it
or how to re-think the problem from a different angle.

------
iamwil
I suppose that's why it's better to like knowing the ground truth, than having
ideas. You delight in finding the actual answer, than what you think might be
the answer.

------
WiseWeasel
So is no one going to bring up the tried-and-true weed and coffee method to
coding motivational success? Fine, let me be that guy.

~~~
melloclello
I hear ya partner. Nothing gets me going like the old hippie speedball.

