
Overcoming Intuition in Programming - bpierre
http://amasad.me/2016/01/03/overcoming-intuition-in-programming/
======
pixie_
I talk with a lot of programmers that think that abstracting everything is
super important. Everything must be a black box and the programmer should have
no idea what's happening inside of it. Their code is an over engineered mess
of interfaces, dependency injection, and code so deep and isolated you have no
idea how anything ever works (I mean like even if you had the source code it's
so convoluted you could never make any use of it.)

~~~
zamalek
> I talk with a lot of programmers that think that abstracting everything is
> super important.

OOP languages basically back you into this corner once a problem becomes
complex enough. As an example, DI is supposed to decrease the coupling so that
tests can actually test units and not wind up performing integration testing.
Low coupling results in a more agile codebase, but does not intrinsically lead
to more understandable code.

What goes wrong is people think that you need to "DI all the things." Like any
tool it can be used incorrectly and, from what I have seen, violating the
single responsibility principle [when using DI] is what most often leads to
unwieldy code. "Got a model class/object? That needs an interface too."

Abstraction is not to blame, it is the incorrect usage of abstraction that is
to blame; which is sadly what you are taught in school and college. Spending
even one day learning a functional language is enough to drastically improve
your understanding of how abstraction is supposed to work in OOP languages.

> I mean like even if you had the source code it's so convoluted you could
> never make any use of it.

Some IDEs can help with this (VS2015 just added "go to implementation" for C#
interfaces), as does the most reliable way to understand code: stepping
through it line-by-line (or at least method-by-method) in a debugger - once
you have a run-time itable the callee is no longer ambiguous.

TLDR; it's a necessary evil for OOP languages but is often taken too far due
to bad theory that is taught to us all.

~~~
seanmcdirmid
I don't know if functional languages are much better in preventing abstraction
abuses. I mean, a triple indirect function, an incredibly pointless (aka point
free) chain of applications, a heavy reliance on data flow, can make the
program difficult to understand, and in some languages (the lazy ones),
stepwise debugging is difficult.

~~~
jules
OO suffers from the design pattern kind of abstraction as well as the
explosion of nouns kind of abstraction because of trying to model everything
as analogies to real world concepts rather than focusing on what the program
is meant to _do_. Functional programming suffers from a different kind of
gratuitous abstraction, namely the category theory abstract nonsense kind.

I think the main source of overabstraction is the idea that abstraction is a
virtue in itself rather than only a tool to accomplish a task. This often
takes the guise of different artificial value systems like "testability",
"decoupling", and "reusability". These can be good things, but only as a means
to an end. People lose track of that sometimes.

~~~
seanmcdirmid
I don't think that is an accurate portrayal of OO. There are plenty of nouns
out there that do not model real world concepts. I can create an "expression"
class/object in a OO program (and do all the time!) to represent parts of an
abstract syntax tree. As far as I know, expressions don't exist in the real
world. Human language is quite powerful like that.

Overabstraction is often simply a matter of poor problem understanding.
Sometimes you don't really know how to solve a problem right, so you bite off
parts of it, work your way through many layers until it is solved. The
abstractions are not ideal, because not much planning was involved, and you
didn't actually have the experience to generalize appropriately but...you
needed some way to decompose the problem.

Abstraction is a tool. It has a cost, but sometimes you need to pay it and
sometimes you are better off not. We really need better ways of abstracting
and, when necessary, unabstracting.

~~~
jules
I think that's right, especially the second paragraph about the cause of the
explosion of nouns. A common approach to OO programming is to take the problem
and identify the nouns in the problem, and then make a class for each of those
nouns before having a clear idea of how the program is going to work. This is
especially prevalent in combination with TDD. This is a recipe for disaster if
you ask me. A good example of this is when Ron Jeffries tried to write a
sudoku solver.

~~~
zamalek
Agreed.

> identify the nouns in the problem

This is what I was getting at by mentioning education as a possible culprit.
You get out of university with this bad habit right off the bat, instead of
the correct one:

Identify the _responsibilities_ in the _solution_.

Identifying all the nouns in the problem sets you up for failure because you
are implementing a solution before you know what the solution actually is.
Said another way: you are in a very literal way implementing the problem and
not the solution.

~~~
seanmcdirmid
What is so wrong about that? You have to understand the problem in some way.
Perhaps we could spend a lot of time at the whiteboard understanding the
problem waterfall style before we design the actual program? Realistically,
this isn't very effective either.

------
userbinator
I wouldn't call it "intuition" exactly, but more of a "desire to avoid
responsibility" \--- I've worked with the programmers he describes, the ones
who will spend literally hours searching for a library/library function to do
some trivial task when he could've written it in tens of minutes at most. The
functionality of the one that gets chosen invariably remains 99% unused,
adding bloat and making the dependencies more complex. Software ecosystems
with many large framework-ish standard libraries tend to encourage this
behaviour more.

 _we should be careful to not enable the belief that programming should be as
easy as gluing things together_

Absolutely agree, although a lot of it is.

~~~
rileymat2
On the other hand I have seen many functions that were written in 10 minutes
that are utterly broken except for the most common cases. Where a library
function should have been used because the programmer did not understand the
depth of the problem.

~~~
BurningFrog
Yeah, being able to distinguish these cases is an important skill.

One case I learned the hard way (once!) is parsing/writing CSV files. It seems
so simple. But there are 3-5 special cases that are hard and rare enough that
you always should use the standard library written for your language.

Perhaps admitting your original decision didn't work out and and starting over
is the real important skill...

~~~
userbinator
Parsing standard CSV isn't that hard (I've written one) when you have a spec
to follow:

[https://tools.ietf.org/html/rfc4180](https://tools.ietf.org/html/rfc4180)

But in my experience, a lot of generated CSV is non-standard, and in that case
finding a library to do it doesn't help --- you need to write your own parser
so you can adapt it to the quirks of the variant(s) you're consuming.

~~~
wnoise
Although cleaning filters and a standard parser works well too.

------
bordercases
Edsger Dijkstra's work was almost entirely oriented towards breaking
programmer intuition, emphasizing that we should reason /as/ the machine and
eventually /ahead/ of the machine.
[http://www.cs.utexas.edu/users/EWD/ewd10xx/EWD1036.PDF](http://www.cs.utexas.edu/users/EWD/ewd10xx/EWD1036.PDF)

~~~
mrestko
Wow, he has interesting handwriting.

~~~
poof131
Yep, you can even get it as a font.[1]

[1] [http://www.fontpalace.com/font-
details/Dijkstra/](http://www.fontpalace.com/font-details/Dijkstra/)

------
Animats
I used to refer to this as "ritual-taboo programming", where people know what
rituals work, and which to avoid, but don't understand why. This comes from
reading books which are all examples, but don't provide reference
documentation. I used to consider this a vice; now, it's normal. It's
necessary to deal with the flood of sort-of-working software components one
has to deal with today.

Dijkstra was once the head of the "programming is hard and programmers should
_suffer_ " school of thought. Few people read his books today.

~~~
edejong
Dijkstra would agree that programming is hard, but not that those who program
should suffer. Quite the contrary, I'd like to add, as he would often point
out that programmers currently suffer because of their leaky abstractions and
continuous bug fixing.

He wanted to bring programming back to mathematical fundamentals, so it is
simple to reason about. It is my personal opinion that this ethos is the drive
behind the renaissance of functional programming.

Programming is hard. I've been programming for 25 years now and if anything,
it never gets easier. To find the right abstraction, so _only_ the correct
things become simple, is one of the hardest tasks.

~~~
klibertp
FP is not "easier to reason about" (for humans; it is easier for machines).
It's getting popular because it promises composability and reusability: the
very thing OOP promised and failed to deliver. FP will fail, too - there's no
silver bullet! - and then we'll move to something else.

~~~
strmpnk
Yet composability is exactly what makes it easier for humans to reason about.
It's the small surface of a composable part that fits into more immediate
representations in our mind.

Whether or not it's natural mapping functions to and from concepts people
might already operate with isn't the same problem. I'd argue that one can be
offset with practice, though it's certainly not something that is easy for
everyone. In that sense, FP might be a bit more immune to bad intuition, if
only that you have to give up on mapping structure to insufficient analogs.
The late object oriented point of view makes this step trivial for our minds
and is thus a significantly complicated trade-off to quantify.

------
nbouscal
The experiment referenced about disfluency has repeatedly failed to replicate.
Meta-analysis here:
[https://www.academia.edu/11918605/Disfluent_Fonts_Don_t_Help...](https://www.academia.edu/11918605/Disfluent_Fonts_Don_t_Help_People_Solve_Math_Problems)

------
quanticle
I wonder how much "nice" IDE features like auto-completion and auto-
refactoring go towards encouraging an "intuitive" mode of thinking. I've
definitely noticed that when I work on Java or C# code in vim, I have to think
about it much harder than when I work on the same code in IntelliJ or Visual
Studio. The lack of automated refactoring forces me to slow down and think
about what I'm doing, and reason about all the implications of my change,
rather than just clicking some buttons and letting the IDE take care of the
rest.

That said, while I'd _like_ to use as my sole development tool, I'm not sure
I'd be _able_ to. The (proprietary) framework I'm forced to use is basically
impossible to navigate without some kind of IDE support. Yeah, it's not ideal,
but it's the state of the world I have to deal with.

------
CodexArcanum
Although his actual point had almost nothing to do with intuition, I still
would like to argue in favor of intuition. Maybe not while writing code, but I
find intuition very powerful when reading, reviewing, and refactoring code.

This is what people often talk about as code smells. When you've trained
yourself in how good code looks, then bad code stands out to you. Once
intuition has led you towards where attention is needed, then analysis can
begin to devise the best solution. I find this to be a critical technique and
tell people all the time to train their intuition. It's a tool like any other,
to be honed and applied with skill.

------
DanielBMarkham
I have become very framework-adverse over the past few years as I've switched
from corporate programmer to startup programmer, and from OOP programmer to FP
programmer. Frameworks, especially OOP frameworks, tend to throw you into a
universe of nouns and verbs that can be quite complex and completely of the
author's making.

Everything works great -- until it doesn't. And then you're spending your
valuable time and brainpower learning about some framework instead of about
how to do the job you're trying to do. That's almost always a fail. As the
author mentions, it just gets worse: many times you're suckered into thinking
the answer for your problem is yet _another_ plug-in or framework. Now you
have even more stuff to keep track of.

OOP forces you to answer the question: what does this system represent? Once
you do that, then you're left with several metric tons of _how to represent it
well_ , which can lead to a _lot_ of stuff. FP, on the other hand, forces you
to answer the question: what, exactly, do you want the system to _do_?

As it turns out, unless you're strictly happy-path, the shortest way to an
answer is just to do the work. (Of course, nobody is saying to throw out core
libraries and start writing in machine code. But if you're bleeding to death
from paper cuts on WhizBang 4.0 that you just downloaded last week, you
screwed up.)

~~~
ourmandave
Esp. when WhizBang 4.0 is a complete rewrite of the codebase so throw out
everything you know about WhizBang 3.0.

~~~
DanielBMarkham
And your existing code already has workarounds for 3.0 which are invalidated
by the upgrade.

~~~
ourmandave
Don't worry about those workarounds. They're on features that were deprecated.
=)

------
jakejake
It's an interesting idea and one that I find to be true, but also complicated.

We're also trained as programmers not to reinvent wheels poorly. Or not
install a library to do something that is already supported by an existing
library you're already using. When you need to do something like, say, open a
new browser window, it's not that surprising to find a lot of posts about how
to do it with jquery.

I don't find it always bad. As long as you understand that you are using a
library, it's ok to go searching for its edges.

------
lfender6445
some of the pitfalls to intuition echo the side effects of programming by
coincidence, outlined in pragmatic programmer [https://pragprog.com/the-
pragmatic-programmer/extracts/coinc...](https://pragprog.com/the-pragmatic-
programmer/extracts/coincidence)

> Don’t code blindfolded. Attempting to build an application you don’t fully
> understand, or to use a technology you aren’t familiar with, is an
> invitation to be misled by coincidences.

------
d23
What a convoluted way of saying: "use frameworks and libraries, but not all
the time." Yeah, we get it, some people use libraries too much. Some people
reinvent the wheel too much. It's not too hard to understand that judgment is
key when deciding on one over the other.

Did I miss something, other than the attempt to coin the phrases "framework
negative space" and "framework intuitive space"?

~~~
girvo
> _not too hard to understand that judgement is key_

The state of most commercial codebases and a lot of colleagues I've worked
with, and my own code as well show that it is indeed that hard, I'd like to
argue. It's easy to state, difficult to do well.

------
ilostmykeys
The idea behind the ultimate user interface is for programming labguages to
dissolve over time leaving us with a natural conversation between man and
machine (and at some point the distinctions betweem man and machine will
dissolve)

While I understand what he and others have said, which is what I had believed
in the past, I think that the emphasis to think like a machine is heavily
misguided as we as users need to steer programming languages toward their
natural end (pun intended) which is where AI is headed.

But we're miles away from that so the argument he puts forward is temporarily
valid.

~~~
Sacho
I'm not sure this is a great idea. "Natural" communication is incredibly
inefficient and lossy. Note that academics communicate via meticulously
written papers, often involving very formal language. Business processes are
structured in worksheets, graphs, etc. What we seem to yearn for is the
ability for a computer to transcend our inability to express ourselves. I am
not so sure that is where we are(or should be!) headed. There is a certain
internal clarity when you manage to formalize your thoughts and arguments, to
test them for logical inconsistency and errors. Programming languages are an
inherent helper in that regard.

I think the biggest hurdle for programming languages currently is flow
control. Specifically, programming languages are mostly still written top-
down, leaving the implicit assumption that code is executed in the order you
read it in. This is obviously a false impression for pretty much any program
written nowadays. Inventing a new way to present code so that program flow is
easier to visualize, is in my opinion the "next big thing" in programming
language UI. This is something tools and concepts like UML have tried and
failed to revolutionize, so I'm not sure what the way forward really is.

~~~
ilostmykeys
Well, those are some deep thoughts. I appreciate the response:)

------
aalhour
I have been thinking about this since last night, and here are my thoughts. I
totally agree with Amjad on avoiding framing programming as an intuitive
activity. But on the other hand, framing it as a hard intellectual activity
for beginners is also troublesome. I am facing this now as I am helping my
younger brother in his Computer Science B.Sc. program, and taking different
isolated courses for few months two or three times a year makes
academic/intellectual studying for this industry without experience a hassle.
I keep listening to him and one of the most common complains he says is: "I
don't know how all these - OS, Automata, Java, Web, Algorithms and Math -
pieces fit together."

I think I can relate to this frustration/confusion because I went through it
when I was obtaining my B.Sc. degree but I cannot exactly remember nor
reproduce how I overcame this phase, or made sense of "it" all (for lack of a
better term).

Can you folks help me with this? How would you guide or help a beginner like
this person in programming?

------
freedrull
> I get a lot of questions from aspiring programmers on what’s the best tool
> or languages to learn. It’s almost always a premature question to ask.

I often think about how to answer this question to beginners as well. I think
he is right, the answer is probably "it doesn't matter".

~~~
TurboHaskal
Explain that to HR drones.

Tooling matters a lot if you want to be doing this for a living.

~~~
TeMPOraL
Every now and then spend a whole weekend (or just a week of after-work hours)
learning the new hot X to the point you can honestly say you know something
about it (it doesn't take that much for an experienced coder to learn a new
thing) and bam, you have another buzzword for your CV. Select as needed to get
through HR filters.

~~~
vonmoltke
Which is great, if the HR filter is simply "does X appear on the resume". It
it is "does X appear on previous job experience" you are still screwed.

~~~
marcosdumay
I don't have many resumes out there today, but I've never let HR people know
if any X belonged to what previous experience, or home project or what.

If they wanted to know, they'd have to ask me. Very few times, they did.

------
z3t4
I think this is the reason why "functional" programming results in fewer bugs
... or that you need to be really smart to do it. Or because you most of the
time is refactoring the program (from an imperative language).

