
What Makes Code Hard to Understand? - denzil_correa
http://arxiv.org/abs/1304.5257
======
groovy2shoes
We need to have more research like this. Programmers spend a lot of time
arguing about features, syntax, paradigm, methodology, etc., but no one really
has any scientific proof. I laud the researchers who are coming up with new
paradigms and extending current paradigms, but we _really_ need researchers
going behind and finding what's beneficial and what's not in terms of human
factors.

~~~
macspoofing
>We need to have more research like this.

There is a lot of research done in this and other areas of Software
Engineering. Problem is that it's usually locked away behind a huge paywall.
If you're not affiliated with some university, you essentially have no access
to it.

~~~
groovy2shoes
Another problem is that the research hasn't been collected into a manager-
friendly tome like the Gang of Four book. The software industry is overflowing
with snake oil, and we need to replace it with science if we want to see this
turn into a _real_ engineering discipline. I'm tired of bikeshedding and holy
wars -- show me the science!

~~~
zwegner
If it involves studying human behaviors, it's probably not going to be
science. I would prefer the holy wars over some manager telling me it's been
"proven" that one side is correct.

~~~
groovy2shoes
Good point. The thought of a manager saying that sent a chill down my spine.

------
Arjuna
Interesting paper.

The following code is probably my favorite, "What will this program output?"
example. This is taken from the Quake III Arena code in "q_math.c" [1].

Note line 561 [2]. Non-obfuscated code, and one is left wondering... just
what, exactly, is going on at that line?

I understand that the point of the paper is to analyze code without having
comments to help, but it serves as a reminder to me that commenting is
important in helping not only other developers understand the code, but to
help myself when revisiting code particulars that may have faded from memory.

I find this to be a great example when I hear, "I don't comment; the code
itself is self-documenting."

    
    
      552 float Q_rsqrt( float number )
      553 {
      554     long i;
      555     float x2, y;
      556     const float threehalfs = 1.5F;
      557
      558     x2 = number * 0.5F;
      559     y = number;
      560     i = * ( long * ) &y; // evil floating point bit level hacking
      561     i = 0x5f3759df - ( i >> 1 ); // what the fuck?
      562     y = * ( float * ) &i;
      563     y = y * ( threehalfs - ( x2 * y * y ) ); // 1st iteration
      564 //  y = y * ( threehalfs - ( x2 * y * y ) ); // 2nd iteration, this can be removed
      565
      566 #ifndef Q3_VM
      567 #ifdef __linux__
      568     assert( !isnan(y) ); // bk010122 - FPE?
      569 #endif
      570 #endif
      571     return y;
      572 }
    

[1] [https://github.com/id-Software/Quake-III-
Arena/blob/master/c...](https://github.com/id-Software/Quake-III-
Arena/blob/master/code/game/q_math.c#L552)

[2] <https://en.wikipedia.org/wiki/Fast_inverse_square_root>

~~~
stephencanon
I would probably write it this way today (well, actually I wouldn’t write it
today, I would instead use the reciprocal square-root instructions provided by
all the common architectures, but let’s pretend):

    
    
      static uint32_t toRepresentation(float x) {
        uint32_t rep;
        memcpy(&rep, &x, sizeof x);
        return rep;
      }
    
      static float fromRepresentation(uint32_t rep) {
        float x;
        memcpy(&x, &rep, sizeof x);
        return x;
      }
    
      static float rsqrt_linearApproximation(float x) {
        uint32_t xrep = toRepresentation(x);
        uint32_t yrep = 0x5f3759df - xrep/2;
        return fromRepresentation(yrep);
      }
    
      static float rsqrt_newtonRaphsonStep(float x, float y) {
        return y*(1.5f - (0.5f*x)*y*y);
      }
    
      float Q_rsqrt(float x) {
        float y = rsqrt_linearApproximation(x);
        y = rsqrt_newtonRaphsonStep(x, y);
        y = rsqrt_newtonRaphsonStep(x, y);
        return y;
      }
    

The only semi-cryptic thing about it, to someone versed in numerics, is line
with the magic number 0x5f3759df. While I would expect a professional to be
able to quickly re-derive it and understand the intention, I would still write
a comment, along the lines of “We approximate the floating-point
representation of 1/sqrt(x) with a first-order minimax polynomial in the
representation of x.”

All that said, I tend to write a _lot_ of comments. Much of the code I write
professionally is math library code, and it’s not uncommon for me to have a
ratio of several paragraphs of error analysis or correctness proofs to a few
lines of code.

~~~
InclinedPlane
It's not just semi-cryptic, it's magic. Your example above includes "xrep/2"
instead of i>>1, but this does nothing to increase the understanding of what
the fuck is going on here.

This is a floating point number and some deep juju bit twiddling is happening
with it, it's not just being divided by 2. The exponent is being effectively
divided by 2 and the fractional part of the mantissa is also being effectively
divided by 2. Except in the case where the representation of the exponent
ended in a 1, in which case that 1 is shifted down into the fractional part of
the mantissa. Exactly why this is ok and doesn't hurt the approximation is a
subject that could fill a report.

~~~
stephencanon
Floating point representations are approximately logarithms. It should be
clear to anyone with working knowledge of floating point numerics that
dividing by two and negating the representation of a floating point number
corresponds to taking the reciprocal square root of the number. That's not
magic, that's math.

The constant simply accounts for the bias in the floating point representation
and balances the error over the domain of interest. Again, not magic.

There is no "bit-twiddling" at all.

~~~
lmkg
Explaining the details of the bit-twiddles does not make it any less bit-
twiddling. And explaining the details behind the opaque steps does not make
them any less opaque to the uninitiated.

You're using bit operations on a data type with a non-trivial bit structure.
That's pretty much the definition of bit-twiddling. The fact that blunt bit
operations on such a finely-defined structure can correspond to useful
mathematical operations is non-obvious.

~~~
mrbrowning
The fast inverse square root trick is clever, but this is right on. It's
bypassing the provided interface for FP operations and mucking around with
implementation details. That's not to say that that's a bad thing, or that
floating point numbers aren't a leaky abstraction, or that the real world
doesn't demand this sort of optimization from time to time. But it's still
true.

------
cecilpl
"We were surprised to ﬁnd a signiﬁcant eﬀect of Python experience on the
probability of making a very speciﬁc error (base = 0.13, OR = 1.44, p < 0.5).
Instead of [8, 9, 0] for their last line of output, 22% of participants wrote
[8] (for both versions). After the experiment, one participant reported they
mistakenly performed the “common” operation on lists x btwn and y btwn instead
of x and y because it seemed like the next logical step"

I made this mistake too. In fact, even after reading that most people wrote
[8] instead of [8,9,0], it took me a solid minute or so to figure out how
anyone could possibly think the correct answer was [8,9,0].

This is something I've noticed happens very often while debugging. I make
certain assumptions about the code and even when staring it right in the face
it's hard to see where my assumptions differ from the code as written.

My assumption in this case was "why would you calculate the between lists if
you weren't going to use them right away".

~~~
qu4z-2
I like the ff => ﬀ ligature that you've managed to paste from the .pdf. :)

EDIT:

I also made the mistake, but in their defence the between lists are used
straight away: They're printed. But the code seems to do three unrelated
things in a row, so I'd expect an implementation like their one to have three
separate function calls if there's no dependency between them.

------
acqq
My "between" implementation would be like this:

    
    
        def between( a, low, high ):
            r = []
            for x in a:
                if low < x < high:
                    r.append( x )
            return r
    

or even

    
    
       def between( a, low, high ):
           return [ x for x in a if  low < x < high ]
    

instead of their:

    
    
        def between(numbers, low, high):
            winners = []
            for num in numbers:
                if (low < num) and (num < high):
                    winners.append(num)
            return winners
    

The more variable is local the shorter it should be. Also by using
mathematical notation you display "functionality" instead of distracting
readers with the terms you coined. "Winners"?

~~~
mattmanser
Honest question, why? What do you think you gained? Why use short variable
names in this day and age given text editor auto-completion, refactoring
tools, etc.

The very plurality of `numbers` conveys so much more meaning than `a` in the
method signature. Is that a Python idiom, to use `a` to mean array as a
parameter? What happens if you don't use `a` until line 23 of the function,
far from its original declaration?

Being someone who likes reading random code in random languages, I find single
or 2 letter variable names a horrible curse to the uninitiated of a problem
domain. It means you have an extra step figuring out a program, what the
abbreviations even mean in that particular domain.

The only exception being when it's the standard way of using the syntax of a
language (i in a for loop for example).

Obviously these code examples are trivial, but when you start getting a
slightly longer function, it starts becoming hard to remember what's what
half-way down a function or where they even came from if they're just
virtually meaningless single or dual letters.

~~~
btilly
_What happens if you don't use `a` until line 23 of the function, far from its
original declaration?_

Then you have horribly violated the previous poster's note that the more local
a variable is, the shorter its name should be.

 _I find single or 2 letter variable names a horrible curse to the uninitiated
of a problem domain. It means you have an extra step figuring out a program,
what the abbreviations even mean in that particular domain._

It is a trade-off. If you master that domain, you'll be using that all the
time, and you'll get the savings all the time.

You need to amortize the cost of memorizing the notation over the expected
usage of that notation. Programmers, by profession, have to use a lot of
notations from a lot of fields, and therefore benefit from using notations
that require a minimum of memorization. But that is not always the right
trade-off.

As I pointed out in <https://news.ycombinator.com/item?id=5157539>, _The rule
is not that you need long or short variables. You need meaningful ones. A
short variable name is inherently ambiguous, which can lead to confusion and
mistakes. Thinking through anything with a long-variable name abuses your
working memory, limiting how complex your thoughts can be._

------
wting
When reading the paper's verbose versions, I have to act as a human compiler
and keep track of state which leads to mental errors. Contrast this with a
shorter version reducing the LOC count from 21 to 6:

    
    
        x = [..]
        y = [..]
        between = lambda ls, low, high: (n for n in ls if low < n < high)
    
        x_between = between(x, 2, 10)
        y_between = between(y, -2, 9)
        xy_common = set(x).intersection(y)
    

Experienced users made mental stakes because _they skim code and make
assumptions from previous sections._ This can be solved by taking advantage of
a language's expressiveness to clarify intent.

------
agentultra
I've come to believe that it is the cognitive dissonance between the
programmer and the machine. The language used often obscures what is actually
executed. Often we find this useful but without proper cues we can write
programs that make the divide between program and process really wide.

Personally I find declarations about properties, relationships, and categories
of things easier to understand than descriptions of the tedious processes that
calculate them. The former give me the ability to reason about the program
elements in a logical way. The latter requires me to become a stack machine
and execute the program in my head... a process that is error-prone and full
of false assumptions.

eg: how many people believe that "all arrays are just pointers," in C? When is
an array not like a pointer?

Research like this is good. Are there any studies that go beyond trying to
trick programmers with trivial programs in the Algol family and look at the
difference in performance between categorically different styles? I mean
languages that are declarative vs imperative vs functional vs concatenative in
nature. I think that would be very interesting to read.

 _updated for clarity_

~~~
jeremyjh
> eg: how many people believe that "all arrays are just pointers," in C? When
> is an array not like a pointer?

Ok I know this wasn't your point but I got stuck on this and am just curious
to know: what you were thinking of here? One thing that occurs to me is that
in a recursive function you will run out of stack space a lot faster if you
are using arrays rather than allocating from the heap. I don't think that is
what you were referring to though, hence the question.

~~~
agentultra
Well my example was meant to support how programming languages can create a
cognitive dissonance between what the programmer _thinks_ will be executed vs.
what is actually executed.

One area where I think there is a high dissonance is in how C uses the same
syntax for defining arrays, indexing into arrays, and referencing pointer
offsets. Other examples are the array decomposition rules, array function
parameters, etc. Even experienced programmers get tripped up by them:

<http://c-faq.com/~scs/cgi-bin/faqcat.cgi?sec=aryptr>

------
jdlshore
Interesting study. Here's a summary of their results:

1: "between". Experienced programmers were more likely to incorrectly assume
that the results of earlier calculations would be used in a later calculation.

2: "counting". All participants, regardless of experience, were more likely to
assume that a statement separated by vertical whitespace was outside of a
loop, when it was actually still inside the loop. (Note that the programming
language is Python, which doesn't have a loop termination token.)

3: "funcall". Whitespace between operators (e.g., 3+4+5 vs. 3 + 4 + 5) had no
effect.

4: "initvar". No interesting result. (This one seems to have been mis-
designed.)

5: "order". Respondents were slower when functions were defined in a different
order than they were used. Experienced programmers didn't slow down as much.

6: "overload". Experienced programmers were more likely to be slower when
faced with a "+" operator used for both string concatenation ("5" + "6") and
addition (5 + 6), rather than used for just concatenation.

7: "partition". There was a result, but I don't think we can draw meaningful
conclusions from it.

8: "rectangle". Calculating the area of a rectangle using a function that took
tuples (e.g., "area(xy_1, xy_2)") took longer for programmers to understand
than a function that took scalars (e.g., "area(x1, y1, x2, y2)"). Calculating
the area using a class (e.g., "rect = Rectangle(0, 0, 10, 10); print
rect.area()" took the same amount of time as using scalars, despite being a
longer program.

9: "scope". No conclusive result.

10: "whitespace". Using horizontal whitespace to line up similar statements
had no effect on response time. (There was another result relating to order of
operations that deserves further study, but it wasn't thoroughly tested here,
so I don't think it's conclusive.)

Note that all programs were exceedingly simple (the longest was 24 lines), so
be cautious applying these conclusions to real-world work.

------
incision
Forgive my ignorance - IANAP.

Why can't/don't we have more tools along the lines of App Inventor [1] that
provide a visual structure for programming?

The vast majority of time I've spent writing anything has been spent
struggling with specific syntax, not any particular function of the code.

Coding feels like trying to manipulate a clear model that exists in the
machine by spelling my intentions out one letter at a time while peering
through a straw.

1: <http://appinventor.mit.edu/>

~~~
kbenson
Possibly because while we have a rich ecosystem of tools for dealing with text
in myriad form and ways, and have conditioned ourselves over the years on how
to read text in a general form, I don't think the same can be said for more
graphical representations of programs.

While a representation that uses graphical cues in lieu of textual ones may be
easier to learn and understand initially, the disadvantage of not being able
to use the _thousands_ of utilities for searching, refactoring examining code
are then less useful, if not entirely useless.

P.S. In a way, idiomatic structure to code puts a graphical representation on
the textual form. Programmers learn to recognize this. Python goes as far as
to enforce it.

~~~
incision
_> Possibly because while we have a rich ecosystem of tools for dealing with
text in myriad form and ways, and have conditioned ourselves over the years on
how to read text in a general form, I don't think the same can be said for
more graphical representations of programs._

Yes, I completely agree.

To go off on a bit of a tangent, I wonder if it's just a matter of the right
technology and implementation? Is there an analog to the rise of touch
interfaces waiting to happen in the way we record, manipulate and relay
information?

 _> While a representation that uses graphical cues in lieu of textual ones
may be easier to learn and understand initially, the disadvantage of not being
able to use the thousands of utilities for searching, refactoring examining
code are then less useful, if not entirely useless._

That's a good point. I was thinking of this purely from the standpoint of
creating as an individual, not maintaining or sharing.

 _> P.S. In a way, idiomatic structure to code puts a graphical representation
on the textual form. Programmers learn to recognize this. Python goes as far
as to enforce it._

Certainly, learning that word (idiomatic) was a boon to my understanding of
code.

------
dmlorenzetti
_Programmers often feel like the physical properties of notation have only a
minor inﬂuence on their interpretation process. When in a hurry, they
frequently dispense with recommended variable naming, indentation, and
formatting as superficial and inconsequential._

It's ironic that the authors of this statement can't be bothered to typeset
their opening quote marks in the right direction-- a typical noobie mistake in
TeX.

~~~
denzil_correa
I am not sure if I understand you correctly. Is it ok if I ask you for an
example?

~~~
ajanuary
At the beginning of the second paragraph on page 2:

    
    
        The second question is: ''How are programmers
    

In LaTeX, `` is used for open quotes, and '' for close quotes. [0]

[0] <http://www.maths.tcd.ie/~dwilkins/LaTeXPrimer/QuotDash.html>

~~~
denzil_correa
Thanks, I do know how open quotes are used in LaTex though. I couldn't
understand the initial comment. Thanks for the clarification. Up-voted.

------
randscreen
From the paper: "This program prints the product f (1) ∗ f (0) ∗ f (−1) where
f (x) = x + 5. ... Most people were able to get the correct answer of 60 in 30
seconds or less."

Unless I'm losing my mind, the correct answer for the code as stated is 6 * 5
* 4=120, not 60. It would be 60 if f (x) = x + 4.

~~~
synesthesiam
You're correct. This was a (very) unfortunate typo. We had changed from x + 5
to x + 4 during the design phase to try and lessen the mental burden without
making it too easy.

Thanks to you and a few others, I'm making corrections and will be uploading a
fixed version of the paper soon.

------
synesthesiam
Lots of great discussion here! If anyone is interested, I added a blog post
about the paper ([http://synesthesiam.com/posts/what-makes-code-hard-to-
unders...](http://synesthesiam.com/posts/what-makes-code-hard-to-
understand.html)) and put the data up on github
(<https://github.com/synesthesiam/eyecode-tools>).

------
keefe
"expectation-congruent programs should take less time to understand"

this statement feels tautological, I don't think there's a single correct set
of expectations

~~~
happimess
It's not a tautology. If the brain's benefit from "branch prediction" is low,
then some other factor (e.g. length of code) could overwhelm the effects of
predictability. So they're making a genuine claim, which is that expectation-
congruence is the most important factor.

My experience bears this out. In most of my professional life, I've abided by
coding standards, even if I consider the standards imperfect. That's because
of the very real improvements to the team's productivity when code does what
it looks like it does.

~~~
keefe
not quite a tautology, I was being a bit facetious... imho the concept of
understanding is tied up with the meaning of expectation, so much so that by
definition the code I don't understand doesn't do what I expect - if it did,
then I'd understand it!

------
raverbashing
I was thinking along those lines the other day.

I guess the biggest hurdle is that people essentially think "functionally" not
procedurally

Because we fundamentally think x = x _2 in terms of "the value is being
doubled" without thinking of all the steps needed and everything that can go
wrong there.

The article is certainly interesting, from what I can quickly glance, naming
is _very important _

~~~
friendly_chap
I think functional programming and immutability goes hand in hand, so I
wouldn't call x = x*2 an example of the functional thinking.

With the same logic, C is functional because you don't care about the bit
fiddling in x++...

~~~
raverbashing

      Yes, x = x*2 is not (y = x*2 would be better)
    

But expecting a while loop to exit as soon as the condition is met (as opposed
as while the loop block finished) is a common case

------
kruhft
Code is hard to understand because most programmers only learn how to write it
and never take the time to learn how to read it. If you can't read something
with ease, it will always be hard to understand.

~~~
nooneelse
Also, if you never learn to read it, you can't begin to learn to write for the
reader.

