
How do we read code? - erjiang
http://blog.theincredibleholk.org/blog/2012/12/18/how-do-we-read-code/
======
Chris_Newton
Program comprehension is a fascinating (and IMHO very enlightening) field.

For anyone curious, I wrote up some of what I learned while exploring the
research a few years ago: <http://www.clarityincode.com/readability/> (I
apologise for the less than stellar formatting; I haven’t updated that page
for some time. I also apologise to the actual researchers I cited, if I’ve
dumbed down their work too much in aiming for a non-expert audience.)

For the eye tracking reported here, I wonder whether the early emphasis on the
top part was a combination of trying to figure out the data flow in the
between() function and then its significance to the wider program.

I think it would be interesting to compare the results with a similar eye
track of a program written with more emphasis on data flow rather than control
flow, e.g.,

    
    
        def between(numbers, low, high):
            return [n for n in numbers if low < n < high]
    
        def common(list1, list2):
            return [i for i in list1 if i in list2]
    
        x = [2, 8, 7, 9, -5, 0, 2]
        x_btwn = between(x, 2, 10)
        print x_btwn
    
        y = [1, -3, 10, 0, 8, 9, 1]
        y_btwn = between(y, -2, 9)
        print y_btwn
    
        xy_common = common(x, y)
        print xy_common
    

It might also be interesting to compare the results with a functional
programming language that expresses those ideas more concisely and/or with
tools like between() and common() as part of the standard library that
programmers would probably be familiar with.

Final thought: How much does the absence of a clearly marked starting point
(like a main() function in C) affect how a reader approaches unknown code in
Python? If this had been a C program, would the reader have aimed straight for
main() and then worked down from there to functions like between() and
common()?

~~~
dchichkov
I think it is very important to use code from real-life applications, rather
then synthetic examples like that, or like ones presented in the original
article.

Otherwise the results of that research would be biased towards comprehension
of meaningless chunks of code, rather that of a real thing.

~~~
Chris_Newton
I agree, but isn’t this the curse of most experimental research in programming
techniques?

Ideally, we’d compare like-against-like using industrial scale applications,
implemented by professional practitioners, controlling for everything except
what we’re trying to investigate. Finding opportunities to do that _in
practice_ is rather harder, because obviously most real software development
projects don’t get implemented twice by identical teams making exactly one
significant change in their approach.

What I thought was interesting here was that even though it is only a toy
program, there were still some patterns to how the reader explored it that
might suggest more general trends. As long as we understand the limitations of
the experiment and don’t overgeneralise any conclusions, isn’t some data still
better than random conjecture?

~~~
dchichkov
Yes, but why can't some regular code (from an open sourced project) be used in
the experiments? I'm sure there should be a way to do that.

It is just, that this synthetic code is broken. It doesn't read. There is no
flow in it. It just looks _from the first glance_ , like a non-interesting,
unimportant piece, that doesn't do anything, so there is no point in reading
it.

Does that sound convincing enough, that there is a big difference between
reading synthetic examples and actual code?

~~~
jejones3141
Yes, it does. There was a study of how well people could remember positions of
chess pieces on a board. The general public did equally badly for both random
arrangements and those that could actually appear in a game, but skilled chess
players did well remembering arrangements that could appear in a game, but
were no better than the general public on random configurations.

------
evincarofautumn
“One of the things that stood out to me in watching the video was how much my
mind seems to work like a computer.”

One of the things that stood out to _me_ in watching the video was how much
his mind seemed _emphatically not_ to work like a computer at all. His process
gave the appearance of a network self-training for a little while, then
simultaneously training and producing output. The more times an area of the
program was visited, the better the training, and consequently the longer it
could be retained. Notice how the results of calculations are “picked up” from
the source and “dropped” almost immediately into the output, as though they’re
heavy and difficult to hold on to!

------
synesthesiam
It's awesome to see that people are interested in my research! I've made a
blog post with another video and a few more details:
<http://synesthesiam.com/?p=218>

------
darrennix
I'll be very interested in findings on the terseness of code and its effects
on readability by experienced coders.

For example, the ternary operator is consider by many to be an elegant
solution to simple if statements but from a comprehension (and therefore bug-
finding) standpoint, is it superior? Also, how does this effect change as the
size of the codebase grows from a single page (as depicted in the video) to a
more complex class file.

    
    
      value = test ? (some_value * multiple) : false_value
      
      # vs
      
      if (test) {
        value = some_value * multiple
      } else {
        value = false_value
      }

~~~
tokipin
since you asked for my opinion, i think that ? : syntax is a failure.
languages where everything is an expression (usually functional languages)
have a much better implementation:

    
    
        value = if (test) {some_value * multiple} else {false_value}

~~~
InclinedPlane
How is this "much better"? Once you understand the syntax of the ternary
operator it's easy to read and easy to use.

~~~
dchichkov
Actually, it is much worse. Extra symbols certainly contribute to clutter and
reduce readability.

More to that, in my opinion, "code flow" in functional languages is completely
broken.

------
robomartin
Language has to be an important factor here. If I re-write this in APL --and
you know APL-- reading the solution is pretty linear:

    
    
        R ← data BETWEEN limits
        R ← ((data>limits[1])∧(data<limits[2]))/data
    
        R ← a COMMON b
        R ← (∨/a ∘.= b)/a
    
        x ← 2 8 7 9 ¯5 0 2
        y ← 1 ¯3 10 0 8 9 1
    
        x_btwn ← x BETWEEN 2 10
        y_btwn ← y BETWEEN ¯2 9
        xy_common ← x_btwn COMMON y_btwn
    

If you know APL the above pretty much reads like the palm of your hand.

Since APL is unknown to most, here's a quick explanation.

    
    
        R ← data BETWEEN limits
    

Dyadic function declaration. Takes two arguments.

    
    
        R ← ((data>limits[1])∧(data<limits[2]))/data
    

Let's break this up:

    
    
        (data>limits[1])
    

Takes the "data" vector and compares it to the first element in "limits",
which happens to be the "low" limit. You get a binary vector as the result
with a "1" anywhere the comparison is true and "0" otherwise.

If "limits" is 2 10:

    
    
        0 1 1 1 0 0 0 0
    

Now:

    
    
        (data<limits[2])
    

Does the same thing with the upper limit:

    
    
        1 1 1 1 1 1 1 1
    

Then:

    
    
        0 1 1 1 0 0 0 0 ∧ 1 1 1 1 1 1 1 1   
    

Performs a logical AND of the two binary vectors, resulting in a new vector:

    
    
        0 1 1 1 0 0 0 0
    

Finally:

    
    
        R ← 0 1 1 1 0 0 0 0/data
    

Selects elements from the "data" vector based on the values in the binary
vector and returns the result vector:

    
    
        8 7 9
    

Anyhow, that's why I think that language is important. If I wrote this in
Forth the pattern would be very different and the thought process required to
understand it more convoluted. Probably true as well for assembly.

------
dschiptsov
I don't think eye tracking and do any good, because recognizing of familiar
shapes and then mapping them to a familiar constructs and "structures" comes
first.

So, reading a code with familiar shape is one thing (that why Lisp has such
emphasis on the form of an expression, and Python put that into extreme),
while reading long.chains.of.unfamiliar.methods.is.another.)

Then comes recognition of a familiar zones (areas) of an expression, expecting
particular kind of sub-expressions here and there. Then match what you have
seen with known whole things.

Lets say it is a recursive process of reduction to something already known by
examining shapes, forms, and details.

So, shape matters. Small procedures, around ten lines matters. Naming matters,
and, especially, using one-letter, non-confusing (no meaning) names for just a
placeholders matters.

Let say that this solved in Lambda Calculus (by a naming strategy), and then
in Lisp (by shapping strategy) by accident.)

~~~
mcherm
I find it rather amusing, given the context, that I spent a long time (more
than 30 seconds) trying to figure out the match for your unbalanced
parenthesis. I don't think I would have done that if it hadn't been a
paragraph about Lisp: I kept going back thinking "Wait, I didn't realize I was
in a parenthetical expression... does this change the interpretation of what
is being said?".

~~~
kolektiv
Seconded. Surprising how distracting something so trivial is given recent
(unrelated) context. My mind immediately sprang to the conclusion that perhaps
this sentence is partially applied - a legacy of an evening spent writing some
interesting functional code for treating MIDI as observable sequences.

------
akshaykarthik
One of the things I noticed from the Eye Tracking was that this is very
similar to how we were trained to work through programming problems in high
school level computer science competitions. Some competition problems involve
tracing through code and determining output (of usually recursive functions)
but the 'proper' method we were taught is almost exactly how he describes his
thought process.

------
JoeAltmaier
100 times more frequently I read code I already read (or wrote) before. It
works differently. I page through, scanning the code, noting its "shape" as I
go without actually reading any lines.

Once I've found the place the problem might be, I start hand-executing (or
eye-executing?) for problems.

Does it really matter much how we read new code? We read many tims faster than
we write; re-reading is the norm.

------
gtani
Funny, i mentioned this a few days ago. I was going to do some googling on
other interesting code quality/dev productivity/language metrics, but, uh,
never got around to it.

It seems to me that in the first few passes on-sighting or sight reading code
(as climbers and pianists call it), you're looking for easy to comprehend
structures, and blocking off difficult to decipher, simultaneously, so maybe a
bimodal distributions at work here

<https://news.ycombinator.com/item?id=4926313>

------
Raphael_Amiard
> In programming language terms, I seem to be doing some kind of just-in-time
> compilation

Seems a lot more like abstract interpretation to me ! Would be a lot more
logical too :)

------
tejaswiy
Would it be possible for you to open source the eye tracking piece of this ?
I'm interested in doing this for photographs or paintings ...

~~~
lomendil
Eye tracker hardware is tricky and therefore usually expensive, but on the
software side there are good tools. I've used VisionEgg[0] with an Eyelink II
system and found it quite easy to deal with.

[0] <http://www.visionegg.org/>

~~~
dchichkov
Not that expensive. For a couple of hundred dollars you can build very nice
60-90fps IR eye tracking system. You'll need IR camera, IR tele lens (very
important), couple of IR sources and open source tracking software. See
<http://www.gazegroup.org/develop/>

There is also alternative approach. If you are just recording and don't need
interactivity - all you need, is a regular camera with zoom and a small
tripod. You can record on an SD card and synchronize the stream stream time in
post-processing. Just flash the monitor screen couple of times at the start
and the end of the recording, and play tracking alignment sequence.

Here's an attempt to do just that:
<http://www.youtube.com/watch?v=zJqKRL9D2qY>

------
3pt14159
Great post. Very relevant to this audience and very interesting research may
come from it. I look forward to seeing follow ups.

------
akennberg
"One of the things that stood out to me in watching the video was how much my
mind seems to work like a computer."

A more accurate way of thinking about this is: how much the computer works
like our mind. Not surprising considering that people build things based on
how we understand everything else. Whether it was done consciously or
subconsciously.

~~~
eholk
I read somewhere once that we tend to think of our mind in terms of whatever
the current most complex technology is. At one point we thought of the brain
as a fantastically complicated plumbing system, and later as a telephone
switchboard. I wonder if in another decade or so will the idea that our brains
work like computers seem as wrong as the idea that they work like plumbing
systems.

~~~
lomendil
I think that decade is now. Brains work nothing like computers. Well, as much
as they do like plumbing systems ;)

------
graue
I'm surprised no one pointed out that the answer given in the video is wrong.
The second line should be "10 0 8 1" and the third "8 9 0". The author writes
"10 0 9 1" and "9".

Perhaps it's not relevant to the experiment, but it seems worthy of mention at
least. Following along, I was second-guessing myself because my answer didn't
match...

------
route66
From <http://infoscience.epfl.ch/record/138586> :

 _... that modern languages, such as Scala, offer advantages as human
communication mediums. I describe an experiment, using an eye-tracking device,
that measures the performance of code comprehension._

------
mattmanser
That code is jarring to my eye because of the variable name, my eye tracking
would keep going back to 'winners'.

Why winners? What a bizarre variable name, to me anyway. Does anyone else use
that? I always go with `matches`, `retVals` or `returnValues` depending on the
language/IDE I'm using.

~~~
synesthesiam
I went back and forth over what to name this variable. I wanted it to convey
some meaning (winners are the items that passed or "won" the test), but not
give away too much.

Several people have commented on this, however, so I may just change the name
to "matches" or something. Thanks for the suggestion :)

------
columbo
This is very interesting. Have they considered using multiple languages
(functional, oo) as well as including 'garbage' languages (brainfuck) to track
differences? I'd also be curious to see how it compares to a standard word
problem (two trains leaving chicago etc).

~~~
synesthesiam
Here's an interesting paper where they compare their language (Quorum), Perl,
and a language with random ASCII tokens:
[http://ecs.victoria.ac.nz/twiki/pub/Events/PLATEAU/Program/p...](http://ecs.victoria.ac.nz/twiki/pub/Events/PLATEAU/Program/plateau2011-stefik.pdf)

(Spoiler alert: Perl doesn't fair much better than the random language!)

------
afhof
Makes me wonder if future languages will be designed to be read in serial.
Perhaps that will decrease the amount of time we need to look at code to
understand what it does.

~~~
saurabh
I would be interesting to do a similar study on some Factor code.

~~~
evincarofautumn
I would expect similar results. Factor and other stack-based languages are not
particularly linear, thanks to, well, factoring. Sure, data flow generally
runs from left to right and top to bottom, but word and tuple definitions are
just as scattered about as in any procedural language.

------
pilgrim689
Interesting how he starts from the components and then rolls up to the main
logic, whereas I would start at the main logic then drill down into the
routines it uses.

------
suyash
This video seems incorrect to me, I read code line by line one at a time most
of the time. The video seems to be showing that the eye is jumping all around.

~~~
Hemospectrum
Your eyes jump around a lot more than you think they do, no matter what you're
looking at. Your brain compensates for the effect somewhat, so it's impossible
to notice the full extent of the movement on a conscious level.

Some quick reading on the phenomenon: <http://en.wikipedia.org/wiki/Saccade>

------
bryceneal
Didn't this guy get it wrong? I think the answer should be, right ?

8 7 9

1 0 8 1

8

