
A test which predicts ability to program before the start of training - ColinWright
http://www.eis.mdx.ac.uk/research/PhDArea/saeed/
======
Jabbles
This paper is from 2006. Its results have since been shown to be not as
impressive as first thought:

<http://crpit.com/confpapers/CRPITV78Bornat.pdf>

 _We now report that after six experiments, involving more than 500 students
at six institutions in three countries, the predictive e ffect of our test has
failed to live up to that early promise._

~~~
ColinWright
Towards the bottom of the lunk-to page:

    
    
      Meta-analysis of the effect of consistency on success
      in early learning of programming is our fourth paper
      that was presented to Psychology Programming Interested
      Group (PPIG) Annual workshop on July 2009.  
    
      Abstract: ... The test does not very accurately predict
                    levels of performance, but by combining the
                    result of six replications of the experiment,
                    five in UK and one in Australia. We show
                    that consistency does have a strong effect
                    on success in early learning to program but
                    background programming experience, on the
                    other hand, has little or no effect.
    

[http://www.eis.mdx.ac.uk/research/PhDArea/saeed/SD_PPIG_2009...](http://www.eis.mdx.ac.uk/research/PhDArea/saeed/SD_PPIG_2009.pdf)

In short, the jury is still out, but there is some sort of effect showing up,
and it's going to need careful experiments to find out what's really going on.
It does seem to show promise, though, of finding a predictive effect.

------
tokenadult
Previous HN discussion of paper by same author:

<http://news.ycombinator.com/item?id=1145654>

Edit after kind reply by Colin:

I think the earlier discussion was especially valuable because of a checklist
of factors to look at in further research offered by one of the HN
participants who commented there.

<http://news.ycombinator.com/item?id=1146044>

Here in this discussion that began today, in another reply, a participant
posted

 _it seemed like there was a "programming gene" (not literally) that gave some
people innate ability, and without this you were sunk._

Some kind of "nature" explanation is what the human mind jumps to as an
explanation of individual differences, by innate human cognitive biases. To
tease out causation in such a matter would take what Robert Plomin and other
behavioral geneticists call a "genetically sensitive" study design, which is
lacking here. But there is such a huge body of human behavioral genetics
research already, that I can confidently predict what the finding would be for
anything properly called "programming ability":

1) Programming ability will be found to be heritable, that is that it shows
tighter correlations among closely related persons than among distantly
related persons, because all human behavioral characteristics are heritable,

and

2) programming ability is still exquisitely sensitive to environmental
influences (which perhaps are not well understood), especially influences of
"nonshared environment," the kind of environmental variation that results in
siblings brought up in the same household differing from one another.

A reference for both general propositions that is well worth reading is

[http://people.virginia.edu/~ent3c/papers2/Articles%20for%20O...](http://people.virginia.edu/~ent3c/papers2/Articles%20for%20Online%20CV/Johnson%20\(2009\).pdf)

Johnson, W., Turkheimer, E., Gottesman, I. I., & Bouchard, T. J. (2009).
Beyond heritability: Twin studies in behavioral research. Current Directions
in Psychological Science, 18, 217-220.

~~~
ColinWright
Thanks for that.

Some might cheer and say "Ha! Auto-hoist, petard-wise," but this latest
submission points to the page that documents the continuing saga. They have
addressed many of the criticisms leveled at the paper in the submission wo
which you refer, and carried out both further studies and a meta-analysis.

Hence I thought it worth re-visiting.

Having said that, the discussion on that submission is definitely worth
reading.

------
masnick
This is fascinating.

I observed the "double hump" in an intro computer science class I took in
college. Out of 20 people in the class, only 3 had prior programming
experience. As the semester progressed, some were able to learn basic Java
while others were not; aptitude seemed to be fairly independent from both
effort put into this class and performance in other courses.

Since then, I've wondered why this was the case -- it seemed like there was a
"programming gene" (not literally) that gave some people innate ability, and
without this you were sunk.

It seems like there would be more research on phenomena, but this is the first
work I've seen that hypothesizes why this happens.

~~~
DavidMcLaughlin
Using Java as the language to detect if someone has the "programming gene"
does not make any sense. There is so much cognitive overhead to writing a
simple program in Java compared to, say, Python that there will be many other
factors involved in why a student would end up performing badly.

In my own case I disliked Java so much in my first year of University because
I couldn't understand why there was so much effort just to get this little
black screen to print "Hello, World."

I would go on but I actually already wrote about this a couple years ago on
Reddit:
[http://www.reddit.com/r/programming/comments/84c90/i_fear_as...](http://www.reddit.com/r/programming/comments/84c90/i_fear_as_far_as_i_can_tell_that_most/c087mnf)

~~~
masnick
I absolutely agree.

This was in 2009, so today's obvious alternatives to Java (Ruby or Python)
were not as popular.

However, it should be noted that for the course I mentioned, the friction of
getting started in Java was not an issue: there were only 20 people in the
class and we all lived in the same dorm so people could get help when they
needed it.

~~~
imd
Sure, Python is even more popular today, but it showed up frequently in /.
"what language should I learn programming with" threads in at least as far
back as 2005. I seem to remember much earlier.

------
wccrawford
If I'm reading that right, it doesn't test the ability to get the 'correct'
answer, but rather that they are able to come up with a system that makes
sense (at least to themselves), and follow it faithfully and repeatedly.

I've met programmers who can't do this, and they were really bad... Suitable
only for mindless tasks where they were given all the logic and just had to
key it in and test it.

------
jamesrcole
Though this is only relative to current techniques for teaching programming,
and we shouldn't assume we know all there is to know about how to teach
programming.

There's reasons to think there's a lot of scope for improvement.

For a very general reason: that we don't really know that much about how our
brains and thinking work. Once we do we are likely to be much better at
identifying precisely why some people are poor at certain tasks, and at
knowing how to train people to improve their skills at those tasks.

[EDIT: wording]

------
skimbrel
_Told that there were three groups and how they were distinguished, but not
told their relative sizes, we have found that computer scientists and other
programmers have almost all predicted that the blank group would be more
successful in the course exam than the others: “they had the sense to refuse
to answer questions which they couldn’t understand” is a typical explanation._

I almost cracked up at this. As much as we'd like to say that's how we behave,
I'd say just about all of us are more likely to try to answer anyway.

------
Atropos
I maybe took 3 weeks of programming in high school 8 years ago, so I just took
the test for fun and here is what I don't understand: I picked the mental
model that the left variable ALWAYS takes the value of the right variable and
changes its own value permanently.

Of course this mental model makes all questions trivially easy, for example:

"Question 12": int a = 5; int b = 3; int c = 7; a = c; [a becomes the value 7
permanently] c = b; [c becomes the value 3 permanently] b = a; [b becomes the
value 7 permanently] a = 7; b = 7; c = 3

If I had picked a different mental model by chance that was more complicated
for my working memory, I find it quite likely that I would have made some
careless mistakes. So maybe some participants were not incapable of forming a
consistent mental model, but picked a very hard mental model to start with?

~~~
MostAwesomeDude
It has been suggested that the three core concepts of assignment,
encapsulation, and inheritance are unteachable/unlearnable and have to be
present _a priori_ in order to become a programmer with any competence.

I've seen people not be able to answer these questions, even after a term of
CS. :c

~~~
orillian
I would tend to agree. I have seen over the last couple years a number of
people who were passionate and dedicated to learning programming fail, and
fail hard.

Two years ago I was tutoring a young lady that had a serious passion for
programming, it was what she talked about constantly, and she was in college
taking a number of beginning courses in programming. She studied day in and
day out, working countless hours every night trying to "practice being a
programmer". She did every exercise given in the text books, and even had a
number of other textbooks on VB.NET (the language being taught in the course.)
and she studied them for hours every night. At the end of two semesters she
still could not program anything if the steps to the goal were not outlined
for her. She could recite to you the syntax of VB.NET and many of its
companion (.NET) libraries by heart, she could tell you what a piece of code
did when she looked at it (she was strangely adept at reading code and knowing
what it did) but to ask her to come up with the required steps to sort an
array of numbers from highest to lowest and her brain simply could not break
down the logic to a point where she could write it herself.

I also tutored a young fellow last fall that was in a similar predicament. He
was trying to pass a basic programming course so he could get an IT diploma
and this one beginner programming course was a requirement for this diploma.
He was driven to study, and was willing to put forth the effort to get through
this course, and he most likely would have (given multiple choice options
been) able to see the pattern. On the other hand ask him to assign the value
of “b” to ”a” and he would start to get lost. Ask him to assign “b” to “a” and
“a” to “c” and “d” to ”b” and he completely shuts down. No matter how many
examples he was given, no matter how hard I tried to show him alternate ways
of looking at it, the act of assigning values to variables and then
interacting with those variables to manipulate the values just would not
stick.

My opinion after dealing with a number of others like these two students is
that some people lack the ability to develop patterns. Not necessarily to see
patterns, but to come up with their own. They tend to have poor visualization
skills and they tend to find it hard to understand the pattern links when
given something like an analogy. I would say that the ideas behind
encapsulation and inheritance stem from assignment but they look at larger
assignment patterns than a simple a = b. Thus, if the individual lacks the
basic ability to conceive the pattern of assignment required to get to the
goal of the code they are not able to learn the ability to program.

O.

------
ctdonath
I've noticed a few key factors impacting ability to learn programming:

Hard work. Too many students just won't do the work. They'll show up, go thru
the motions, rattle something passable into place, and wonder why they didn't
pass (or almost didn't). The work has to go beyond just what is required into
self-motivated learning beyond.

Grokking variables. Many don't comprehend the nature of a variable, and thus
cannot succeed in using them. They know the right things to say about
variables, and may achieve passable results using them, but just don't get the
concept.

Task decomposition. Many can't get past "just do it", viewing the term as an
end instead of a beginning.

Strict adherence to rules. Whatever the rules of syntax are, that's what you
have to work with, and deviation just doesn't work.

Semantics. One must be capable of realizing that certain phrasing influences
perception.

Abstraction. Many can't.

Get those down and you'll learn. Miss one or more and it will be very hard.

------
jwingy
I'm of the school of thought that anything is trainable, including
programming, barring actual physical limitations. I think up to a point, even
a low point, programming is something almost any average person can do. I
think the brain has proven time and time again how adaptable and elastic it
is.

~~~
mkrecny
I agree that the brain is "adaptable and elastic". But it is still subject to
physical limitations. Probably to a much lesser extent than physique.

It's not a very motivational message but I really believe some types of
thinking are beyond the reach of certain people, due to the physical (read
genetically determined) constraints on their brain.

------
SoftwareMaven
I do wonder if there is a pedagogical solution to teach programming to people
in the "bottom hump". If you could identify them, you could take them down a
path that teaches the required thinking (problem decomposition, linear
behaviors, etc) without using any code at all.

Of course, ideally, this would be something everybody comes out of high school
with, and most of the top hump have probably intuited this from math classes.
Looking at my kids' curricula, though, there is nothing explicitly trying to
teach these concepts.

------
xutopia
"Untitled Document"... some people don't know how to do web.

------
PedroCandeias
I take issue with pretty much all tests of potential. In my humble view, all
they really do is measure how your experience, up to the point when the test
is taken, has prepared you for whatever it is you're testing for. A
sufficiently intelligent person should be able, given enough training, to
become proficient in any area of expertise.

I have no hard data to back this up, save for my own observations.

~~~
esrauch
I think the two are not as mutually exclusive as you present it to be.

Consider something like language acquisition for example; if you had a 30 year
old that already knew 3 languages they would almost certainly perform better
on an exam to test potential to learn new languages than a 30 year old that
knew one language.

If I am reading you correctly you are saying "that's not actually a legitimate
potential predictor; it just indicates what experience you have, not a true
measure of your capability going forward", but in reality if you have only
learned one language by the time you are 30, you are at a point where your
gene pool is not a great predictor. By the time you are 30, your lack of
previous experience has actually had a physical effect on your brain that
prevents you from being as good at language acquisition going forward.

It's not merely the fact that you have experience in a particular area that
improves your direct ability to answer questions; lacking experience actually
makes it more difficult to acquire the same levels of experience.

In other words, having experience makes you more able to gain experience; a
test that measures your experience is a legitimate indicator of future
learning potential. It's exactly along the line of colleges teaching you "how
to learn" and any practical skills that you gain along the way are largely
incidental.

------
NHQ
I don't believe that some people just can't learn to program. Certain types
need a certain type of teaching method. I failed my first programer course in
college (Java), and totally left the idea behind. Almost a decade later I
learned to program. I taught myself, with the help of the internet. In other
words, the internet supplied me a methodology for learning to program.

------
thurn
If anything, this test illustrates that assignment is a very confusing
concept, even more so if you pick a silly symbol like = for it. I wonder if
they tried to teach any of these students a more functional language that
doesn't rely on the idea of assignment.

------
tomlin
Not to be a dick, but broken graphic links and "Untitled Document" and this
test is to do what? Filter the herd?

------
suivix
Even if the test were accurate, an average but motivated guy would focus well
and quickly become better than the naturally gifted guy who slacks off reading
websites all day.

------
mikerg87
There is a test for midi-chlorians now?!??!

