
A Pedagogical Analysis of Online Coding Tutorials [pdf] - noelwelsh
https://faculty.washington.edu/ajko/papers/Kim2017CodingTutorialEvaluation.pdf
======
startupdiscuss
I wish the authors — or any authors — studied which courses actually deliver
results as opposed to what they ought to deliver.

For instance, immediate feedback ought to deliver learning. So they looked at
which courses give immediate feedback.

But it would be even better if somone looked at which learners actually
learned. You took the course now build xyz. Or pass a test or something.

There might be a difference in the sorts of “immediate feedback” that are
effective. It’s an indirect proxy of what you are after.

~~~
andyjko
Author of the paper being discussed here. I'd love to study what students
actually learn too! As we note in the paper, the analytical approach is a
limited, indirect proxy.

Unfortunately, as others have noted, doing this is intractably hard. We've
reached out to dozens of coding tutorial companies, but none want to share
their visitors' contact information (understandably). We've tried contacting
students on our campus and others to find people who've used tutorials, but
few have. If you have ideas about how to contact tutorial users, please share!

The even harder problem is measuring learning. There are essentially no
reliable, valid measures of any knowledge of programming (exams in courses are
mostly unreliable, invalid measures). It's something we're working on in my
lab, but it will take years, as it has in math and physics education.

Andy Ko Associate Professor University of Washington
[http://faculty.washington.edu/ajko](http://faculty.washington.edu/ajko)

~~~
startupdiscuss
I love that the author of the paper responded to this. Thank you HN, and Andy.

I suspected it was hard or else, I assume, you would have done it. But I
thought it was hard because getting everyone to take a standardized test
before and after the courses would be hard, not because even the contact
information was not available!

Also, what do you mean exams are unreliable? Is there something you can point
me to. I did not know that was the case.

------
sagefy
I like a great deal that one of the considerations the authors make is
adapting to prior knowledge. (3.1) describes what they looked for in detail.
In their summary, "None provide personalization based on prior coding
experience or learner goals, other than rudimentary age-based
differentiation." Its hard to overstate the importance of prior knowledge.
Prior knowledge has some of the highest effect sizes in educational research.
In other words, what you already know going into it is going to determine how
much you learn. It's rare for other factors to have the same impact on the
results as prior knowledge. And yet, even given decades of learning science
research, its pretty rare to see much of any attempt in Western learning to
account for prior knowledge. It would be a refreshing change of pace to see
some of these coding tutorials work in preemptive assessment to figure out
where to focus the learner before getting into the "how" of it.

Disclaimer: I'm working on sagefy.org

~~~
anonymous5133
For your website, sagefy, how do you plan on being able to "assesses" the
student's prior knowledge?

It seems to be a double-edge sword if you plan on using assessments (the only
way I can think of to assess prior knowledge) to do it because people want
immediate satisfaction. Unfortunately if you told people to take a 25 question
pre-quiz before starting the course I am sure most would simply leave the
website.

~~~
yorwba
You could make the pre-quiz part of the "immediate satisfaction" by
calibrating the questions so that the learner can feel good about their
current knowledge without making them spend a long time solving difficult
problems. Making a quiz fun is probably easier than doing the same for the
actual learning part, which might require much more time before seeing any
satisfying results.

Another way to assess knowledge would be to give the learner topics of varying
difficulty to select from; it's unlikely that they'd choose something so easy
that it'd be boring and in case the topic is too hard, there should be an
escape route to something easier.

------
ListeningPie
Instead of lecture based courses, I’d like to see more an adaptable textbook
style, where a learner creates their own path of learning.

A simple example is a user manual online,
[http://support.casio.com/global/en/wat/manual/5411_en/](http://support.casio.com/global/en/wat/manual/5411_en/)
. The topic of each section is clearly indicated and when parts of another
section are mentioned a direct link is given. In this way the information is
presented as the reader wants it.

The book Code Complete: A Practical Handbook of Software Construction,
provides a brief description at the start of book of what a chapter contains
and suggests reading orders based on what the reader is looking for. These are
examples of the reader choosing order.

Examples of relevant questions is in Maths textbook Mathematical Techniques:
An Introduction for the Engineering, Physical, and Mathematical Sciences 4th
Edition , where sections contained check your understanding boxes. In these
boxes a problem relaying on the information in the section was given followed
by a full worked out solution. The sections ended with more problems related
to the immediate section followed by questions that involved knowledge from
pervious chapters. In the way same code examples could be presented and at the
end of chapters a build your own program challenge.

------
dreamache
I've created probably over 100 video courses, hundreds of tutorials (written
and video), for networks like TutsPlus (Envato), DigitalTutors, Pluralsight,
LinkedIn Learning/Lynda.com, and my own site / yt channel.

I've always been a huge proponent of example-based learning, and less about
theory and generalized concepts.

Still, though, I think the online tech ed space has a lot of growing up to do.
In time, we'll all wise up to provide better experiences for learning. Until
then, for the most part, we're stuck with:

1\. Google: "How to do X in Y" 2\. Watch 3 different videos or read 3
different articles 3\. Apply it. 4\. Run into a snag. 5\. Google /
Stackoverflow

~~~
dofly
To me untangling abstractions is always more difficult then building them up
from the given premises. In other words, I believe generalizing from a given
example to any level of abstraction is a straightforward process, but
descending from built-up heights of abstraction down to the level of examples
is a bit more involved.

------
keyle
That was a very interesting read. As developers, let's face it, university was
a tiny fraction of our knowledge. Most of it is ad-hoc real life problem
solving, followed by seeing someone else's solution and saying "wow that's
neat" and getting better that way.

------
kingkongjaffa
We're teaching recipes, when really we should be teaching self-sufficiency.

Too many new learners are stuck unable to make the leap out of the browser
based IDE and into running their own code locally on a machine or on a server.

~~~
andyjko
Author of the paper being discussed here. Re: self-sufficiency, our evidence
agrees. You might be interested in this paper from my lab, which successfully
attempts to teach self-regulation skills to novices:

Loksa, D., Ko, A. J., Jernigan, W., Oleson, A., Mendez, C. J., & Burnett, M.
M. (2016, May). Programming, problem solving, and self-awareness: effects of
explicit guidance. In Proceedings of the 2016 CHI Conference on Human Factors
in Computing Systems (pp. 1449-1461). ACM.
[http://faculty.washington.edu/ajko/papers/Loksa2016Programmi...](http://faculty.washington.edu/ajko/papers/Loksa2016ProgrammingProblemSolving.pdf)

Andy Ko Associate Professor University of Washington
[http://faculty.washington.edu/ajko](http://faculty.washington.edu/ajko)

~~~
kingkongjaffa
Thanks for the link I'm just reading your paper now, using Cloud9 IDE is an
interesting choice which I am not sure is beneficial to the learners..

I think self-sufficiency should include building a local environment you can
work work in without having to rely on some SAAS / Cloud service based IDE.

------
qwerty456127
IMHO the problem with coding tutorials is they usually target people that have
little to no idea about programming essentials and teach them just the very
basics like key primitive concepts (what classes, variables and functions are
etc), language syntax, frameworks elementary basics etc.

Is there even a single tutorial that a self-taught undergraduate coder (who
has already went through a number of simple tutorials, can code a number of
languages like Python and C# and use GitHub) can take to learn how are real-
world apps made, what do professional (professional ≠ veteran, it just means a
person who earns their living doing it, working in a team usually) coders
actually do all day long at their workplaces, what are they meant to know, how
to solve interview puzzles and everything enough to go and get a coding job
with reasonable ease?

As far as I know many companies need coders that can join a team and start
writing/maintaining plain ordinary code, not necessarily brilliant code, just
code that is readable, maintainable, reasonably fast and reliable and solves
simple tasks it (a particular piece of code) is meant to. As far as I know a
huge portion of people can potentially learn to do this with ease (you don't
need IQ>150 for this, 100 is enough, even 90 will do which means the majority
of the population). But people don't know how. And knowing a programming
language syntax and being capable to code a simple script for yourself isn't
enough: 99% of people who speak a language and can write a simple essay have
no idea about how to write a novel (not necessarily a best-seller, even a
mediocre one) - they are to be taught to structure it, introduce and develop
characters, describe places and situations nicely, bake an idea in, maintain
suspense, organize their writing process etc - it seems arcane to them while
it essentially is just a number of plain and simple principles that can be
outlined in a single manual. If only somebody having both the knowledge
(skill) and the talent to explain would bother to write one and wouldn't write
a huge tome practically obsolete by the release date instead.

~~~
gonzo41
The problem would be picking all the bits of tech, so much of what you're
talking about is stuff you learn when you start working with your first big
ball of mud.

Lets use java for simplicity. We're going to do a non trivial app thats web
based so we dip into the exciting worl of java EE.

Learn about maven eclipse jax-ws jax-rs cfx tools wsi tools Hibernate stuff DB
stuff. Probably need to throw spring boot or cdi onto the pile of things to be
a little familiar with.

Then you need to teach how app servers work. getting one setup. which
computer/ os etc.

Zooming in take the build process, 'Setting up jenkins' Thats a big tutorial
all on its own. you'd need to know Git or SVN or one of the others.

then you step into building a CI pipeline, Building a test suite, functional,
unit and integration test. building reporting ontop all of that.

That would be mega tutorial series. And the moment you finished it, people
would start pulling strips off it for being out of date. Or they would attack
it for language and tooling choice.

~~~
qwerty456127
Perhaps a good list of all the subjects with detailed commentaries could do.
One could go find tutorials on specific parts then given the knowledge o what
to look for and what's that about. May be short (yet illustrating the way it's
done in the real world) examples on particular subjects could be included.

~~~
gonzo41
That would be one way to go, and probably a pretty good one.

But yeah, the churn in the industry is crazy, so it would be some work to
compile a list that's coherent and orthodox that didn't go out of date the
moment you published.

------
cglee
This is something we spend a lot of time thinking about at Launch School. You
need the right pedagogy for the right phase of learning
([https://launchschool.com/pedagogy](https://launchschool.com/pedagogy)).

Most article writers aren't differentiating their audience enough, making
their articles difficult to read for most people. There is a huge variance of
"beginners", for example. Further, most developers who write these articles
don't have a holistic learning framework in mind. Lastly, I'm seeing far too
many technical tutorials just for SEO or attracting eyeballs lately. These
articles are great for attracting attention but are typically technically
shallow and add to the confusion rather than help.

~~~
anonymous5133
Agree. Most of the tutorials I see seem to be written entirely to generate ad
revenue rather than actually be a quality educational resource. Google's
algorithms only promotes this type of garbage.

------
kevintb
Very interesting read! I've not heard of "Gidget, Lightbot, Code Hunt, and
tutorials provided by Code.org" before, but the paper seems quite
comprehensive.

~~~
codezero
This paper was written by the PhD advisor of the creator of Gidget, which
appears to also have been funded by the NSF and Microsoft (like the research
in the paper).

Probably not a big deal, but I thought it was an interesting connection.

~~~
andyjko
Author of the paper being discussed, and of Gidget here (along with
collaborators). The work was primarily funded by a grant I received from the
National Science Foundation:

[https://nsf.gov/awardsearch/showAward?AWD_ID=1240786&Histori...](https://nsf.gov/awardsearch/showAward?AWD_ID=1240786&HistoricalAwards=false)

After the research, I raised some money through a Microsoft Research grant to
support some additional work and pay for bandwidth for a decade. Just $4,000
left until I have to pull the plug :( If you like Gidget, please make a
donation to support our skeleton crew.

Andy Ko Associate Professor University of Washington
[http://faculty.washington.edu/ajko](http://faculty.washington.edu/ajko)

~~~
codezero
Thanks for the hard work Andy!

------
muzani
Summary: Every online coding tutorial is a lecture. They might pretend to be a
little interactive, but the learning path is railroaded, and doesn't cater to
individual progress.

~~~
anonymous5133
The reason it is like that is because it is very efficient to have lecture-
based learning. The second you add individualized progress to the course it
begins to become a huge programming logic problem.

~~~
visarga
It's not necessarily so difficult. There are blindspots, they need to make
more videos to cover them. Take a survey of questions in the forum and make
new videos covering those topics. These videos could be optional. The learner
can then watch as much as needed, thus, the course adapts to the learner.

~~~
anonymous5133
It is difficult because to develop the programming logic side of it involves
knowing programming and how to implement a system that takes in student data,
analyzes it and then creates a new learning path/customized learning
materials.

Most people creating educational content simply do not have those skills.

------
j45
I've been reconnecting to the academic edtech space for the past year. After
spending the past 20 years in both industry and academic online learning,
academic behaviour remains a bit odd:

1) Academics in education talk about the use of technology in education but
often don't know how to use technology in depth.

Story: In attending the World Conference of Online Learning - held at a very
capable hotel - there were no power bars for laptops, and no sessions were
recorded. Less than 5% of 1500 attendees (academics) carried or used a laptop
or tablet for notes. Typing fast to take notes received comments. This was one
of the leading conferences (apparently) in the world. Over 100 countries, 500
speakers, and 200 sessions.

2) Some academics seem to regularly trash education that technology may be in
fact delivering well for students beyond the Academic's ability to understand.

One talk I attended at this conference was a Ph.D. essentially skewering how
Duolingo is not a good way to learn a language. The arguments may have had
merit, I found it hard to take the talk seriously, well, because the millions
of users reporting a positive learning experience with tools like Duolingo is
well known. Learning to speak a language is incredibly valuable for many
people.

3) If academic's don't use the right terminology...

It's a little baffling why academics use the word pedagogy when speaking about
the use of technology in education.

Pedagogy is how children learn, not adults. Andragogy is how adults learn.

"Pedagogy", "improving learning experiences", "improving student outcomes"is
on a long chain of lean startup type groupspeak where people continue doing
what they do but sprinkle a little innovation dust on what they're doing. Want
to upset an academic? Correct them in using Andragogy instead of Pedagogy as a
non-academic.

4) Innovation in Education + Technology.

Technologists can learn education easier than educators will learn technology.
These two groups need to come together, or as I suspect, educators competent
in understanding the capabilities and possibliites of technology will need to
exist in higher learning. The rate of change in society may be hard to keep up
with a tenure seeking mindset.

5) K-12 educators seem a lot more tech friendly than post-secondary educators.

Post-secondary educators seem often threatened by technology replacing them,
and fail to recognize that improved digital learning experiences will be like
better text books.

If our educators aren't levelling up on tech use, they wont be able to help
the students of the future.

~~~
andyjko
As a Ph.D. in higher education with both a computer science and learning
science background, I can confirm many of your claims. Most education
researchers and practicing educators have a weak grasp of computing and it's
powers. And indeed, many faculty outside of computing are terrified of
computing. There's no context for them to learn about it. I respond to these
troubling trends by collaborating with people in other disciplines, sharing my
expertise. Right now, I'm working with faculty in our College of Education, in
Psychology, and many interdisciplinary researchers with backgrounds in
Feminism, Anthropology, and other fields. Interdisciplinary research will save
academia; the National Science Foundation has shifted to a strong expectation
of interdisciplinary teams. I'm hopeful.

Other claims you made are wrong. That people report positive learning
experiences with Duolingo is not evidence that they can read, write, or speak
a language. In fact, there are several studies of Duolingo that show the exact
opposite: there's massive variation in learning outcomes, and it really only
supports the most motivated and resourceful of learners. But that's what
research is for: to test assumptions with data.

Andy Ko Associate Professor University of Washington
[http://faculty.washington.edu/ajko](http://faculty.washington.edu/ajko)

