Author of the paper being discussed here. I'd love to study what students actually learn too! As we note in the paper, the analytical approach is a limited, indirect proxy.
Unfortunately, as others have noted, doing this is intractably hard. We've reached out to dozens of coding tutorial companies, but none want to share their visitors' contact information (understandably). We've tried contacting students on our campus and others to find people who've used tutorials, but few have. If you have ideas about how to contact tutorial users, please share!
The even harder problem is measuring learning. There are essentially no reliable, valid measures of any knowledge of programming (exams in courses are mostly unreliable, invalid measures). It's something we're working on in my lab, but it will take years, as it has in math and physics education.
I love that the author of the paper responded to this. Thank you HN, and Andy.
I suspected it was hard or else, I assume, you would have done it. But I thought it was hard because getting everyone to take a standardized test before and after the courses would be hard, not because even the contact information was not available!
Also, what do you mean exams are unreliable? Is there something you can point me to. I did not know that was the case.
Indeed it would, but it's probably really difficult to research this, as you need a large population, and there will be plenty of external factors acting on those populations. You'd ideally want to use students with the same (or similar) age, and similar ability levels. Where do you find a suitably large number of those from? Then you'd need to remove any external influences - the effect of a good or bad teacher, for example.
That's not to say it can't be done, but is a much larger and more expensive piece of research.
Excellent idea. However, I think the main reason they don't review by that aspect is because there simply aren't any of those types of courses. The courses coming out of the traditional universities (on edx or coursera) seem to be very watered down versions of real university courses. For the free online, independently created courses they seem to be catering to the "immediate stimulation" crowd rather than the small group of people who want truly comprehensive online courses.
I agree though that project-based learning where the student takes an active role in the creation and development of projects is crucial. If you think about it, when the student enters the real world/work force, everything they do will be focused around projects of various size.
Courses are usually tied to revenue. (Some) post-secondary institutions can run like small countries. Other schools are hopefully starting to take the above approach and stay that way.
I like a great deal that one of the considerations the authors make is adapting to prior knowledge. (3.1) describes what they looked for in detail. In their summary, "None provide personalization based on prior coding experience or learner goals, other than rudimentary age-based differentiation." Its hard to overstate the importance of prior knowledge. Prior knowledge has some of the highest effect sizes in educational research. In other words, what you already know going into it is going to determine how much you learn. It's rare for other factors to have the same impact on the results as prior knowledge. And yet, even given decades of learning science research, its pretty rare to see much of any attempt in Western learning to account for prior knowledge. It would be a refreshing change of pace to see some of these coding tutorials work in preemptive assessment to figure out where to focus the learner before getting into the "how" of it.
This is great. I have been tinkering with the idea of building something that helps people learn something in the most efficient way possible. I have been pretty frustrated with most learning materials for almost anything, especially when you have limited time and a lot of interests. Keep up what you are doing and hopefully I will be able to share something after a couple of months.
For your website, sagefy, how do you plan on being able to "assesses" the student's prior knowledge?
It seems to be a double-edge sword if you plan on using assessments (the only way I can think of to assess prior knowledge) to do it because people want immediate satisfaction. Unfortunately if you told people to take a 25 question pre-quiz before starting the course I am sure most would simply leave the website.
You could make the pre-quiz part of the "immediate satisfaction" by calibrating the questions so that the learner can feel good about their current knowledge without making them spend a long time solving difficult problems. Making a quiz fun is probably easier than doing the same for the actual learning part, which might require much more time before seeing any satisfying results.
Another way to assess knowledge would be to give the learner topics of varying difficulty to select from; it's unlikely that they'd choose something so easy that it'd be boring and in case the topic is too hard, there should be an escape route to something easier.
Instead of lecture based courses, I’d like to see more an adaptable textbook style, where a learner creates their own path of learning.
A simple example is a user manual online, http://support.casio.com/global/en/wat/manual/5411_en/ . The topic of each section is clearly indicated and when parts of another section are mentioned a direct link is given. In this way the information is presented as the reader wants it.
The book Code Complete: A Practical Handbook of Software Construction, provides a brief description at the start of book of what a chapter contains and suggests reading orders based on what the reader is looking for. These are examples of the reader choosing order.
Examples of relevant questions is in Maths textbook Mathematical Techniques: An Introduction for the Engineering, Physical, and Mathematical Sciences 4th Edition
, where sections contained check your understanding boxes. In these boxes a problem relaying on the information in the section was given followed by a full worked out solution. The sections ended with more problems related to the immediate section followed by questions that involved knowledge from pervious chapters. In the way same code examples could be presented and at the end of chapters a build your own program challenge.
I've created probably over 100 video courses, hundreds of tutorials (written and video), for networks like TutsPlus (Envato), DigitalTutors, Pluralsight, LinkedIn Learning/Lynda.com, and my own site / yt channel.
I've always been a huge proponent of example-based learning, and less about theory and generalized concepts.
Still, though, I think the online tech ed space has a lot of growing up to do. In time, we'll all wise up to provide better experiences for learning. Until then, for the most part, we're stuck with:
1. Google: "How to do X in Y"
2. Watch 3 different videos or read 3 different articles
3. Apply it.
4. Run into a snag.
5. Google / Stackoverflow
To me untangling abstractions is always more difficult then building them up from the given premises. In other words, I believe generalizing from a given example to any level of abstraction is a straightforward process, but descending from built-up heights of abstraction down to the level of examples is a bit more involved.
That was a very interesting read. As developers, let's face it, university was a tiny fraction of our knowledge. Most of it is ad-hoc real life problem solving, followed by seeing someone else's solution and saying "wow that's neat" and getting better that way.
We're teaching recipes, when really we should be teaching self-sufficiency.
Too many new learners are stuck unable to make the leap out of the browser based IDE and into running their own code locally on a machine or on a server.
I am a self-taught programmer. I see what people in this thread are getting at, and I believe I stumbled my way into that realm of knowledge by accident.
Learning multiple languages (of different paradigms) led me to implement code in one language to one in another. Even though the first set may have been heavily based on some tutorial, when translating my own code, I had to be able to reason about it, break it apart, and re-compose it.
This line of thought helped make me more capable of talking about and thinking through my code as an idea, where the implementation was just an exercise.
If anyone else is teaching themselves, I would advise the following:
+ Learn a high-level OOP language
+ Learn a high-level functional language
+ Translate programs between the two, focusing on really working with the respective paradigm.
+ Move on to something lower level. Many people will say go with C to learn manual memory management. I found Rust interesting, if not difficult at first. Up to you, just seek lower-level understanding.
+ Learn an assembler, fake or real. Shenzhen I/O and Human Resource Machine are good introductions to this.
+ Build skills in your languages through katas. Learn to think and write in a language with minimal reference. This is simple, persistent, repetitive practice.
+ Read about a discrete math you find interesting. You will likely not be as strong at math as educated people. Discrete maths can help at least give you some good flair.
+ Seek constant review from peers.
+ Don't always agree with peers. Find ways to defend your viewpoint. Good arguments- not "Java sucks." Be able to say why you have the right idea.
+ Learn simple circuits. They help you think atomically. There are even sites that have virtual breadboards so that you can experiment and learn for free.
Recipes can work, but they are just the first step. Learning how to "remix" a recipe and make it your own is the next. It's a long journey, for sure. I would agree that online resources that are the most visible are generally very entry-level and hand-holdy. I would also add that there is a vast amount of deep knowledge freely available for those wishing to go further.
I relate to this so hard--about two years elapsed between when I "learned" Javascript on Codecademy and when I first used it in a real project. These courses teach you syntax, and do a pretty shit job at even that.
Author of the paper being discussed here. Re: self-sufficiency, our evidence agrees. You might be interested in this paper from my lab, which successfully attempts to teach self-regulation skills to novices:
Loksa, D., Ko, A. J., Jernigan, W., Oleson, A., Mendez, C. J., & Burnett, M. M. (2016, May). Programming, problem solving, and self-awareness: effects of explicit guidance. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 1449-1461). ACM.
http://faculty.washington.edu/ajko/papers/Loksa2016Programmi...
Thanks for the link I'm just reading your paper now, using Cloud9 IDE is an interesting choice which I am not sure is beneficial to the learners..
I think self-sufficiency should include building a local environment you can work work in without having to rely on some SAAS / Cloud service based IDE.
IMHO the problem with coding tutorials is they usually target people that have little to no idea about programming essentials and teach them just the very basics like key primitive concepts (what classes, variables and functions are etc), language syntax, frameworks elementary basics etc.
Is there even a single tutorial that a self-taught undergraduate coder (who has already went through a number of simple tutorials, can code a number of languages like Python and C# and use GitHub) can take to learn how are real-world apps made, what do professional (professional ≠ veteran, it just means a person who earns their living doing it, working in a team usually) coders actually do all day long at their workplaces, what are they meant to know, how to solve interview puzzles and everything enough to go and get a coding job with reasonable ease?
As far as I know many companies need coders that can join a team and start writing/maintaining plain ordinary code, not necessarily brilliant code, just code that is readable, maintainable, reasonably fast and reliable and solves simple tasks it (a particular piece of code) is meant to. As far as I know a huge portion of people can potentially learn to do this with ease (you don't need IQ>150 for this, 100 is enough, even 90 will do which means the majority of the population). But people don't know how. And knowing a programming language syntax and being capable to code a simple script for yourself isn't enough: 99% of people who speak a language and can write a simple essay have no idea about how to write a novel (not necessarily a best-seller, even a mediocre one) - they are to be taught to structure it, introduce and develop characters, describe places and situations nicely, bake an idea in, maintain suspense, organize their writing process etc - it seems arcane to them while it essentially is just a number of plain and simple principles that can be outlined in a single manual. If only somebody having both the knowledge (skill) and the talent to explain would bother to write one and wouldn't write a huge tome practically obsolete by the release date instead.
The problem would be picking all the bits of tech, so much of what you're talking about is stuff you learn when you start working with your first big ball of mud.
Lets use java for simplicity. We're going to do a non trivial app thats web based so we dip into the exciting worl of java EE.
Learn about
maven
eclipse
jax-ws
jax-rs
cfx tools
wsi tools
Hibernate stuff
DB stuff.
Probably need to throw spring boot or cdi onto the pile of things to be a little familiar with.
Then you need to teach how app servers work. getting one setup. which computer/ os etc.
Zooming in take the build process, 'Setting up jenkins' Thats a big tutorial all on its own. you'd need to know Git or SVN or one of the others.
then you step into building a CI pipeline, Building a test suite, functional, unit and integration test. building reporting ontop all of that.
That would be mega tutorial series. And the moment you finished it, people would start pulling strips off it for being out of date. Or they would attack it for language and tooling choice.
Perhaps a good list of all the subjects with detailed commentaries could do. One could go find tutorials on specific parts then given the knowledge o what to look for and what's that about. May be short (yet illustrating the way it's done in the real world) examples on particular subjects could be included.
That would be one way to go, and probably a pretty good one.
But yeah, the churn in the industry is crazy, so it would be some work to compile a list that's coherent and orthodox that didn't go out of date the moment you published.
I would be surprised if any single tutorial could cover all that. I know of courses/tutorials that are intended to cover each part of that, although I don't know if anyone's assessed them as a whole. e.g how real-world apps are made: https://www.edx.org/course/software-service-uc-berkeleyx-cs1...
yes. Most of the "good" online courses are simply very watered down versions of real university courses. No one close to replacing the real university course (and probably done intentionally as well).
This is something we spend a lot of time thinking about at Launch School. You need the right pedagogy for the right phase of learning (https://launchschool.com/pedagogy).
Most article writers aren't differentiating their audience enough, making their articles difficult to read for most people. There is a huge variance of "beginners", for example. Further, most developers who write these articles don't have a holistic learning framework in mind. Lastly, I'm seeing far too many technical tutorials just for SEO or attracting eyeballs lately. These articles are great for attracting attention but are typically technically shallow and add to the confusion rather than help.
Agree. Most of the tutorials I see seem to be written entirely to generate ad revenue rather than actually be a quality educational resource. Google's algorithms only promotes this type of garbage.
Very interesting read! I've not heard of "Gidget, Lightbot,
Code Hunt, and tutorials provided by Code.org" before, but the paper seems quite comprehensive.
This paper was written by the PhD advisor of the creator of Gidget, which appears to also have been funded by the NSF and Microsoft (like the research in the paper).
Probably not a big deal, but I thought it was an interesting connection.
Author of the paper being discussed, and of Gidget here (along with collaborators). The work was primarily funded by a grant I received from the National Science Foundation:
After the research, I raised some money through a Microsoft Research grant to support some additional work and pay for bandwidth for a decade. Just $4,000 left until I have to pull the plug :( If you like Gidget, please make a donation to support our skeleton crew.
Summary: Every online coding tutorial is a lecture. They might pretend to be a little interactive, but the learning path is railroaded, and doesn't cater to individual progress.
The reason it is like that is because it is very efficient to have lecture-based learning. The second you add individualized progress to the course it begins to become a huge programming logic problem.
It's not necessarily so difficult. There are blindspots, they need to make more videos to cover them. Take a survey of questions in the forum and make new videos covering those topics. These videos could be optional. The learner can then watch as much as needed, thus, the course adapts to the learner.
It is difficult because to develop the programming logic side of it involves knowing programming and how to implement a system that takes in student data, analyzes it and then creates a new learning path/customized learning materials.
Most people creating educational content simply do not have those skills.
I've been reconnecting to the academic edtech space for the past year. After spending the past 20 years in both industry and academic online learning, academic behaviour remains a bit odd:
1) Academics in education talk about the use of technology in education but often don't know how to use technology in depth.
Story: In attending the World Conference of Online Learning - held at a very capable hotel - there were no power bars for laptops, and no sessions were recorded. Less than 5% of 1500 attendees (academics) carried or used a laptop or tablet for notes. Typing fast to take notes received comments. This was one of the leading conferences (apparently) in the world. Over 100 countries, 500 speakers, and 200 sessions.
2) Some academics seem to regularly trash education that technology may be in fact delivering well for students beyond the Academic's ability to understand.
One talk I attended at this conference was a Ph.D. essentially skewering how Duolingo is not a good way to learn a language. The arguments may have had merit, I found it hard to take the talk seriously, well, because the millions of users reporting a positive learning experience with tools like Duolingo is well known. Learning to speak a language is incredibly valuable for many people.
3) If academic's don't use the right terminology...
It's a little baffling why academics use the word pedagogy when speaking about the use of technology in education.
Pedagogy is how children learn, not adults. Andragogy is how adults learn.
"Pedagogy", "improving learning experiences", "improving student outcomes"is on a long chain of lean startup type groupspeak where people continue doing what they do but sprinkle a little innovation dust on what they're doing. Want to upset an academic? Correct them in using Andragogy instead of Pedagogy as a non-academic.
4) Innovation in Education + Technology.
Technologists can learn education easier than educators will learn technology. These two groups need to come together, or as I suspect, educators competent in understanding the capabilities and possibliites of technology will need to exist in higher learning. The rate of change in society may be hard to keep up with a tenure seeking mindset.
5) K-12 educators seem a lot more tech friendly than post-secondary educators.
Post-secondary educators seem often threatened by technology replacing them, and fail to recognize that improved digital learning experiences will be like better text books.
If our educators aren't levelling up on tech use, they wont be able to help the students of the future.
As a Ph.D. in higher education with both a computer science and learning science background, I can confirm many of your claims. Most education researchers and practicing educators have a weak grasp of computing and it's powers. And indeed, many faculty outside of computing are terrified of computing. There's no context for them to learn about it. I respond to these troubling trends by collaborating with people in other disciplines, sharing my expertise. Right now, I'm working with faculty in our College of Education, in Psychology, and many interdisciplinary researchers with backgrounds in Feminism, Anthropology, and other fields. Interdisciplinary research will save academia; the National Science Foundation has shifted to a strong expectation of interdisciplinary teams. I'm hopeful.
Other claims you made are wrong. That people report positive learning experiences with Duolingo is not evidence that they can read, write, or speak a language. In fact, there are several studies of Duolingo that show the exact opposite: there's massive variation in learning outcomes, and it really only supports the most motivated and resourceful of learners. But that's what research is for: to test assumptions with data.
I think you are 100% spot on. I used to run an edtech startup that sold directly to government run schools. When I attempted to introduce the technology to the school I was often met with hostility because they said my technology was designed to replace teachers rather than empower them. Needless to say, I got out of the business quick and now I focus my efforts on building technology to allow self-learners to directly learn.
The existing government run education model is completely broken. They are openly hostile to innovative technology. It is a matter of philosophy and it cannot be fixed. If the system is broken, it is time to have it replaced with something better. Eventually, something better will come a long and citizens/taxpayers will begin to question why this old model still exists, just like how people ask why certain businesses still exist.
If there is any innovation to happen in education, it will not come from traditional universities or schools.
For instance, immediate feedback ought to deliver learning. So they looked at which courses give immediate feedback.
But it would be even better if somone looked at which learners actually learned. You took the course now build xyz. Or pass a test or something.
There might be a difference in the sorts of “immediate feedback” that are effective. It’s an indirect proxy of what you are after.