
For Online Courses, Questions Over How Success is Measured (2014) - reedwolf
https://www.texastribune.org/2014/01/21/low-completion-percentages-moocs-prompt-debate/
======
ChuckMcM
I am always amazed when people make comments like this:

 _" The results of the University of Texas at Austin’s first full-semester
foray into massive open online courses, or MOOCs, are in."_

 _" Professor Michael Webber’s “Energy 101,” which had an enrollment that
peaked at around 44,000 students, had 5,000 receive a certificate of
completion — about 13 percent of the roughly 38,000 students who ultimately
participated."_

So let's unpack this a bit. Professor Webber created a class called "Energy
101" and processed 5,000 students through it to completion. Your typical 100
level undergraduate class might have anywhere from 50 to 200 students in it.

UT Austin this year had 8690 freshman _total_.

So assuming the largest possible class of 200, this professor in one semester
taught the equivalent of 25 semesters of 200 student classes, in _one_
semester.

Why should we care that 32,000 people signed up and then said "Woah, really
don't have the time to commit to this right now?"

It is one thing to move to a campus dorm and start studying some topic. It is
quite another to click a few links on a web page and say, "Yeah, that could be
an interesting class."

~~~
pixelperfect
Also, taking but not completing MOOC shouldn't count as a failure. I've
enrolled in ~8 courses on Coursera and never completed any of them, but I
still gained a lot of value from what I learned in those courses.

------
gamesbrainiac
I think this emphasis on completion rates are a problem. I take a lot of
MOOCs, but not all the content in said MOOCs are important. I just watch the
parts of courses that are important or relevant to what I am doing, and call
it a day.

For people learning from non-conventional sources, they are used to using bits
and pieces from different online sources and building up their own courses and
their own paths.

This fascination with completion rates is simply our propensity to value what
we measure instead of measuring what we value.

~~~
Chathamization
Very much this. I think a large issue with MOOCs is that they're too focused
on emulating the structure of a university class - focus on lectures,
community built around specific classes, focusing on the metric of course
completion rather than on attaining skills, etc. People who successfully teach
themselves things (and right now, there are many people using the internet to
teach themselves things) often have a very different approach.

It's interesting that so many people have trouble even considering that there
might be other ways to approach education.

~~~
hos234
People who "successfully" teach themselves are overrated (good examples being
Trump or Zuckerberg). They might be successful in a few dimensions (purely out
of self discipline and above avg energy levels) but totally miseducated in
many others because people don't know what they don't know. Without the right
guides, feedback processes, environment and time(lots of time) most people
usually end up taking all kinds of wrong turns.

Education not something intuitive, even if there are many paths. If it was we
would just be leaving kids in libraries.

~~~
jdc
Your point that self-education often tends toward narrowness is well-taken,
but I strongly disagree with the implication that any significant number of
autodidacts are overrated in the way that your two examples are.

------
turadg
We grappled with rates when I was engineering manager for the team at Coursera
that created course statistics for our partners.

The completion “rate” does not matter because the denominator is a very noisy
signal. It’s a count of enrollments and what does “enrollment” signal in an
online course, when it takes two clicks from a tweet.

There are other complexities but completion rate in the “are MOOCs worth
offering” question is silly and tired.

What matters is people getting value they otherwise wouldn’t.

~~~
prepend
How did you end up measuring the course? Completion by students who logged in
at least once? Hours spent in course? Cost of course per completion?

It seems like we would want to compare what is learnedC that’s how I value.
But not sure how you would test that.

------
tony
These are for MOOCs, not online degree courses, though.

There's no certifications with these. You can't get an accounting, law, or
medical degree, bachelor/associate certificate, or anything you'd need
professional accreditation for.

Maybe the completion rate is similar for people who buy books. I'd end up
getting a book thinking it'd be an investment as it would be good as learning
a subject. The next day I didn't really want to spend 12 straight hours
learning Visual Basic.

Also these aren't like auditing a class and listening in on a real lecture
either. Just from the list for UT Austin (a very good school):
[https://www.edx.org/school/utaustinx](https://www.edx.org/school/utaustinx)

There are university lecturers people listen to all the time though, through
the Great Courses / Teaching Company. Those sort of fit the leisure-learner
path and don't require logging into some system for it.

~~~
the_jeremy
> Maybe the completion rate is similar for people who buy books.

There's an even lower bar to entry: all the MOOCs I've participated in have
been free.

~~~
strken
They might be free, but all the MOOCs I've participated in have required a
weekly time commitment to complete them, which is a sort of hidden cost
completely separate from money. I've started the Stanford ML course on
Coursera three times now, and each time some real-life thing has come up and
taken me out of action for a couple of weeks.

~~~
learnstats2
That's just what free means, isn't it? The cost is separate from money.

I don't think it's hidden that you have to study in order to learn.

~~~
strken
Studying isn't the problem, deadlines are the problem. If you could work at
your own pace then I'd guess the completion rate would be substantially
higher.

------
toofy
The fact these classes dont allow one to preview the first few courses before
registering has to influence these stats.

I’ve done a number of them over the last couple of years and the fact that I
can’t preview has caused me to abandon so many of them. If I don’t like my
professor, the way the course is organized, or whatever, I’ll abandon the
course and go find some other source for the same information which does a
better job resonating with me.

And I know I’m not alone in doing this, off the top of my head, I know at
least 3 workmates and at least a number of friends who do the same thing.

The completion rate would have to shoot up much higher were they to allow
people to know what theyre getting into ahead of time. And strangely, these
companies have a much better ability to do this than traditional schools.

------
computator
Can I bounce an idea off Hacker News readers?

How about doing live events at which you simply play -- on an overhead screen
-- the very best videos or courseware from the web on selected technical
topics. You get the best possible presentation and the chance to mingle with
people who are keen on the same subject. You'd still need a person to act as a
moderator or host, but there would be no official speakers. Would you go to
such a thing?

This idea came about because I was at a seminar recently with multiple
presenters on technical topics. At least 90% of what was said is available on
the web. And the average presentation was awful. Yet, the auditorium was
_packed_. Two hundred people showed up to hear publicly available information
presented by mediocre speakers, none of whom were experts in the field (just
people interested in the topic).

This is a regularly held seminar and people come. A lot people like attending
a talk even if we could have stayed home and watched a better video on the
same technical topic. I think part of it is that some people absorb material
better in a classroom setting and the other aspect is the chance to socialize
and talk with like-minded people.

~~~
aty268
I think the issue isn't that people aren't learning properly, it's more due to
with the recognition of work and time required for no tangible reward. It's
the same reason it's really hard for most people to sit down and learn coding
on their own. It's not because lectures are hard to watch, or they aren't
learning effectively, it's because you aren't getting anything tangible from
it. Bootcamps are the same way, you get a "certificate" but it doesn't really
have much value, and people know it.

Getting a degree from a good school (I go to UT Austin actually) is different.
That's why people go to college.

~~~
fouc
A CS degree covers more than just programming. A bootcamp focuses on
programming only.

Let's assume the CS degree has on average 1 out of 5 courses per semester that
directly involves practicing programming, and not just filling up the
requirements on liberal arts, and math, etc. 13 weeks * (3 hours lectures per
week + 5 hours homework per week) * 2 semester per year * 4 years = 832 hours.

So a full-time developer bootcamp would need to be at least 5 months long to
match.

Most are only 2.5 to 3 months long, but there are some 6 month long developer
bootcamps. The short bootcamps might not have as much value, but still a good
minimum start to development.

------
mathnmusic
It's interesting how content discovery has remained ad-hoc despite the arrival
of ed-tech platforms like Coursera or EdX. People still prefer to ask for best
books, courses, podcasts, cheatsheets on Slack/Discord/Twitter etc.

I think this has to do with the conflict of interest. Any platform that
creates its own educational content, necessarily becomes bad at content
_curation_. For ex, 3Blue1Brown has amazing series on linear algebra and
calculus but we don't find Coursera pointing to it.

~~~
mileycyrusXOXO
3Blue1Brown made videos for Khan Academy for a while but ultimately decided to
take his career in a different direction. He talks about it on the first
episode[1] of the Numberphile podcast, which if you are a fan of 3Blue1Brown
I'd highly reccomend giving it a listen.

[1]:
[https://www.numberphile.com/podcast/3blue1brown](https://www.numberphile.com/podcast/3blue1brown)

------
georgeburdell
I have a PhD, earned while on a prestigious national fellowship, and graduated
with highest honors in undergrad. I graduated earlier this decade. I have
enrolled in a single MOOC to keep my skills current in my field, and I
couldn’t finish it. Maybe it’s because I work now, but I simply couldn’t find
the time to complete the lab assignments when only putting in 10 hours a week.
My sample size is low, but I get the impression that some of these MOOCs
basically demand that you have traditional student-level time and focus to
complete them. Is that wrong? Perhaps not, but 4% then becomes unsurprising
for a completion rate.

------
bythckr
The issue is with how MOOCs is used and what the creators expect from it.

I have signed for many MOOCs but not completed even 1 till date. But I got the
knowledge I needed and got the task done. To me MOOCs is an organized google
search or detailed stackoverflow/wikipedia.

Plus my extreme aversion towards "exams", is also a reason. Never found an
incentive to get a paper that tells I know something. Plus, despite having
certificates from school & university, I don't remember the stuffs I learnt. I
forgot most stuff after writing the exams. Sadly, some even before writing the
exam.

------
qyirius
Maybe because the definition of completion for some of the courses includes a
lot of things that aren’t valuable to the learner. I consider being able to
skip lectures one of the benefits of online courses

~~~
jdnenej
Also because some of them don't let you really see what the content is unless
you sign up. I have signed up for free online courses before just to check
what they contain.

------
barry-cotter
That’s because overwhelmingly people don’t care about learning, they care
about certification. The completion rates for online degrees are vastly higher
because people get that piece of paper that marks one as an intelligent, hard
working conformist. They have to care enough to apply, and pay for a course. I
don’t know how much higher the completion rate is for EdX or Coursera students
that pay for verified certificates over auditing courses but if it’s less than
four times as high I’d be surprised.

[https://www.usnews.com/education/online-
education/articles/2...](https://www.usnews.com/education/online-
education/articles/2015/01/30/experts-debate-graduation-rates-for-online-
students)

> While many believe that there is a huge difference between online and on-
> campus completion rates, [Russell Poulin]’s research suggests the difference
> is slight. Based on a survey completed by more than 200 North American
> school officials in 2013, Poulin found that course completion rates averaged
> three to five percent better for on-campus courses than for online courses.

[https://www.edsurge.com/amp/news/2019-06-06-moving-
from-5-to...](https://www.edsurge.com/amp/news/2019-06-06-moving-
from-5-to-85-completion-rates-for-online-courses)

> Today, 2U reports completion rates of up to 88 percent for their online
> degree programs. Harvard Business School’s online programs claim similar
> success, with completion rates of 85 percent.

> At Acumen, where I design online courses, we’ve also been offering selective
> cohort-based programs for the past year that achieve completion rates of 85
> percent. That’s a far cry from five years ago, when only 5 percent of the
> students were finishing the MOOCs I was designing.

> How have instructional designers collectively moved the needle so
> dramatically on completion rates? Unsurprisingly, some of the biggest
> drivers of these improved metrics include making people pay for online
> programs, increasing the selectivity of courses, and adding program managers
> and teaching assistants to follow up with learners.

~~~
swiley
I’m not sure people don’t care about learning, I’m certain I’m not the only
one that’s gone through a lot of stuff on OCW on my own. I think people are
more likely to jump through specific hoops if they get something out of it,
that has little to do with learning though.

~~~
barry-cotter
I didn’t say people don’t care about learning, I said that overwhelmingly they
don’t. There is a tiny minority’s who value learning for its own sake. They
are most of the 4% who complete MOOCs with neither entrance criteria nor
certification. They are the people who read complex works of literature, in
whatever genre, outside an educational context. They’re the people who don’t
just occasionally go to an art gallery but can speak knowledgeably about
influences, styles and periods.

They’re nerds of whatever persuasion. Normal people work on their three hours
of tv a day, nerds learn about Nabokov or the Assyrians or decide that it
would be better to work through a modern introduction to algebra than Euclid
for fun.

------
manwe150
2014, but I find it interesting the title focuses on the completion
percentage. Instead, it seems it could have equally been that graduation at
the online class was more than 10 times that of his average semester (also
from the article). Negativity sells?

------
ryan_j_naughton
Georgia tech's online masters in CS is the complete opposite (and orders of
magnitude closer to a normal academic environment) than MOOCs, and the reason
is all obvious design decisions / incentives.

Like any degree, Gatech's program has a permanent academic record, rules for
grade quality in order to not be on academic probation, etc.

If you don't have the incentives in place, then many people will find excuses
to not commit. MOOCs are lacking those incentives.

~~~
tmh88j
>If you don't have the incentives in place, then many people will find excuses
to not commit. MOOCs are lacking those incentives.

I haven't researched this, but I would guess the $6,600+ cost has more to do
with the high completion rate than any other factors. Most MOOC's seem to be
pretty cheap, if not free.

Not to mention, completion of the Georgia Tech online program results in a
degree. MOOC's typically just offer a certificate of completion, which carries
little weight.

------
porknubbins
Why “complete” a MOOC though? Most MOOCs still have the same BS hoop jumping
requirements as real courses alongside the meat of the course, but unlike real
world courses theres no degree at the end. As a self learner I ruthlessly
apply the 80/20 principle to maximize useful learning and move on. I’ve only
completed a few that are mercifully all autograder based.

~~~
riffraff
I loved the "achievement unlocked" feeling of completing online courses and
getting a pdf saying "you did it!", so I tried to complete the classes for
that (such a PDF has no real value, especially since I did classes on random
topics I like, not job related).

I can see why Coursera/edx etc stopped giving out these certificates if you
don't pay for them, but it has caused me to basically stop completing free
online classes.

------
irrational
I assume this is just free courses. I’ve been an online instructor for a
university for 4 years as a side gig. I’d say the completion rate is about
85%.

~~~
modernyogihippy
> I assume this is just free courses

That is what I thought as well. People who pay a handsome sum for an online
course are not going to let their money go to waste. Something about paying up
simply ensures that you turn up and actually attend/finish a course.

------
rfrey
This might be outdated on some platforms, but the last time I had time to
indulge in some online courses, you had to enrol in the course _just to see
the syllabus_.

My completion rate was not stellar.

------
codeisawesome
Until Visa officials around the World wake up and smell the coffee that
Degrees shouldn't matter for immigration - I will care about and be anxious
about my lack of a Degree. Until MOOCs don't confer a degree - it's going to
be very hard to convince me to spend time getting a Certificate from it (as
opposed to watching the most interesting and important bits and skipping the
rest, in the "YouTube University" style).

~~~
modernyogihippy
Unfortunately, I don't think degree requirements are going to go away anytime
soon for immigration purposes. Just imagine the backlash in the US/EU if the
voting public finds out that people without degrees can now move to their
countries. Many will link his to a mass-movement of unskilled and unwanted
labor as opposed to talented and hardworking individuals.

~~~
codeisawesome
That's very sad, but I suppose I can take a twisted comfort from it: the fact
that this public attitude is unlikely to change in my productive lifetime
means, it's worth girding down and getting a degree. At least it's unlikely
that 10 years later I'll regret the time I wasted chasing a piece (or two) of
paper(s) ~6 years prior.

------
rhacker
I watched the Andrew Ng videos in coursera - I honestly didn't care about any
course work, I was just trying to watch the vids... did I do something wrong?

~~~
vintermann
Not exactly, but the coursework is worthwhile. It's well designed so that you
get to only focus on the topic at hand, not setup or scaffolding of any sort.

You really get a feel for how to turn an equation into vectorized code, which
the video lessons can't give you.

I've completed the first Ng course, the later five-course specialization, and
a few other courses (Odersky's Scala course among them). Still, 4% doesn't
sound unreasonable: I too have enrolled in more courses than I completed.

------
acd
The cost of signup for a MooC is low and the time and effort needed to
complete Mooc is rather high. That could explain the lower completion rather
and something is still a lot better than nothing. Also why are we comparing
against normal courses were people has dedicated time versus online where
there is not dedicated time?

I think it would be good to design course content for the medium. Professional
adults does not have unlimited spare time, especially not if you have other
duties to upkeep that require time family and friends. Thus maybe it would be
wise to divide the course content in smaller chunks and reward people for
completing each chunk. This would be a design for Mooc that would be very
similar to game level design which already works brilliant online! Aka divide
and conquer algorithm. Reward people for completing levels.

[https://en.wikipedia.org/wiki/Divide-and-
conquer_algorithm](https://en.wikipedia.org/wiki/Divide-and-conquer_algorithm)

------
gumerd
If you want to complete them I really recommend pairing up with a friend.
Going at it alone makes it easy to quit. While 3 friends deciding to "drop
out" is less likely to happen by an order of magnitude.

~~~
analognoise
I've never successfully convinced a friend to take anything.

Turns out taking a Stanford class (or anything) on the introduction to logic
isn't most people's idea of fun.

------
rajesh-s
In my opinion, the lack of geography-specific pricing makes the evaluation
(tests and assignments) and certification inaccessible to a significant
population. Of course, one can audit the course anytime but it’s hard to have
the same level of sincerity and commitment when there’s no incentive
(certification that can probably be showcased on LinkedIN or CV) or deadline
based timeline for the course.

I appreciate that these courses have been made available to anyone who wants
to learn, but my point is concerning the meagre completion rate.

------
monochromic
As it stands, MOOCs don't have the same recognition or importance in society
as traditional education does. So naturally completion rates are going to be
low. Doesn't mean the courses aren't good quality or the information being
shared doesn't mean people's needs. Just means that the psychological
contracts that people enter when they sign up for these courses are a lot less
burdensome and easier to break than within higher education, which might even
be a part of their draw.

------
dantheman
I think coursera is amazing, i've finished a few courses signed up for a lot
more. It's great, some i stopped because I wasn't really interested in, others
I stopped because I got what I needed. I think it's truly amazing and I think
completion is the wrong metric; maybe hours of instruction divided by average
per course would be better.

------
Scoundreller
This is great news.

I remember a time when/where this kind of knowledge was hard to find and
devoured whenever it was made available to you.

------
sprucely
I dropped one on automata after an exercise in the first exam required making
use of an undisclosed algorithm that wasn't central to the topic and not
covered in any of the required reading. When asking the lecturer about it, he
directed me to chapters n through n+5 in some other book. I guess I didn't
want it enough.

------
agumonkey
I registered to a lot that were out of my league (say advanced physics). But
also a few normal IT MOOCs, like relational DB, made me strenghten some new
skills even though I didn't finish it. Mostly due to trivial time constraints
making me miss deadlines.

------
rambleraptor
A lot of it has to do with credentialing. MOOC credentials aren’t worth the
time investment.

~~~
randomdata
Plus nobody cares about credentialing at all. What they do care about is
solving problems, and it seems reasonable that 96% of problems become solvable
with the course material provided before reaching the very end of a course.
Once the problem is solved, there is little reason to continue.

------
Daub
From having taught a MooC, I can share some light on this. Most people who
sign up and drop it do so to access the course ware. Once they have downloaded
this, they disappear or fade away.

------
axython
I'd be very curious to learn the difference between free/two clicks to enroll
and a paid course.

I'd assume completion rate for something that you actually paid would be way
higher.

------
lquist
What length of course? I can't imagine that completion rates are the same for
a 50 hour course vs a 200 hour course. Can't seem to find this data.

------
balaam
That makes mse think that are probably a lot of easy wins in the funnel of
getting people successfully through the course.

------
znpy
The question is, does it matter ?

I've been following the MOOC phenomenon since its early days in 2010-2011, and
taken a lot of moocs in the meantime.

Sometime the mooc reveals to be low quality, and I quit because I can get
better information from books and/or by actually messing with the technology
(xkcd.com/519). Other times, I quit the part after the interesting part, or
don't care about going through all the single details to get a worthless
recognition ("a website says i've watched all the lessons").

Other times I pay for a course, go through all the lessons, do all the labs,
get all the certificate of completion.

It really varies a lot, and most people ask the wrong question: it's not "how
do we measure success" but "how do we measure success with respect to the
original goals?"

Very little people care about an automated system somewhere recognizing my
100% completion of the course materials. But I (and possibly, possible
employers) care about actual competencies. It's the same thing all over again:
in real life pieces of paper don't matter, what matters is the capability of
solving problems and handling situations.

------
grendelt
Average completion rate for _MOOCs_ Is 4%

------
scoobyyabbadoo
Not surprising at all. I often follow the first few weeks of a MOOC but after
a bit I am satisfied and don't feel the need to push through all 13-17 weeks
of material. Isn't that a success? I certainly learned a lot along the way all
the same! Much more about a topic than I would reading the wikipedia page or
an introductory blog post, because MOOCs (being regular college classes) take
a lot of pains to lay out the course material in a logical and structured
format, which is itself alone already much more valuable than a single
person's hodgepodge thoughts.

~~~
onlyrealcuzzo
Came here to say this. It's the same with text-like books. There's 100 pages
of what you bought the book for, and 400 pages of filler at the end in an
attempt to add more value.

