
Khan Academy: The Illusion of Understanding - akbarnama
http://theeducationscientist.blogspot.com/2014/02/khan-academy-and-illusion-of.html
======
ZeroGravitas
Luckily not a single student ever left traditional education with a single
misconception about anything, or else this would seem like a ridiculously
slanted article.

edit: on re-reading, he's still clearly using Khan and MOOCs as eye-catching
buzzwords, symbolic of "bad education", but a) he mention (in passing) offline
education having similar issues, and b) he intends to talk about how online
education could fulfil his pet theory of learning in another article, so it's
not quite the usual reactionary post, although it does lean that way.

~~~
jerf
The reason I'm confident that in the long term MOOCs will succeed where
traditional education fails is not that MOOCs will come out of the gate a
superior product. The reason is that MOOCs can be tuned and developed over
time in a way that a classroom experience can not, because every class room
experience is too unique, delivered live, unrecorded, etc. It will take time
for this to happen, but it is likely inevitable, and eventually the idea of
flinging out an "education" into a room, where it shall disappear forever with
no feedback or development, will be considered as archaic as the horse buggy.

(And also people can't seem to help comparing the perfect conventional
education that _does not exist_ to the worst of MOOCs, which may feel good,
but is not relevant.)

~~~
VLM
I watched video tapes in the early 80s and before that, professionally
produced film strips. Even film strips with 33RPM vinyl records, before tape.
Also I saw a couple really old analog film movies in my youth. Currently,
youtube videos and DVDs are supposedly popular. Why didn't they replace
traditional classroom education using the same criteria you're applying to
MOOCs?

You may be correct about outcome, but the reaction mechanism is lacking.

I'm a classically self educated well rounded kind of guy, so I feel education
is developed from within not a mechanical external process much like
weightlifting. The idea it merely requires butts in seats, and the result
depends on what technique someone else applies to those seated butts, is more
likely to disappear. Or at absolute most, perhaps "necessary but not
sufficient".

~~~
jerf
"Why didn't they replace traditional classroom education using the same
criteria you're applying to MOOCs?"

For the same basic reason that the computers of the 1980s didn't replace
newspapers the way computers of the 2010s are; too hard to use, too expensive,
not enough availability. The tech for editing a course may have existed but it
wasn't wide-spread and easy, so there was not a feedback loop of improvement
going on. Imagine being the teacher in 1980; you've recorded your lectures,
you sent them away to be "edited", three months after your course was recorded
they come back, you sample a couple of them just to see how they came out (you
_certainly_ aren't going to watch them all), and that's it. Even if you find
an error, you're going to have to shrug your shoulders and at most issue a
paper errata; you certainly aren't capable of going back and fixing it
yourself, nor is anybody else going to pay for the skilled labor necessary to
do so. For a feedback loop to occur there's a certain base level of ease that
has to exist.

Only now do we have a world in which an instructor can give some bit of
instruction, get a question from a student about something, and simply click
"edit" to go back and fix whatever weakness in the original instruction the
student's question reveals almost as quickly as they could answer the question
in the first place. Run something like Physics 101 through that process over
the course of 4 or 5 years and you'd quite likely end up with the best
possible Physics 101-type course that could be developed. (The next step after
that is fundamental rewrites of how we teach physics at all, but that will
take longer.)

And indeed I'd suggest that we're still in the very early phases of that, too;
it's going to get easier and more widespread. Once we start _building_
educational resources over time instead of firing them like one-shot
ammunition, the improvement feedback loop is all but inevitable.

~~~
wbillingsley
Curiously, with MOOCs replacing OpenCourseware, we've shifted to an
environment that's a little bit harder to iterate on. Editing text is easy.
Editing something someone said in the middle of some video can be harder
(continuity issues, availability of the original speaker, audio environment,
etc).

Quite a few MOOCs have shifted up a notch in their production values recently.
When you've flown to another country to interview a famous scientist for your
MOOC, it's probably a bit trickier to make a minor edit to what was said next
year.

~~~
jerf
"Editing something someone said in the middle of some video can be harder
(continuity issues, availability of the original speaker, audio environment,
etc)."

That's part of what I was thinking when I said this was early days. While the
problems can't be completely glossed over, there's some stuff we can do to
mitigate those problems.

An interview would be a primary source; of course you can't edit that, but
good odds you've got some sort of summary that is part of the core course,
where the interview is immediately available as a primary source. The summary
would be as fungible as anything else, and the interview ought to be treated
as a primary source and archived. (Possibly with one pass of audio cleanup,
but not something to be modified wiki-style.)

On the topic of Wiki, some sort of editor-driven Wiki could help too. Many
Montessori schools already adapt the idea of having the older students teach
the younger, I believe; as MOOCs evolve it's very easy to imagine harnessing
the students themselves to start fixing the courses they've already taken.
They'd move from an isolated course to an integrated part of the whole
education. Again, like I said, I fully believe these are the early days of a
revolution, and writing MOOCs (and more generally, online education in
general) off now is like writing off the automobile because it's a great deal
less convenient in every way than a horse, and smells worse and is more
expensive to boot. Yes, that was true... at the beginning....

------
tokenadult
The guest author is Professor of Education at the University of Texas at
Arlington. It's good that educational research continues around the world. For
the moment, it's especially important that education research focuses on what
happens to young pupils in conventional classrooms, because they far outnumber
older students who use MOOCs to learn. In that respect, I think the title
"Khan Academy: The Illusion of Understanding" is unfair, because there are
plenty of learners with illusions of understanding leaving the elementary
schools of my country and yours, even though those learners have hands-on
experiences in the classroom.

Michael Shayer in Britain[1] has been doing great research for years about how
early experiences of children shape their understanding of the world. He has
discovered that many of Piaget's findings about how children proceed through
developmental stages in understanding no longer replicate in current children,
because children today have different kinds of early life experiences from
children in Piaget's era. What happens as society changes over time is that
some experiences that used to be commonplace become rare, and other
experiences that used to be rare become the most influential experiences in
children's development.

The best way to gain intuition about Archimedes' principle (the example given
in the blog post kindly submitted here) is probably to do the experiment
repeatedly with differing experimental conditions and careful observation. The
history of physics around the world shows that for centuries great thinkers
could be badly confused about very elementary principles of physics. Galileo
did the world a favor by insisting that physicists do more experiments to
check their hypotheses. Teaching elementary pupils to hold their conclusions
about the world tentatively and with an attitude of testing out everything
they believe is the best kind of elementary teaching. Whether by video or in
person, good teaching encourages learners to draw on their life experience
between lesson sessions to test out ideas and to see what ideas are really
correctly understood.

[1] [https://kclpure.kcl.ac.uk/portal/en/persons/michael-
shayer(5...](https://kclpure.kcl.ac.uk/portal/en/persons/michael-
shayer\(5c4f7363-ab67-4fe8-a3a7-3228d757fdbf\).html)

------
pja
As the responses to this HN post demonstrate so clearly, an intuitive
understand of what Archimedes Principle means in the real world is not
something that is easy to acquire: I'm sure most of the commentators on this
HN post are intelligent people who probably did well at school, yet the
majority of the responses here are completely wrongheaded.

If it isn't transparently obvious that the water level will neither rise nor
fall but stay exactly the same then you don't _really_ understand Archimedes
Principle.

The author's question therefore is: how is the Khan academy approach supposed
to improve on this situation? A bare explanation of Archimedes Principle
clearly isn't enough, nor is the ability to parrot said explanation after
viewing the explanation it as many times as required for the viewer to learn
the required words. How then is a learner going to acquire authentic
understanding from Khan academy videos?

(Note that traditional education has clearly failed here too! But that's a
side issue: the question here is why should we expect Khan academy to do any
better?)

~~~
jcampbell1
> If it isn't transparently obvious that the water level will neither rise nor
> fall but stay exactly the same then you don't really understand Archimedes
> Principle.

Surely it will overflow once the ice melts. At about 4C the water will reach
it's lowest point, and then rise and start overflowing at about 8C.

~~~
pja
The density change in the liquid water due to heating / cooling over the
likely temperature range is tiny compared to the result of going from ice to
liquid water. But I'll award you pedantry points nonetheless :)

The precise outcome requires knowing the volumes and shapes of the ice and
water components, their initial temperatures and the temperature of the
ambient environment so that you can calculate the path of the water volume
over time as the ice has melts. In order to make that calculation you'd still
have to make use of Archimedes Principle in order to know that you didn't need
to worry about how much water was displaced by the floating ice in your
calculations, which was the original point.

------
brudgers
The article conflates two separate connotations of "better" which are commonly
applied in educational contexts. When we talk about better teachers like the
author and Mr. Kahn himself, "better" refers to the average effectiveness of
the teacher's methods on individual students. We would for example control for
class size and socio-economic status of the students when deciding whether the
author was a better teacher than Mr. Kahn himself.

But what "better" often means in terms of educational infrastructure is more
access. We would measure the effectiveness by looking at the institution's
effect on an entire population. Here we can look for the effects of access and
raw numbers matter. A MOOC course could produce a significantly lower average
achievement among individual students but produce orders of magnitude more
highly successful outcomes.

It appears likely that they do based on my anecdotal experience. If there are
100x teachers and again I believe that there are in the context of adult
education, then MOOC's can be better in both senses. Mr. Kahn may not be a
100x teacher, but he is a 100x educator.

Dan Grossman of University of Washington has taught "Programming Languages" on
Coursera. I took the second section. Here:
[http://www.youtube.com/watch?v=1T4IQrOJr5U](http://www.youtube.com/watch?v=1T4IQrOJr5U)
he talks about data he collected from the first session. Particularly
interesting are what "number of students" and "passing" mean.

------
yread
WTF people!? Why is everybody discussing what the water is going to do when
the article is about online education?

To stay on topic: I agree with the article, watching videos often leaves you
with feeling of understanding but being unable to answer questions which
require actual comprehension

~~~
wbillingsley
At one of the keynotes for the Association for Learning Technology back in
2007, the speaker had a variation on this question. His point, if I recall,
was a little different --

As normally asked, the question is code for "parrot the Archimedes principle
back to me". But on a realistic "hot summers day" there's not just the
Archimedes principle. There's evaporation that can remove water from the cup.
There's wind, and occasional movement of the ice cubes as they melt and slip
over each other in the glass, that could cause spillage if it's really that
full that it's on the verge of overflowing. As could interruption of the
surface tension (e.g., condensation in just the wrong spot at the edge of the
glass, causing the water in the meniscus to spill over.) The initial
temperature of the water being unstated, its density will also change a very
small amount, potentially in either direction.

Often when we ask our students scenario questions as teachers, we are
essentially asking them to identify which taught principle we have flagged
this question as being about, and base their answer on that, ignoring all
other factors.

If the student answers "A. It goes down", did they fail to grasp the
Archimedes principle, or are they a pedant grumbling that over that length of
time there will be evaporation so it won't be _exactly_ the same?

Take the question out of the context of a course, and suddenly the educational
point of it can change. Is the person asking you this a physics teacher
checking your understanding of Archimedes principle, or an ALT conference
pedant checking whether you understand that in uncontrolled environments where
bumps and spillages can occur you can't blindly expect the ideal result you'd
get in a textbook?

The peril comes when questions are used outside of a teaching context. For
example in technical interviews, where the same question can sometimes be a
signal for many different teaching points, each expecting subtly different
answers, and the interviewee does not know which the interviewer has in mind.

~~~
dredmorbius
_But on a realistic "hot summers day"..._

That's an excellent point and a flaw of many "reasoned conclusions" thought
experiments. There's a world of difference between _contemplating_ what might
happen (which is a useful first step) and _experimenting_ to see what actually
_does_ happen.

In the case of your hot summer's day instance, there are a number of things
which could confound the experiment: how do you distinguish between water
inside the glass (from the initial water + ice), and that on the outside
(mostly from condensation, but possibly also from spillage)? How could you
control for this? How might you run the experiment repeatedly? Etc. Actually,
a chemistry lab where students each mix known weights of water + ice at
specified temperatures and observe what happens, then collaboratively compare
results might make for a useful group + parallel experiment exercise.

And that's just for ice water. In a field I've got some interest in (practical
and applied economics) there's an even greater tendency for assumptions and
ideal circumstances to get constructed.

------
james1071
Suppose we remove an ice cube of mass 10g.

The ice cube was previously floating and was therefore displacing 10g of
water.

If we let our ice cube melt, we will have 10g of water than we can add back to
the glass.

The net result is that the water level remains the same.

~~~
nationcrafting
Exactly.

I've had this discussion many times w.r.t. global warming, and am amazed so
many people, including scientists, just don't get it. The level the seas would
rise is not so much dependent on floating ice (north pole) since the
displacement is exactly the same, but on ice that was on top of land (e.g.
Greenland, Antarctica, etc.).

~~~
pja
Absolutely, although it's worth noting that loss of floating polar ice will
change the albedo of the earth, so there's a positive feedback loop there that
may result in further warming & consequent melting of land-based ice
elsewhere.

So melting of the floating polar ice will not directly result in raised sea
levels, but 2nd round effects of the melt might do so.

~~~
rimantas
I don't see any clouds in your picture.

------
alphaBetaGamma
I think we don't understand anything about how to teach.

Formal education started at least 4000 years ago (earliest writing is at least
4,000 year old, and I think you need a formal education to have scribes who
can read and write). I don't think we have done any substantial progress in
how to teach in that time frame, at least compared to how much progress we
have done in most other subjects: medicine, math, engineering, metallurgy, you
name it. heck, we are still discussing what methodology we should use to teach
reading & writing, and we have had 4000 years to figure what's the best way.

On the other hand, anyone who has had a good teacher can attests that teaching
quality can vary by a huge amount. So my theory for good education is: 1) find
good teachers (which probably includes "pay them well") 2) let them do their
stuff

Anybody who tries to do something better than the above should be able to
explain why their pet theory has not been discovered before.

~~~
DanBC
Don't let them "do their stuff". Have double blind randomised controlled
studies (or as close as you can get) to test various teaching methods. The.
Implement the good methods.

This has not been done because people have a built in desire for "fairness"
and splitting a country or state into two groups means one group is getting a
worse education. (Even though we don't know before the experiment starts which
group that is)

------
arikrak
Schools find it easier to get students to memorize facts and techniques
instead of getting them to really think and understand. This is a particular
problem with Lecture-based learning, and it applies not just to middle-school
students, but to Harvard students too:

[http://harvardmagazine.com/2012/03/twilight-of-the-
lecture](http://harvardmagazine.com/2012/03/twilight-of-the-lecture)

------
bhattisatish
Eric Mazur's Peer learning process
[https://www.youtube.com/watch?feature=player_detailpage&v=Ww...](https://www.youtube.com/watch?feature=player_detailpage&v=WwslBPj8GgI)
kind of highlights the problem and a solution for the same. Namely, Flip the
classroom. Ask questions that highlight the knowledge gap within the students.
Let students debate among themselves and then actually provide the solution.

Though this is not sufficient, it at least improves the ability of the student
to reason about the problem and question the models they have built around
their heads on the given subject.

------
sudhirj
I think it would go down... here's why.

When water freezes, it expands. That's what full bottles in a freezer burst,
as do pipes in winter. So 1 gram of water occupies less space in the glass (1
ml) than 1 gram of ice.

According to Archimedes' Principle, a floating object will displace precisely
as much water as its weight. Which is why a log of wood floats - the water it
displaces weighs more than the wood itself.

If 1 gram of ice is floating, it means it has already displaced 1 gram of
water. When 1 gram of ice becomes 1 gram of water, it will occupy less space.

As the ice melts, the level of water comes down.

~~~
sudhirj
Or maybe the ice above exactly replaces the level of water, and on the whole
it stays the same. The more I think about it the more confused I get, which
seems to be the point of the article.

------
quasque
The author's paper on the topic
([http://sloanconsortium.org/sites/default/files/6-Schwartz_0....](http://sloanconsortium.org/sites/default/files/6-Schwartz_0.pdf))
makes for interesting reading as it expands on the five requirements he states
are necessary for authentic understanding.

Though having read it, the only conclusion I can make is that I don't really
understand anything about the subject.

------
matchu
I disagree with the article's assertion that, because I'd forgotten
Archimedes' principle, I never truly understood it. I'd successfully worked
the problem far enough to realize that my missing piece of information was how
much of the ice cube stayed below water, then I watched the video and
reacquired that information. No problems here, and the KA video was plenty
sufficient for my needs.

------
chris_wot
Whenever I read these pieces, I always think that the establishment is getting
worried. Whilst there are good points to be made, such inflammatory and
sensational headlines make me think "Now why are they so upset about this?" \-
and unfortunately I often feel that it's because they feel threatened someone
is walking on their patch.

------
globalpanic
Interesting and challenging article about most current online education
offerings. It would be interesting to see if the problems of maintaining
engagement in a MOOC were at least partly due to students realizing that some
or all of the requirements for authentic understanding were not being met.

As I understand it, there is a different MOOC philosophy that has not gained
much support (probably because it's much harder to scale) - the connectionist
approach. Perhaps that would support some of the requirements.

(PS: interesting that many of the comments here try to rise to the ice-cube
challenge...)

------
crazy1van
Khan might not be perfect, but it is still pretty high quality. The lessons
are certainly higher quality than most of what I encountered in school.

------
lingben
ok, water expands when it is frozen and contracts when it melts, so the
question is by what factor to allow for the water to either overflow or not.

and are we talking about icecubes floating in the glass or filling up the
glass so they are standing on each other (like a cube skyscraper) from the
bottom of the glass?

~~~
spindritf
FTA:

When examining the ice you note that the cubes rise just above the surface of
the water (like glaciers in the ocean), _but do not extend to the bottom of
the glass_.

------
RankingMember
So it would overflow, right, dependent on how much of the cubes extend above
the surface?

~~~
nkoren
It's a trick question. The question creates doubt because it asks you to
choose between two wrong options. [Edit: no, it gives all three possibilities;
I speed-read through that. Thanks to others for pointing that out. However it
then does predominantly focus on the question of "up or down?", which isn't
the right question.]

Ice is lower density than water, which is why it floats. As the ice melts, the
runoff from the above-surface ice will raise the water level -- but the
shrinkage from the below-surface ice will lower the water level. Assuming the
above-surface and below-surface ice melts at the same rate, the water level
will remain unchanged.

(Note that if this were a glass of hot water in a cold room, then you'd get
non-uniform heating of the ice. The subsurface ice would melt first, causing
the level of the water to lower, until the above-surface ice melted, at which
point the water level would return to its original level. On the other hand, a
glass of cold water in a hot room would initially overflow while the above-
surface ice melted, and then its water level would lower when the subsurface
ice melted. So the answer to the original question is, depending on context,
either "neither" or "both".)

~~~
croddin
It isn't a trick question. The article gives all three possibilities.

~~~
nkoren
You're right - I speed-read through that.

------
VLM
"If you answered “Yes” to this last question, then you just experienced the
Illusion of Understanding first-hand."

Bzzt wrong it could mean you're thinking like an engineer instead of a
scientist. And its not binary, so fuzziness would be partially one or the
other reason at the same time.

Possible engineering challenges all contributing a really small delta, all of
which add up to so few sig figs a good engineer can't predict what will happen
to the water level:

1) You don't know how well mixed the cup is. I assure you, especially on the
"summer day" in the article you can get a spread of 20 or more degrees in that
cup. So the liq at the bottom is already way past the max density temp and
expanding while the top still has some ice.

2) You don't know the starting temp of the water and ice. In addition to #1
above, the ice in my freezer is well below 32 and who knows the starting temp
of the water. Maybe its barely melted meltwater or maybe its out of the tap or
maybe the coffee maker. So its going to go thru some wild temp and density
swings based on unknown initial conditions.

3) The pix shows massive condensation on the glass AKA they're running the
experiment in New Orleans not Phoenix also in written form they're running it
in the summer. Realize that there's nothing magic about surfaces and
condensation... if a sq inch of 32F surface condenses 1 teaspoon per 5
minutes, the top will condense just as well as the sides. On the other hand if
you run this in an environment with a dew point below 32F like where I am
today, then water will steadily evaporate away depending on the intensity of
energy striking the surface, although not too fast.

4) Ice stacks up nicely and clogs pipes. So you'd need more like a slushy to
make sure that icecubes are not mechanically sticking up above the surface by
being jammed by the cup walls. The supplied pix strongly indicates the middle
ice cube is mechanically jammed way above flotation level. One thing for sure,
liquids don't have much shear strength and thats going to eventually melt and
drip down and raise the water level... how much, who knows.

5) How does water surface tension vary WRT temp? I donno. Probably does, a
little. So how much the level can go up or down before it drips is a mystery.
You might be able to tolerate a rise of 0.1% if the surface tension enables a
"lens" above the top of 0.2% because of increased temp. Or maybe it goes the
other way and surface tension drops so "just barely overflowing" at 32F means
a flood (well, at least a drop...) at a liquid temp of 40F.

Its probably possible to very carefully engineer the perturbing contributions
to a low enough, or counteracting against each other enough, level (oh the
pun) such that the science principle will be demonstrated. Or given way the
heck more engineering data, a properly engineered answer might be provided,
maybe with enough sig figs to mean something. Or maybe not enough sig figs to
mean anything. I'm feeling the latter.

------
michaelochurch
I find that I have the opposite problem. I think I know less than I actually
know. Sometimes I feel (as a mathematician would) that I don't understand
something unless I can _prove_ it, assertion by assertion and line by line.

I dropped out of grad school after one year, and it seemed like a bigger
risk/deal than it was. My sense is that the major benefit of having a PhD
(especially from a top school) is the confidence that comes with it. It gives
you faith that you know _enough_ to tackle the interesting problems, and an
assurance that people (usually uninformed people, often without the
educational credentials they overvalue) won't try to peg you down as not up to
it. Socially, it gives you the right to be an expert, which is not the same
thing as expertise but just as valuable (if not moreso) in the work world.

So, even in traditional education, I'd argue that much of the benefit earned
(maybe 85%) is confidence, both your own and others' in you. You can get a lot
on your own by reading the papers and books, but you never get that official
approval or that "proof" that you learned _the right stuff_.

Oh, and the answer is that the water level stays exactly the same. Archimedes'
Principle is that a floating object displaces as much water as it weighs. When
melted, the iceberg (90% submerged) takes up as much space in the water as the
submerged part occupies.

