

Results from Khan Academy experiment - mikeleeorg
http://blendmylearning.com/2011/08/31/the-results/

======
john_horton
In the comments, the authors give the sample size as about 20 in each group.
Here's a more sensible graph (error bars to create a 95% CI on the estimated
group means):

<https://skitch.com/johnjosephhorton/fih62/quartz-2>

Obviously, there are some statistical power issues. It's unfortunate that they
didn't get someone to help them with a power analysis up front---given how
many very tech savvy people are interested in Kahn Academy and want to see it
succeed, I think they could have gotten this sort of assistance pretty easily.

~~~
JunkDNA
Thanks for doing that, I found myself wishing they had put error bars on when
I saw that graph but was too lazy to do it myself.

With respect to the statistical power issue, it may be they just didn't have
enough students. Unlike your typical clinical trial or scientific study where
you're recruiting people, I got the impression this was from an existing pool
of students for a specific school. Once you go beyond this, the design gets
tricky since there are practical issues to how big a class can be. Then you
need more teachers, and you need to normalize for per-teacher variability. Not
saying a study like that wouldn't be great, but it would be complex to get
right.

~~~
john_horton
no problem & agreed about what probably happened - my point was more that
knowing they had that sample size (n=40), someone could have told them that
even if Kahn academy was extraordinarily effective, there was almost no chance
they'd detect statistically significant effect. If they knew that, they might
have pursued a more qualitative approach rather than have people see the take-
away as "no effect."

------
ZeroGravitas
I try to give educationalists the benefit of the doubt, because they probably
mean well, but at the same time, this stuff is _important_ so I want them to
be held to a higher standard.

I can't help but feel that this image they used sums up the whole project. Two
(basically meaningless) numbers, presented in a 3-D bar chart.
[http://blendmylearning.files.wordpress.com/2011/08/resultsgr...](http://blendmylearning.files.wordpress.com/2011/08/resultsgraph.png?w=300&h=180)

~~~
Jabbles
The only "result" that the article could claim is the unfortunate _There was
no significant difference between the two teaching methods._

However, we could probably regain hope for Khan Academy by pointing out that
the (five-week) summer school was almost entirely worthless for 2/3rds of the
students anyway:

 _In terms of distribution, in each class approximately one third of the
students saw some significant gains (ten percent or higher gains in percentage
of questions answered correctly), whereas two thirds of the students’ scores
were essentially flat (less than four percent increase or decrease)._

If anything, it is this that should make us eager to explore alternative
methods of teaching. I am hopeful that Khan Academy or a similar online
learning provider will revolutionise education, and this article is the first
step along the way to discovering _how_ to use it.

~~~
Steve_Baker
I think you may have missed what I think is the most important point made in
the article:

"It is also interesting to consider that students in the treatment group spent
approximately half of the summer working on pre-algebra skills. Because the
Khan software is individualized, it identified that most of our students had
significant pre-algebra skill gaps and delivered instruction and practice
problems to address these deficits. Students in the Khan/treatment group
therefore spent up to 50% less time than the control group on the algebra
content that the MDTP exam measured. The treatment group, however, still
performed at a similar level the control group on the algebra measures."

That says to me that these students spent most of their time filling in gaps
in their knowledge and still managed to keep up and even get ahead of the
other class in a very short amount of time. They're basically saying flat out
that these students learned as much of the advanced material in half the time
as the other group and probably have better foundational knowledge.

No significant difference my ass.

------
tomp
I really hate it when people (mathematicians?) misuse the percentage
notation...

> increased their percentage of correct answers by 5.2% over the five-week
> period > ... > For example, a student who started the summer knowing 60% of
> the correct answers in the traditional class ended the five weeks knowing
> 65.2% of the correct answers.

This means that the percentage of correct answers increased 5.2 percentage
points, which is actually an 8.67% improvement (60% + 60% * 8.76 / 100 =
65.2%).

~~~
irahul
> which is actually an 8.67% improvement (60% + 60% * 8.76 / 100 = 65.2%).

Yes, it is 8.67%.

Illustrating the calculation is better served by:

Initial: 60

After: 65.2

Increase: 5.2

% Increase: 5.2 / 60 * 100 = 8.76

~~~
alexro
8.67

------
JoeAltmaier
I thought the Khan Academy was about 'leave no learner behind'. That is, if
you miss a concept in traditional school, you flail around the rest of the
term. But in Khan Academy, you can review lessons, stop and start them, ask
limitless questions in the middle because they're all taped.

SO, I would expect to see extensive data on the _distribution_ of the scores -
how long is the lower tail of the curve? Did they pick up those that would
have lagged behind?

As is, they give a single percentage - improvement on an algebra test. So no
meaningful data at all. Hm.

------
MaxGabriel
In my experience, I see Khan academy being most valuable in a blended setting
when the teacher isn't very good (because Salman explains things so well), but
also over a longer period of time. Khan academy encourages retaining old
material in at least two important ways: it makes the old material easily
accessible, and Salman's teaching method strongly encourages you to have a
better understanding of the concepts behind the math (for me, that's been in
proving trig identities instead of memorizing and explaining how the process
of multiplying matrices is arbitrary, but it's clearly a fundamental aspect of
how Salman teaches).

------
tzs
Warning for iPad users, since it isn't apparent from the URL: the site is at
Wordpress.com and has the OnSwipe malware.

