

Khan Academy Put to the Test - mikeleeorg
http://jackcwest.wordpress.com/2011/07/08/blended-learning-with-khan-put-to-the-test-edtech-khanacademy-k12-edsurge/

======
edtechdev
Yes this is an awful design with a completely predictable outcome, and I hope
for ethical reasons the control students will get chromebooks and access to
the blended learning stuff afterward at least. And using the same teacher in
both conditions doesn't solve problems - the teacher expects students in the
experimental group to do better, and will influence the result even if not
actively trying to.

But unfortunately A) this is the kind of studies the department of education
wants and funds (randomized experimental designs - see the 'what works'
database, where they ignore anything not using that design). And this is the
case even when the control group is obviously in an intentionally deprived
learning environment. And B) the opposite design, tightly equalizing every
aspect of the control and experimental groups, isn't good either, because it
wouldn't take advantage of the new things blended learning / software afford
(like animations, interactive feedback) that traditional paper-based
instruction doesn't. See page 2 of this article from the Concord Consortium
about the 'hobbled horse race' and so forth:
[https://docs.google.com/viewer?url=http%3A%2F%2Fwww.concord....](https://docs.google.com/viewer?url=http%3A%2F%2Fwww.concord.org%2Fsites%2Fdefault%2Ffiles%2Fnewsletters%2F2007%2Fspring%2Fat-
concord-spring-2007.pdf)

There are 2 alternative research designs I would have done in this case:

1) Design-based research: <http://edutechwiki.unige.ch/en/Design-
based_research> Do the best you can with the Khan Academy curriculum in a
class. Pre-test and post-test, interview and observe and so forth. Measure
what they learned, their engagement, etc. THEN you try to figure out things
that didn't work out so well, things you missed...try to apply a theory or
framework to explain what happened. THEN try it out a second time making
revisions, and see if you can increase the learning gains and engagement even
more the next time. Use of pre-test / baseline measures makes it possible to
do comparisons.

2) Do short studies comparing different designs of the Khan Academy software
and curriculum. This is traditional, individual lab-based studies. Can be
useful if there is a valid issue where the designers / researchers don't know
the best path, and want to test/compare different options.

~~~
wisty
Education is not so special that it can't be subjected to the scientific
method. Medicine is similarly complex, and benefited immensely from randomised
trials. There may be challenges (the placebo effect of new methodologies, and
the impact which observation has on behavior), but it's not so special that
big gains can't be made by studying what works in an empirical manner.

When education catches up to where medicine was in the 1937 (with the
introduction of placebo trials), or even the 1950s (with double blind trials)
they can start looking into methodological improvements.

Until then, there's lots of low-hanging apples.

I guess the best way forward is for the education departments to stop trying
to control everything. They send out instructions, telling teachers what they
are meant to do. Why not just AB test the instructions they are sending
principals? That way, they can learn which of their decisions actually work.

~~~
edtechdev
Students aren't rats.

Which makes things a little more complex for educational research:
[http://www.elearning-
reviews.org/topics/pedagogy/educational...](http://www.elearning-
reviews.org/topics/pedagogy/educational-principles/2002-berliner-educational-
research-hardest-science/)

Like, little Johnny's Mom isn't going to like it when he gets a 'placebo' math
class one year.

------
vannevar
The unavoidable reality is that we won't get a real sense for how well these
kinds of innovative programs work until we try them on large populations over
long periods of time. In doing so, are we experimenting with our kids? Yes.
The key to breaking our educational paralysis is to recognize that we are also
experimenting with them _if we continue traditional teaching methods in the
face of a changing modern world_.

------
vessenes
I suppose all this experiment design will be helpful for academic theorists.
In the real world classroom, Khan Academy is a HUGE step forward for almost
any learning environment children are put in.

I know, because I have experienced it myself -- watching a teacher understand
they can see exactly which math subjects each student struggles with -- and
watching students leap ahead when given the chance to work on topics
interesting them and attach fun bragging rights for 'success' in a social
computer classroom environment.

The leap in engagement and skill building is so hugely marked that I don't
believe it's ethical to keep children from such an experience unless there is
strong evidence that it is actively harmful.

All this is anecdote, not statistics of course. The kids I worked with using
Khan Academy last year would say "Can we stay in from recess to do more math??
Are we doing Khan Academy today? YES!!" This level of response just isn't
present with all but the most magnificent teacher in a more traditional
classroom setting, and it makes me believe that there's something real and
beneficial there.

------
neovive
This is clearly quite a difficult task -- successfully controlling variables
in an educational environment is always going to be difficult. There are so
many students of varying levels, learning styles and parental support
structures that it will be a few years before tools like Khan Academy can be
fully assessed in a real classroom settings.

However, blended learning environments are clearly the future of learning and
the only hope for personalizing education in a system that is under constant
budgetary pressure with ever increasing class sizes. It will be exciting to
see the outcomes of these and future studies. Kudos to Khan Academy for really
driving this forward.

------
jackwest
Thanks for the link, edutechdev. I am speaking on a panel for edtech
entrepreneurs and educators in SF in October on exactly this thread topic,
experimentation. I will reference your design-based research guide. It is very
important that those of us who know something about social science
experimentation weigh in on this national conversation.

