
Spaced repetition for study and learning - gwern
http://www.gwern.net/Spaced%20repetition?2
======
Danieru
I've been using SRS (anki, ankiweb, ankidroid) for almost four years now. I'm
up to 4000 Japanese sentence cards and my amazement is constant. When I
started I thought I was fooling myself, there was no way I'd be able to read
real quantities of Japanese. Instead I can now read the manga for the anime I
watched as a kid and enjoy the stories I thought had ended.

I've gotten into a rhythm, four new sentences per day which at my memorization
rate works out to ~60 cards to review a day. Thus 20-30 minutes per day of
review. It has become such a part of my day that skipping a day makes the day
feel like a massive failure.

Its interesting how 4 cards can translate into 60 even with high memorization
rates: for every card you add you'll see it again in 2 days, ~4 days, ~8 days,
~2 weeks, ~1 month, ~2 months, ~4 months, ~8 months, ~1 year... A review
session is thus made up of multiple standing waves. The effect is noticeable
you'll see a workload increase as the next standing wave of review starts.
Right now some reviewed cards are getting scheduled next in ~8 years!

~~~
jliechti1
I have been using SRS to learn Mandarin Chinese for the last 2 years now and I
can also verify it is extremely effective when used correctly, and when
understanding its' advantages and limitations.

I think sentences are also a much better way to learn than learning single
vocabulary words, because they show the usage of the word in context. This is
very important for speaking naturally and idiomatically.

Once you get beyond a beginner level, choosing a direction of study can be a
little difficult. My technique is to find sentences or phrases from
conversations I have with people, books/articles I read, and from
movies/television shows - and then input it into the SRS. Preferably, each
sentence has 1 new novel phrase or vocabulary word. Although this can be time
consuming it gives a huge benefit. Let me give an example:

    
    
      Question side: “問題是。。。你不覺得這樣，很。。。虛偽嗎？"
    
      Answer side: 虛偽 (xūwěi - hypocritical, artificial)
    

So, in this case the second last two characters made up the word that I was
unfamiliar with, so I put that on the answer side. The huge benefit is that I
found this sentence in a book I am reading - so every time it comes up in the
SRS, I not only recall the sentence, I recall the _context of the story_ (in
this case a dialogue between two characters), and this has a double
reinforcing effect. In the same way, that if a TV or movie character says a
sentence over and over again in a video, it will cause you to remember the
scene of the sentence, and at which part of the story the movie was in at the
time.

The contrast of this method would be to find "pre-made" decks of cards for
sentences in the language you are learning, but I find that it is harder to
remember the items when the sentences are not as meaningful to you.

A key point for using SRS effectively is to use to review material _already_
learned, not to learn _new_ never before seen material. The article itself
even mentions this.

(Also, Mnemosyne is my SRS of choice [http://mnemosyne-
proj.org/](http://mnemosyne-proj.org/), but Anki is a good choice too.)

~~~
riggins
I'd like to learn Mandarin.

Can you give details on how you're doing this?

Like what you use for initially learning the material and what you use for
SRS?

~~~
jliechti1
Definitely - contact me at the email in my profile and I can tell you more
about my specific approach for Mandarin. (Don't want to sidetrack the thread
too much.)

~~~
selmnoo
Yes, please just say it here and share it with the greater community. :)

------
steveridout
I think spaced repetition software, which typically involves flashcards, is
great for memorising a large quantity of facts. E.g. for learning foreign
language vocabulary, anatomy, etc... For this kind of learning it can really
save you time, especially when you have 100s or 1000s of items to remember.

Using flashcards to learn mathematics, computer programming, physics, or
anything requiring more creative or analytical thought is not so useful. For
these subjects the number of raw facts required is usually smaller, and the
thought processes, concepts and problem solving ability are much more
important.

However, spaced repetition could be beneficial to these more creative and
analytical subjects if it was applied more dynamically. In this case the units
to learn could be the rules, laws, concepts which can manifest themselves in
different ways _within_ different problems, instead of being static
flashcards. Does anyone know of software which works in this way?

My interest in this stems from [http://readlang.com](http://readlang.com), a
site I'm working on at the moment for learning foreign languages. I use SRS to
keep track of words, but in future it could be nice to think about applying
SRS to grammatical concepts too.

~~~
kiba
_Using flashcards to learn mathematics, computer programming, physics, or
anything requiring more creative or analytical thought is not so useful. For
these subjects the number of raw facts required is usually smaller, and the
thought processes, concepts and problem solving ability are much more
important._

My programming flash card usually consists of evaluating pieces of code like:

Question -->

Evaluate the following:

puts "Hello world"

Answer --> Hello world

I cannot memorize the solution as if I memorize the answer to multiplication
problems, which is just straight rote memorization. I memorize the process for
code evaluation instead. In any case, I do not think anki and similar spaced
repetition flash card software are in any way a replacement for actually just
writing code and solving lot of different problems. They are always
complementary to learning the materials in other ways.

~~~
steveridout
The danger is that it's very possible to memorize 100s of answers like this
without having any idea of how the code is working.

Imagine if instead of static flashcards there the SRS worked on items like:

\- puts

\- the '+' operator

\- variables

\- functions

Then an algorithm would generate problems incorporating those concepts. The
exact same problem would never appear twice so you can't learn by rote and it
ensures you actually learn the concepts. Your progress on each concept would
be tracked by the SRS algorithm to help schedule your learning.

I think this would be a huge improvement on static flashcards.

~~~
gwern
I agree, for a lot of algorithmic bits of knowledge, it's better to study a
dynamic set of questions revolving around the algorithm, and that's almost the
same example I give in [http://www.gwern.net/Spaced%20repetition#prospects-
extended-...](http://www.gwern.net/Spaced%20repetition#prospects-extended-
flashcards)

But so far no one has stepped up and done the programming & card writing to
demonstrate it, aside from perhaps Khan Academy (they generate questions,
AFAIK, and are supposed to have incorporated spaced repetition at some point).

~~~
kumthekar
I am a co-founder of Lymboo Math ([http://lymboo.com](http://lymboo.com)) We
designed Lymboo Math to be a comprehensive, structured, and rigorous math
enrichment program. While most programs focus only on short-term mastery of
math concepts, Lymboo Math's emphasis is on both short-term proficiency and
long-term retention. We do this by incorporating spiral reviews at periodic
intervals. Our spiral reviews are 'spaced' both in time and in content, i.e.,
students are assessed on multiple topics in the same test. The mixed spiral
reviews improve students' context switching skills and at the same time
enhance retention of learned concepts.

You can learn more about our philosophy here:
[http://lymboo.com/about](http://lymboo.com/about) and
[http://lymboo.com/tour](http://lymboo.com/tour)

Thanks. Bala

------
bayesianhorse
Spaced repetition is great for material which is divided into very small
chunks and where you have a ton of time for comparatively little information
(Vocabulary, especially in languages you are not learning for a degree).

Problems arise in practical situations when you want to pass an exam. Or a
couple of them in a certain period of time.

Proponents of SR will say that it takes at least 6 times the amount of
time/effort over learning "short term". So if you are learning for an exam,
you better have plenty of time to spare. And you better start early, like 6
months before the exam. All that makes it pretty useless in a lot of
situations I've been in.

Short term learning isn't as bad as it looks. If you really "cram" for an
exam, and you think you forgot all of it a couple of days or weeks later,
that's not what happened. Instead, your brain is now primed for the material,
and when you come into contact with it later, a lot of information is still
there and re-learning will be easier.

~~~
kiba
_Proponents of SR will say that it takes at least 6 times the amount of time
/effort over learning "short term". So if you are learning for an exam, you
better have plenty of time to spare. And you better start early, like 6 months
before the exam. All that makes it pretty useless in a lot of situations I've
been in._

This is a bit of a strawman. If you need to pass an exam quickly, you should
and will "cram". Afterward, you can resume your spaced repetition learning.

~~~
bayesianhorse
For me, exam-focused learning has been the norm, not the exception. I don't
hear very often that people use these SR systems continuously for years, and
if so, it's usually in language learning

------
biot
Gwern, how long does it take to put together a comprehensive document like
this? Also, what motivates you the most in compiling the information? Is it a
rabbit hole of fascinating discovery which you joyfully explore until your
knowledge is satiated or does something else compel you?

~~~
gwern
> Gwern, how long does it take to put together a comprehensive document like
> this?

This particular page is one of the most citation-stuffed pages on my site, and
I read everything I link if the fulltext is available (which is currently
almost all of them after my last batch of requests)... I know I spent at least
40 hours compiling the core of the literature review for the LW contest, and I
feel like that was less than half the time I spent writing the rest of it,
updating it with new papers, that sort of thing, so I would guess this took
somewhere upwards of 80 hours.

------
mikevm
I always like sticking to the very best, so SuperMemo was the choice for me. I
remember spending a couple of weeks just reading the whole website
([http://www.supermemo.com/](http://www.supermemo.com/)) because it contained
so much interesting information on learning and memorization. It also took a
while to understand how to use the program properly because it probably has
one of the worst UIs one can imagine ;-).

Anyways, I ended up dumping SuperMemo because once my item database grew I
became too lazy to spend 30 minutes (or more!) each day going through this
drill. I might get back to it at some point, though.

Edit: By the way, SuperMemo offers a few more interesting tools.

One is "Incremental Reading", where you can import an article or text file
into SM and SM will let you incrementally process it. For example, it might
pop up as part of your repetition routine, and you can read a few paragraphs
(you read as much as you want), and pick out the important parts of what
you've read (parts you want to remember). Once you're done reading, you
continue and the whole text will come up again and again, until you finish
reading it and then you can dismiss it. The parts you cut out from the text
become separate learning items and they too will come up in your daily
routines, which let you further work on them until you ultimately reduce them
into a Question & Answer form.

Another tool is sleeping monitoring. SM will let you track your sleep data,
and it will tell you how your sleep affects your memory performance.

~~~
kanzure

        > Anyways, I ended up dumping SuperMemo because once my item database
        > grew I became too lazy to spend 30 minutes (or more!) each
        > day going through this drill.
    

Yeah, same happened to me. But I didn't catch on so soon, and I found myself
spending 2 hours/day on it before I started to scratch my head.

When I started, it wasn't obvious, but clearly there is an upper limit on the
rate of new entries that the system can support. Plus, you have to factor in
time spent doing data entry. I think these factors combine to make for a life-
time limit on the number of entries. Suppose you use SuperMemo for 20 years
(7300 days) at 2 hours per day (14600 hours) and 100 new entries per day.
That's only 1.4 million unique entries.. (this analysis is severely lacking
and deserves more thought).

Insert here some humbling story about the benefits of improving meta-aspects
of how and what you pick to spend time on. Obviously, inserting everything you
ever come across into SuperMemo is a bad idea.

Anyway, then I realized this was all stupid because I remembered everything
anyway. So now I'm off SuperMemo. Hooray memory!

~~~
gwern
> I think these factors combine to make for a life-time limit on the number of
> entries. Suppose you use SuperMemo for 20 years (7300 days) at 2 hours per
> day (14600 hours) and 100 new entries per day. That's only 1.4 million
> unique entries.. (this analysis is severely lacking and deserves more
> thought).

That's fairly similar to Wozniak's estimates:
[http://www.supermemo.com/articles/theory.htm](http://www.supermemo.com/articles/theory.htm)

In the FAQ he estimates that probably 200k is the max for most people.

> Anyway, then I realized this was all stupid because I remembered everything
> anyway. So now I'm off SuperMemo. Hooray memory!

Fortunate for you, but the rest of us must make do with our fallible memories.
:)

------
iandanforth
Spaced repetition is not new, nor fancy, and has direct biological evidence.
It works. That said having a good tool and motivation are required. I can
recommend ankiweb.net / anki desktop for the first, but can't help with the
second. The motivation to follow through on doing cards is what I have trouble
with.

~~~
kiba
I managed to hacked myself to build a habit of 30 minutes a day of spaced
repetition, building off my momentum to consistently do khanacademy an hour a
day(A project which I finished a while ago) in conjunction with my time
tracking tool. The key seems to be a bunch of small repeatable goal that allow
for small wins. It works for me, though I don't know if it will work for
anyone else. Plus, it's only anecdotal, so take it with a grain of salt.

(I wish there is a accessible book that rigorously about which habit building
strategies work according to scientific research)

~~~
joelhooks
The Power of Habit and [http://tinyhabits.com/](http://tinyhabits.com/) are
both great.

~~~
mjmahone17
That sounds like such a scam, the wording is the same as every "With these
easy steps, you'll feel better and be more fit!" program I've encountered.

Not saying it is a scam, but I'm weary of clicking through.

~~~
victorf
Wow it sure does. Only slightly more plausible than "one weird trick".

~~~
enraged_camel
Well, I took one for the team and registered. The world hasn't ended yet. I'll
let you guys know. :)

------
dfan
I largely credit spaced repetition with my improvement as a chess player from
a rating of 1800 to 2000, something pretty unusual for a middle-aged adult. I
wrote about about my experience at [http://dfan.org/blog/2013/07/07/mnemosyne-
part-3/](http://dfan.org/blog/2013/07/07/mnemosyne-part-3/).

------
Jugurtha
This is one of the greatest pieces I've ever read.

The anecdote where Hamming went to Bode's office to talk about Turkey was like
a lightning ! Is it possible to see so many great names in one paragraph !

I've always seen some analogies in things with half-life and the charge and
discharge of capacitors.

I'll give an example: I've always thought about learning as a capacitor
charging through a resistance, and thought about the RC constant. In my mind,
it takes relatively a short period of time to get to 63.2%, but it takes a
very long time to get from 63.2% (it will never reach that, since it's an
exponential).

What this tells me about learning ? Pick a field and then decide how much do
you want to master it. Then know that it'll take you longer to get from 63.2%
to 100% (which you will never achieve). An example would be the fact that
English is my fifth language, which I only started speaking in 2009. Do I want
to be an English major and write in a Kipling way, or do I want to have a
command good enough to get my message to you (and possibly learn other
languages) ?

It depends, but I find it a fascinating subject.

The other funny thing is that I used a mix of cramming and spaced repetition
to study: I used to skip college for a year, and then get a month and go hard-
core. Five courses. I'd draw a pentagon with the five courses and study two
courses a day. 1,2 - 3,4 - 5,1 - 2,3 - 4,5 - 1,2 ...

You'll notice that I go back to each module each other day. The results I got
from that are just astounding, as I was capable to memorize entire pages,
chapters and I would see them. I would know exactly where I've written my
notes on which pages. I aced the final exam of physics and had the best grade
with questions that were never asked before (it was theory, and no 'values' or
'calculations' were there). I was writing things I've never written before,
but I knew my maths checked out because I just knew. These weren't exercises
or something.. I've been accused of cheating many times. A part of me found it
insulting, the other part found it flattering as you only accuse someone of
cheating if what he did was an accomplishment.

------
luckluckgoosed
I recently read this study of Halo players from Microsoft, and the authors
found that spacing out your play made your gain rating faster per game played,
BUT if you space it out too much, then you'll simply be outmatched by players
who play a lot more. In other words, if you want to improve at Halo
efficiently, space out your play more, but if you want to improve in sheer
absolute skill, you will want to simply play more games.

[http://jeffhuang.com/Final_HaloLearning_CHI13.pdf](http://jeffhuang.com/Final_HaloLearning_CHI13.pdf)

------
zxcvvcxz
I've been considering making spaced repetition software to help reteach and
remind me of technical concepts I've learned over college that I don't recall
too well. Stuff like fluid mechanics, microelectronics, thermodynamics,
control systems, complex analysis, computer architecture, etc. Basically I
have access to a lot of relevant course notes I took over the years (and
digitized), but I don't remember it, so I would use SRS on this material to
relearn it. I was thinking 20 mins a day would be a good amount of time for
this.

But wait, let's take a step back. Does it _really matter_ if I can recall all
this engineering material? Supposedly my brain "knows" enough of it such that
if there were a big project in a field requiring that knowledge, then I could
relearn it all, because I've been trained to think in a way pertinent to that
material. Is my time better spent just learning new topics and building
things, which is what I currently do? Why dwell on having perfect recall? It
only seems to benefit test takers.

I would love to hear counter-arguments for why SRS for engineering materials
would be a good idea for those no longer in school, maybe then I'll be
motivated to build some gamified spaced repetition software. It'll need to be
fun and reward-based, imo.

~~~
cicatriz
I've been thinking about this for a while--I created a spaced repetition
system to digitize and learn material in college (alas, I don't use it
anymore). I have a philosophical approach to the question here (also cited in
Gwern's article): [http://blog.learnstream.org/2012/08/spaced-repetition-in-
nat...](http://blog.learnstream.org/2012/08/spaced-repetition-in-natural-and-
artificial-learning/)

Your distinction between learning facts and learning ways of thinking is
interesting. Most of the spaced repetition research and design is around fact
recall--what's called "paired associates" because you recall some response
paired with some stimulus. I don't have any evidence for it, but I'd say that
the schema can be lost just as well as facts can be.

One thing I will say is that spaced repetition is generally known for the
repetition--that you retain something over time, and the spacing is more
thought of as convenience. But the truth is that spacing is also part of
better learning--when recall is difficult it's learned better (called
"desirable difficulties"). I thought of that because Robert Bjork does that
research with skills as well as facts. (See
[http://www.wired.com/geekdad/2012/01/everything-about-
learni...](http://www.wired.com/geekdad/2012/01/everything-about-learning/)
for an overview.)

~~~
zxcvvcxz
Yes it's not obvious to me how to impart knowledge in a SRS format that's more
than fact recall. I was thinking of starting with thermodynamics because it's
one of the more straight-forward equation courses - but then how do you
properly do problem solving? I suppose if you walk through a problem broken up
into enough chunks phrased as questions, and somehow give users a toolbox from
which to choose equations from, it might work.

~~~
cicatriz
Not sure how deep down the rabbit hole you want to go, but here's what I know:

The field of intelligent tutoring systems is essentially how to break up a
problem and estimate your knowledge on each piece (called a "knowledge
component"), and then how to instruct based on the resulting student model. I
got started by reading about Andes physics tutor: [http://oli.cmu.edu/wp-
oli/wp-content/uploads/2012/05/VanLehn...](http://oli.cmu.edu/wp-oli/wp-
content/uploads/2012/05/VanLehn_2005_Andes_Physics_Tutoring_System.pdf) but
there are many examples.

Most of the research doesn't actually incorporate spaced repetition. One
exception is Philip Pavlik's work:
[http://optimallearning.org/](http://optimallearning.org/)

It's still a lot of effort to break things down into knowledge components and
figure out the sequencing and instruction. That's why companies like Carnegie
Learning and Knewton exist, and even those are only targeting elementary or
intro level material for now.

------
pixelcort
I have been experimenting with spaced repetition algorithms for around two
years now, and I've found that for me, an a simple exponent with base of
2.71828 (e) to be optimal for learning short question/answer pairs.

This means, every time I successfully recall the answer to a question, I
increase the interval of when I will be challenged with it again by about
2.71828 times since I had seen it previously.

This is with a recall failure rate of about 5%, which I'm trying to maintain.
(If it's nearing 0, then you're probably wasting time being challenged too
frequently.)

This is on a data set used daily for 30 minutes, reviewing about 50 older
questions and encountering around 3 new ones per day.

------
jacques_chester
Comprehensive, as usual. An obligatory nitpick:

> _At least with physical fitness there isn’t a precisely dismaying number
> indicating how far behind you are!_

Many sports are precisely measurable and in such sports, the measurements
determine who has won, to various resolutions. Shortest running time over a
given distance, heaviest weight snatched, furthest distance jumped and so on.

Athletes in such sports can very accurately track their performance over time.
As of yesterday I demonstrated that I can clean & jerk 25kg less than my
lifetime record, for example.

~~~
gwern
I think the better analogy would be if you had some sort of workout schedule
and you had missed n workouts. Just bad performance doesn't indicate much: if
I score a 2 on all my flashcards today, that's bad, but I'm not 'behind' since
the cards' next review will just be moved up as necessary to compensate.

------
hngiszmo
For me it was absolutely surprising that out of the 200 Android apps, only 4
used spaced repetition but beeing an Android developer myself, I took the
challenge to make it better:
[https://play.google.com/store/apps/details?id=de.leowandersl...](https://play.google.com/store/apps/details?id=de.leowandersleb.fluxcards)

FluxCards provides spaced repetition learning with easy to edit and create own
cards. You can sneak the answer line by line, you can finger-paint and the
stats are fast and meaningful.

It is free to use with 0.5, 1, 2, 4, 8, 16 days intervals. Beyond that you
need the premium version but your cards are yours, so if you consider it not
worth the fee, you can export your cards to SD and use them in Anki.

And yes, spaced repetition rocks. I can only encourage any flash cards
programmer to implement it, as all these apps that implement flash cards but
no spaced repetition learning are just not worth your time.

------
jms
I use SRS (Mnemosyne) for memorising traditional Irish tunes. It feels like
each tune is a bit too much of a large chunk to remember at one time for best
use of an SRS (you're supposed to chop the information up into the smallest
element possible), but the benefits of the system are amazing. Before I
started using SRS I'd just practise the tunes I liked and already knew!, with
the new tunes, and anything in the middle got forgotten about. Now mnemosyne
forces me to practise tunes that I don't know, and the number of tunes I'm
fluent in has increased dramatically.

I'm also building a language learning startup that uses SRS. If you're
interested in trying the system out in beta, or being notified when it's ready
for general consumption, email me at languagebeta@jh.id.au and let me know
which language you want to learn.

------
gingerlime
at kenHub[1], we built an online trainer for anatomy. It uses a game-like
sequence of exercises designed to teach and test you at the same time about
anatomy structures and their terminology. We've implemented a very basic
spaced-repetition algorithm, but there's always more to improve.

One interesting thing we've found early on, was that when we encouraged people
to actually "take a break", not to repeat too much, but instead let your brain
process and come back to it later, people didn't like it. They wanted the
ability to keep training and training. This resonates well with what the
article describe about students' cramming tendency. So some time you have to
manipulate even a solid algorithm, to fit with reality.

[1][https://www.kenhub.com](https://www.kenhub.com)

------
jordwest
I combined spaced repetition with machine learning to attempt to predict
memory recall for my thesis. Turned out fairly effective, correctly predicting
recall up to 70% of the time.

Spaced repetition is pretty amazing. Adding records feels like you're entering
them into your brain, provided you use it every day. I managed to go from
knowing almost nothing to scoring 99% in a Japanese kanji exam once, entirely
thanks to Anki.

My thesis is available here: [https://github.com/jordwest/Thesis-
Report/raw/master/report....](https://github.com/jordwest/Thesis-
Report/raw/master/report.pdf)

The online SRS I built is also available:
[https://github.com/jordwest/Membit](https://github.com/jordwest/Membit)

------
styrmis
SRS software (I mostly used Anki) was invaluable to me when I was learning
Japanese, both for vocabulary and the kanji.

I always wanted to make my own SRS, one that had ease of use , stability and
data integrity at its core rather than cutting edge features. This year I
finally set aside time to make it: feel free to try it out and let me know
what you think. It is very simple (there is only one deck), but multiple decks
will be the next feature to be added.

[http://benkyo.co](http://benkyo.co)

On top of that platform is an implementation of a rather interesting way to
learn Japanese reading which it would also be great to have feedback on:

[http://benkyo.co/iikanji/](http://benkyo.co/iikanji/)

~~~
Ideka
iikanji looks good, but have you seen wanikani? The concept seems similar.
[http://www.wanikani.com/](http://www.wanikani.com/) Some feedback: I know how
to read all of the stuff I tried (the first 10 kanji), mostly thanks to
wanikani. But I haven't studied much gramar, so I'm not always sure what the
sentences mean. You should probably add translations for these in the back of
the cards.

------
narwally
Does anyone know of any command line spaced repetition tools? if not could be
a fun side project.

~~~
fsiefken
i am looking for that as well.. if you use emacs there are org-drill.el and
flashcard.el:

[http://sachachua.com/blog/2013/02/rediscovering-emacs-
featur...](http://sachachua.com/blog/2013/02/rediscovering-emacs-features-or-
what-to-do-after-you-get-carried-away-installing-packages/)

------
thinkersilver
It's great for learning facts and exploring a subject. Spaced repetition is
fascinating. Especially in the realm of language learning. Vocabulary
acquisition can be optimised by deciding the right order of words to maximise
your ability to read a foreign language text. gwern has a link at the bottom
of the article on modelling vocabulary lists as a travelling salesman problem.
Feeding this into SRS Flashcard program is powerful. I guess one of the big
problems ( if speed is your goal) is the creation of the cards themselves. The
time to create the card is much greater than the time to learn the card and
can be time a sink if you are trying to learn your first 3000 words of
language.

------
peterderivaz
I've been using Anki for learning Japanese. I've particularly appreciated the
decks that include audio of complete sentences and have found that it has
greatly improved my ability to understand spoken language.

One thing that I've wondered is whether it is a good idea to try to think up
mnemonics for tricky words when using spaced repetition. It certainly seems
effective in helping recall but I worry that I might train my brain to always
rely on going via an indirect path to get to the information and that this
will slow down my ability to understand real language.

Has anyone seen any research along these lines?

~~~
dunstad
From my personal experience, using such a method doesn't cause problems. I
studied the kana using Heisig's Remembering the Kana, where it connects each
character and pronunciation to a mnemonic story. I'm long past the point where
I actually need the stories to read kana, the sounds are simply tied to the
characters now, with the stories available for recall if i have a brain flub
and can't remember one for a bit.

------
seanlinehan
I was working on mobile app that helped with spaced repetition as a side
project. I built a pretty good looking user interface but ran out of time to
work on it between school and my company. If anybody is excited about spaced
repetition and wants to join me on my side project, I'd love to try to make a
little bit of extra time to get it live. I _really_ wish that the application
already existed - my midterms would be going much, much better :)

Email me at sean@einstein.com

~~~
hngiszmo
[https://play.google.com/store/apps/details?id=de.leowandersl...](https://play.google.com/store/apps/details?id=de.leowandersleb.fluxcards)
would need a developer for a browser version. I have big plans and if you
convince me, I might pay you for that or we agree on a share. It already has
10k users and a 4.5* rating.

------
SRSthrow
I used anki throughout high school for classes and SAT prep - pretty much the
main reason I ended up at an ivy league school (now a junior). It was
especially helpful for classes like AP Bio / AP US History where there was a
ton of pretty obscure stuff to memorize.

It's not so useful in college now that I'm majoring in applied math, but
occasionally I'll use it for a humanities class. (throwaway b/c I always use
throwaways)

------
Ixiaus
For those using org-drill in Emacs, I have a write-up of my own card/directory
layout and custom elisp to provide a navigation prompt to select which
"category" you want to train: [http://ixmat.us/articles/2012-12-01_usable-org-
drill.html](http://ixmat.us/articles/2012-12-01_usable-org-drill.html)

------
hoilogoi
I don't have anything constructive to say besides that I'm always impressed by
your work, Gwern.

Thanks for all the references as always.

------
tunesmith
I use Anki every day. I think part of the project is in learning how to use it
best. I have little interest in learning word-of-the-day or country names. I
am more interested in learning new skills, and I shared the concern that it
wouldn't work so well for that.

I think it's doing all right though. Here are some of the decks I'm most proud
of:

\- Using lilypond snippets and a midi-to-wav generator, I wrote a perl script
that generates a deck for every (non-accidental) note on a guitar fretboard.
It shows the fret position with a question mark, and then when you identify
the note, it plays the pitch and shows the note on a treble clef. I'll
probably expand it to include accidentals soon.

\- Also using lilypond, I wrote a jazz theory deck for major chords. Questions
like, "What chord quality and inversion is this?" where it plays a chord, and
answer would be m7b5 (half-dim) 2nd inversion. My ear training has improved a
ton. And, what is the key scale for bII7/iii in B major? (Plays the chord
audio). Answer: B melodic minor or E lydian/mixolydian, playing and showing
the scale on treble clef. And an effect like gwern described has happened. I
haven't actually memorized the answer to every card in the deck (since the
perl script generated about 2500 of them). Instead, I've gotten faster of
doing the actual music theory processing in my head to calculate/recognize the
right answer almost instantaneously. End result is that it has improved my
speed in reading lead sheets. (It has not, however, improved my melodic
improvisation abilities. :-) )

The rest of my decks are basically based off of Coursera courses.
Unfortunately, I haven't finished any of them yet. I routinely run into the
problem where partway through the course, the amount of time I need to spend
keeping up with the new cards limits my time available to watch new videos and
do new assignments, so I don't finish the course. I am now taking the Scala
course again, and I'm hopeful that since I'm fully reviewed up through week 4,
I might be able to power through and finish it with a deck that fully reflects
the course.

This also means that I am improperly creating a separate deck for each subject
matter, when Anki doesn't let you scramble them. I should be putting them all
in one deck, but I kinda don't want to before a deck is "complete". I'm also a
couple of chapters into "Learn You A Haskell", with the rest of the book on
pause.

Finally, I can echo that practicing helps a lot. For instance with the Haskell
deck, I got up to fully reviewed, and each card was still kind of hard. Then I
spent a couple of hours playing with the first Project Euler problem. After
that, the entire deck was much easier and faster. So, retaining familiarity
still is somewhat different than learning the concepts. If I have a fully
reviewed Anki deck from Odersky's Scala course (on coursera), it won't mean I
know Scala. But if I go start a Scala project, the Anki knowledge will help me
a ton, and then afterwards, the experience from hacking on the project will
make the Anki cards much easier, maybe even to the point that I could delete a
bunch of them.

BTW, that's feature I wish I had from the deck - some kind of advice for when
it might be appropriate to delete a card. I worry that I'm doing the
equivalent of writing pointless unit tests sometimes.

~~~
zodiac
How much of a music theory background did you have before you started making
those cards? What do you think is the minimum someone needs to know to be able
to benefit from making such a deck?

~~~
tunesmith
Well, my college degree was in music and I always had a knack for theory,
testing out of college courses etc. But you don't have to have university
level music theory knowledge to study jazz theory. I'd say the prerequisites
for the more complex jazz deck of mine are knowledge of scales (major, melodic
minor, major bebop, minor bebop), chord qualities (M7, 7, m7, m7b5), chord
inversions, and treble clef notation.

For the people reading this that do jazz, it's based off of Randy
Halberstadt's concepts of planets and harmony. Scale choices differ from the
classic guideline of being simply chord based. For instance, dmi7 does not
necessarily imply a d dorian scale. If it's in the key of C at that point,
then you'd instead play C major bebop.

------
simplyinfinity
Derek Sivers wrote a blog post on this back in january[0]

[0] [http://sivers.org/srs](http://sivers.org/srs)

------
espeed
Rethinking the exam structure in US schools to better accommodate and motivate
spaced learning would be a significant step toward improving education
quality.

'cperciva noted this effect from his time at Oxford...

"When I tell North Americans about my time at Oxford University, one aspect of
its undergraduate program inevitably surprises them: Final examinations.
Rather than writing examinations at the end of each term, students in most
subjects write a single set of examinations at the end of their final year, on
which their entire degree performance is measured. (In recent years, some
subjects have switched to annual examinations.) As terrifying as this might
sound, the system has some merits: Whereas many students at institutions with
termly examinations study the day before an exam and forget the material the
day after the exam, attempting to 'cram' three years worth of material at once
is largely an exercise in futility; as such, Oxford's examinations often
provide a better measure of how much material a student has learned and
retained. Given some of the conversations I've had with — and code I've seen
from — people who have received computer science degrees from major
universities and are actively working as software developers, I think that
last part is very important: There's a lot of people out there who have
forgotten some very basic — and very important — material"
([http://www.daemonology.net/blog/2012-10-08-software-
developm...](http://www.daemonology.net/blog/2012-10-08-software-development-
final-exam.html)).

You can obviously practice spaced repetition regardless of your school's exam
structure, and Cal Newport specifies a method for doing this in "The Art of
Stealth Studying: How To Earn a 4.0 With Only 1.0 Hours of Work"
([http://calnewport.com/blog/2007/10/03/the-art-of-stealth-
stu...](http://calnewport.com/blog/2007/10/03/the-art-of-stealth-studying-how-
to-earn-a-40-with-only-10-hours-of-work/)).

However, most students are unaware of this method nor have the disciple to
follow it on their own so to realize its benefits on a mass scale, we would
need to revamp the education system around these principles.

And as 'qwern noted, sleep is a key factor in spaced repetition learning.
Research shows sleep is when the brain prunes itself by separating signal from
noise...

"Sleep researchers at the University of Wisconsin-Madison School of Medicine
and Public Health believe it is more evidence for their theory of 'synaptic
homeostasis.' This is the idea that synapses grow stronger when we're awake as
we learn and adapt to an ever-changing the environment, that sleep refreshes
the brain by bringing synapses back to a lower level of strength. This is
important because larger synapses consume a lot of energy, occupy more space
and require more supplies, including the proteins examined in this study."

"Sleep — by allowing synaptic downscaling — saves energy, space and material,
and clears away unnecessary 'noise' from the previous day, the researchers
believe. The fresh brain is then ready to learn again in the morning"
([http://www.sciencedaily.com/releases/2009/04/090402143455.ht...](http://www.sciencedaily.com/releases/2009/04/090402143455.htm)).

------
dxhdr
Spaced repetition, also known as rote memorization, is a terrible way to
"learn" anything. The trouble is, you aren't learning at all, you're hacking
your brain to force information into it that has no relevance and really
doesn't want to be there. It's not a useful or pleasurable activity. It
definitely won't make you smarter. Even Mr. Supermemo himself admitted that
after FIVE YEARS of constant daily marathon memorization, 60% of it is gone
within the first year of stopping
([http://supermemo.com/articles/kowal.htm](http://supermemo.com/articles/kowal.htm)).
The brain does not want to work this way.

I'll propose an alternative to rote memorization for learning: critical
thinking.

~~~
tunesmith
That's funny because you can write a deck to help you learn concepts of
critical thinking. Necessity, Sufficiency, Contributory Causes; Propositional
Logic, Predicate/First-Order, Modal... the deck won't TEACH you critical
thinking, but it will make it much easier for you to learn it.

~~~
btilly
Learning the names of things is not learning those things. This is
particularly true when you're learning ways of thinking.

What is important is not that you can identify and name methods of reasoning.
It is important that you can apply the appropriate reasoning method to
problems in your life.

