Hacker News new | past | comments | ask | show | jobs | submit login
Spaced repetition for study and learning (gwern.net)
219 points by gwern on Sept 28, 2013 | hide | past | favorite | 111 comments

I've been using SRS (anki, ankiweb, ankidroid) for almost four years now. I'm up to 4000 Japanese sentence cards and my amazement is constant. When I started I thought I was fooling myself, there was no way I'd be able to read real quantities of Japanese. Instead I can now read the manga for the anime I watched as a kid and enjoy the stories I thought had ended.

I've gotten into a rhythm, four new sentences per day which at my memorization rate works out to ~60 cards to review a day. Thus 20-30 minutes per day of review. It has become such a part of my day that skipping a day makes the day feel like a massive failure.

Its interesting how 4 cards can translate into 60 even with high memorization rates: for every card you add you'll see it again in 2 days, ~4 days, ~8 days, ~2 weeks, ~1 month, ~2 months, ~4 months, ~8 months, ~1 year... A review session is thus made up of multiple standing waves. The effect is noticeable you'll see a workload increase as the next standing wave of review starts. Right now some reviewed cards are getting scheduled next in ~8 years!

I have been using SRS to learn Mandarin Chinese for the last 2 years now and I can also verify it is extremely effective when used correctly, and when understanding its' advantages and limitations.

I think sentences are also a much better way to learn than learning single vocabulary words, because they show the usage of the word in context. This is very important for speaking naturally and idiomatically.

Once you get beyond a beginner level, choosing a direction of study can be a little difficult. My technique is to find sentences or phrases from conversations I have with people, books/articles I read, and from movies/television shows - and then input it into the SRS. Preferably, each sentence has 1 new novel phrase or vocabulary word. Although this can be time consuming it gives a huge benefit. Let me give an example:

  Question side: “問題是。。。你不覺得這樣,很。。。虛偽嗎?"

  Answer side: 虛偽 (xūwěi - hypocritical, artificial)
So, in this case the second last two characters made up the word that I was unfamiliar with, so I put that on the answer side. The huge benefit is that I found this sentence in a book I am reading - so every time it comes up in the SRS, I not only recall the sentence, I recall the context of the story (in this case a dialogue between two characters), and this has a double reinforcing effect. In the same way, that if a TV or movie character says a sentence over and over again in a video, it will cause you to remember the scene of the sentence, and at which part of the story the movie was in at the time.

The contrast of this method would be to find "pre-made" decks of cards for sentences in the language you are learning, but I find that it is harder to remember the items when the sentences are not as meaningful to you.

A key point for using SRS effectively is to use to review material already learned, not to learn new never before seen material. The article itself even mentions this.

(Also, Mnemosyne is my SRS of choice http://mnemosyne-proj.org/, but Anki is a good choice too.)

I am using Pleco on my phone as an SRS for Mandarin. (Setting it up to be a proper SRS hurts, though.)

The absolutely great thing about Pleco is that it is a great dictionary app and a document reader in the same app. When you see the solution of a flash card, it usually includes a handful of example sentences. You can then tap on words in these sentences and see their definition again... It is a little like getting stuck in Wikipedia - sometimes I spend half an hour only reading examples starting from my current flashcard.

I have no solid statistics on this, but I feel that the context really helps me to memorise things :)

I'd like to learn Mandarin.

Can you give details on how you're doing this?

Like what you use for initially learning the material and what you use for SRS?

For me, I downloaded a chinese deck from the shared Anki deck website. There's a series of ten of them. I also wanted to learn the strokes and how to say them and draw them, so I downloaded the "eStroke" app for my iPhone, and practiced drawing them in the air. Wasn't motivated enough to get ink and paper. :-) I also figured out how to do pinyin on my computer, and input strokes on the phone for character recognition. So, I dunno - I would do a few cards a day, but my retention was poor. I realized I was more interested in learning how to learn Chinese, rather than learning Chinese itself. And, doing the cards 10 minutes a day isn't enough - you have to be doing some sort of other activity with it simultaneously. I deleted the deck eventually. :)

Definitely - contact me at the email in my profile and I can tell you more about my specific approach for Mandarin. (Don't want to sidetrack the thread too much.)

Yes, please just say it here and share it with the greater community. :)

Please do sidetrack the thread!

Please do sidetrack the thread!

I've previously had good experiences with ProVoc (and now iVocabulary, a fork of ProVoc) but stopped using it and instead use Duolingo now because I lacked a good flashcard resource. I didn't have the time to think up new flashcards and learn them on a regular basis. Where do you get your sentences from for Anki? Is there a good repository online that you'd recommend (preferably with some kind of community review)?

I recommend creating your own cards, they just feel more weighty. I did jump start my sentence deck with the 500 card starter deck from AllJapaneseAllTheTime (http://www.alljapaneseallthetime.com/blog/my-first-sentence-...). Yet those cards feel different. If I were to redo things I would start from zero and build cards as you go.

My process is to add 4 cards everyday. These sentences come from posts in my RSS feed. Over the years I found websites that interest me. Take for instance http://blog.esuteru.com/ which has a very 2ch style of news, not safe for work but often interesting. Or I'll stumble upon people's blogs. For example "KEN's": http://qoqpop.blog123.fc2.com/ He is a phone "otaku" who often travels to china and Mexico for business trips. I found his blog in a N900 related search. The open source community also has plenty of japanese bloggers. For instance http://d.hatena.ne.jp/naruoga/ is the blog of a Debian translator and a journalist who I found through the OpenPrinting working group.

Oh I forgot to mention another benefit of self deck creation: it grows with you.

Some words are so well known that new cards do not contain their pronunciation. A new card may have 7 kanji words but only the hiragana for only 2 of those words. Likewise a card may only contain translation for one or two words. As time goes on I also select more difficult sentences. In both cases a premade deck cannot match your learning curve like self-made decks can.

I must also give credit for AllJapaneseAllTheTime for getting me excited about learning and for how to use SRS.

A typical card of mine might be: Front: エナメル質は歯の歯冠の最表層にある、 Back: 質=しつ//quality 歯=は//tooth 歯冠=しかん//tooth crown 最=さい//most 表層=ひょうそう//surface

That example has every pronunciation and translation. A real card would not have 歯 at all since I know it by heart. 質 and 最 would not have translations. Also notice how this is not a full sentence. Japanese often features many run-on sentences so I split them. Also notice there is no sentence scope translation.

AllJapaneseAllTheTime now suggests a newer format which features partial-ization of compound words which should work better for new decks: http://www.alljapaneseallthetime.com/blog/10000-sentences-is...

Can you comment on how are you translating the sentences at esuteru to add to your deck? Ie, if you can adequately translate those sentences already, seems like you don't even need to study them in the first place?

I've been using a standalone SRS app on my iPhone (StickyStudy) and about a year ago learned all N4/N5 vocab (ie, not just the kanji). But I'm having trouble making the jump to full-on sentences/text like that since there are too many unknown kanji or grammatical structures. StickyStudy does have a bunch of sample sentences for any one kanji or vocab word, which is great, but the learning focuses on the kanji or vocab words themselves, not sentences. So the ability to use the kanji in context of sentences is much harder.

I'd love to get to the point of being able to read webpages like that.

You're right, what I am getting out of the sentences is most often the vocab.

Earlier on I could not understand most sentences. At that time I would select the sentences I thought I knew. My understanding has expanded over time I think with great credit to anime and drama. I'm at ~330 series of anime/drama completed. Even before I started studying I had maybe 100 series completed.

Still a big thing for me is to card a sentence even if I do not understand it. The critical attribute I am looking for is "Is this a sentence I would like to learn?". Once it is in the deck I should understand it at some point in time. It may take a year but at least by then I will be able to pronounce every word in said sentence.

Even still there are some sentences I do not understand but those are often found in subjects I haven’t explored. Computer interfaces are 100% ok. Tech news likewise.

Start small and just never stop. Some of my first sentences were from children’s comics. I even have a few from pokemon games.

Speaking about 'repositories of sentences', you can try Tatoeba.org. It’s not without its faults, but it’s a fairly large downloadable (CC-BY) database of translated sentences.

I'm interested that you chose sentences rather than words. That seems super smart now that I think about it because you can pick up idioms and word relationships as well, and it's probably a little less boring too.

You're right, it is much more interesting. A few of my cards are a single word and I've found those hard to remember.

Instead sentences also let me remember context. Some cards might be speculation on a video game long since released. Others topical news reports long since over. Others might be internet slang which you would not want to memorize without the context to avoid confusion. I think this is vital since Japanese is so context sensitive.

Did you use it for kana too? How did that work for you?

I downloaded a nice kana set (with audio) but somehow they just didn't click. Anki eventually classed most of them as leeches and I stopped trying.

I must admit to not remembering how I handled kana. I know I did not start it until I was a decent way through the heisig book. I only remember even that because my mother was doing a masters in the university's German/Japanese department and one of the Japanese professors mentioned how odd the order was.

Don't worry too much about kana. It feels impossible now but it will become second nature. At some point you will hate it when someone uses romaji since kana is so much easier to read.

I do remember using the "Remembering the Kana" book: http://www.amazon.co.jp/Remembering-Kana-The-Hiragana-Kataka... which I can recommend 100%. The 3 hours claim is accurate from what I remember but do follow the book's suggested breaks.

> At some point you will hate it when someone uses romaji since kana is so much easier to read.

I'm currently reading The Structure of the Japanese Language, and that is indeed the case (the book doesn't use kana, maybe it was difficult to typeset at that time). And I'm not at all proficient with the language at this point, nor have I studied kana specifically. I've struggled a lot with it initially, but consistently refused to read or write romaji, which may have helped.

(Having just recently returned to the RTK deck after a break, I feel it's not enough and I probably should be working through words or sentences already, maybe in parallel. Thank you for describing the various aspects of your workflow in this thread!)

I’ll second this vote, this book is very good. When I learned from it there were two separate editions, now they are together, which is great.

To get over the hump of learning hiragana and katakana, I think you're better off using any mnemonic trick you can find. That's how I learned them as an adult.

This site looks like it might be on the right track. http://drmoku.com/hiragana-free/

did you make your own cards? or are pre-made cards available?

Just wanted to reply to you so you'll see it: I mentioned my card creation algo in https://news.ycombinator.com/item?id=6462709

Cool. Thanks so much.

I think spaced repetition software, which typically involves flashcards, is great for memorising a large quantity of facts. E.g. for learning foreign language vocabulary, anatomy, etc... For this kind of learning it can really save you time, especially when you have 100s or 1000s of items to remember.

Using flashcards to learn mathematics, computer programming, physics, or anything requiring more creative or analytical thought is not so useful. For these subjects the number of raw facts required is usually smaller, and the thought processes, concepts and problem solving ability are much more important.

However, spaced repetition could be beneficial to these more creative and analytical subjects if it was applied more dynamically. In this case the units to learn could be the rules, laws, concepts which can manifest themselves in different ways within different problems, instead of being static flashcards. Does anyone know of software which works in this way?

My interest in this stems from http://readlang.com, a site I'm working on at the moment for learning foreign languages. I use SRS to keep track of words, but in future it could be nice to think about applying SRS to grammatical concepts too.

Using flashcards to learn mathematics, computer programming, physics, or anything requiring more creative or analytical thought is not so useful. For these subjects the number of raw facts required is usually smaller, and the thought processes, concepts and problem solving ability are much more important.

My programming flash card usually consists of evaluating pieces of code like:

Question -->

Evaluate the following:

puts "Hello world"

Answer --> Hello world

I cannot memorize the solution as if I memorize the answer to multiplication problems, which is just straight rote memorization. I memorize the process for code evaluation instead. In any case, I do not think anki and similar spaced repetition flash card software are in any way a replacement for actually just writing code and solving lot of different problems. They are always complementary to learning the materials in other ways.

The danger is that it's very possible to memorize 100s of answers like this without having any idea of how the code is working.

Imagine if instead of static flashcards there the SRS worked on items like:

- puts

- the '+' operator

- variables

- functions

Then an algorithm would generate problems incorporating those concepts. The exact same problem would never appear twice so you can't learn by rote and it ensures you actually learn the concepts. Your progress on each concept would be tracked by the SRS algorithm to help schedule your learning.

I think this would be a huge improvement on static flashcards.

I agree, for a lot of algorithmic bits of knowledge, it's better to study a dynamic set of questions revolving around the algorithm, and that's almost the same example I give in http://www.gwern.net/Spaced%20repetition#prospects-extended-...

But so far no one has stepped up and done the programming & card writing to demonstrate it, aside from perhaps Khan Academy (they generate questions, AFAIK, and are supposed to have incorporated spaced repetition at some point).

I am a co-founder of Lymboo Math (http://lymboo.com) We designed Lymboo Math to be a comprehensive, structured, and rigorous math enrichment program. While most programs focus only on short-term mastery of math concepts, Lymboo Math's emphasis is on both short-term proficiency and long-term retention. We do this by incorporating spiral reviews at periodic intervals. Our spiral reviews are 'spaced' both in time and in content, i.e., students are assessed on multiple topics in the same test. The mixed spiral reviews improve students' context switching skills and at the same time enhance retention of learned concepts.

You can learn more about our philosophy here: http://lymboo.com/about and http://lymboo.com/tour

Thanks. Bala

Hey steveridout, you've got a point there. It's a weakness I've ran into many times when I move away from languages into learning something like statistics or hard algorithms. It's possible, by simplifying the information and using a combination of text and images. The problem when you start doing this is the time it takes to create a deck of cards that make sense. It would be a great idea to be able to use a templating language for your flashcards which you can embed simple rules and plug in a source of data. SRS programs have cloze deletion but it will go much further than that. It may be a good idea for an Anki plugin, perhaps. I hadn't thought about it until just now. ......

I use it for math at the undergraduate to graduate level and have for over a year, but not super religiously. There are plenty of things in math that benefit from this treatment. It's good to just know definitions, then when you come across something that satisfies it you'll know what to look for to find out more about it. You'll know the exact requirements of theorems so that you don't have to once again look on wikipedia to see that yes this does require a compact set. You'll find a card that says something you've reviewed before but go "wait, why is that true?" and then work out the general idea again. You can memorize known identities that are just plain useful, things that can't be easily Googled but can be easily recognized. You can put warnings to yourself that certain theorems can be easily applied incorrectly. You can learn the most common names for theorems so that it is easier to communicate with others about them.

And above all, it's just another chance to look over things you already should know. I won't claim that it's more or less useful than reading a textbook for the equivalent amount of time or doing problems, but it's certainly more useful than doing nothing.

P.S. Anki has very nice LaTeX support! I don't know about the others, but this is invaluable for math.

For mathematics and physics, the 'unit' of spaced repetition should be skills, not questions. You would need a program which can generate questions that test a particular skill, such as partial differentiation or factorizing quadratics. If you get a question wrong, that indicates that you need to spend more time practicing that type of question.

I'm currently using SRS software (Anki) to learn more about software development. Instead of trying to memorize facts I am using cards as a cue that it is time to revise and deepen my understanding of a particular topic.

Every time I learn about a new topic/concept/technique I will add a card to my anki deck with the topic name. E.g. Today I have added JS Promises.

Then when a topic comes up for review I spend 10-15 minutes reviewing my knowledge and trying to probe my mind for parts I do not understand and trying to link it to, and see how it can be used with, other related techniques.

After reviewing I then spend 30 minutes to 1 hour investigating the technique in more depth. At this point I'll try and implement an aspect of it in code or see if I can answer a few problems people are having with the topic on stack overflow.

Then I will add a few notes to the card detailing what I have done so that I can make my next "review" of the card more challenging.

All up each review for a card takes close to 1.5 hours. I tend to have 1-2 cards come up per day to review.

I find it has given me a broad, and usable, understanding of software engineering practice and principles and has made me a better developer. The downside is I feel you will never become a master at these topics using this technique as the time investment per topic is just not enough.

I started using it because I found I had an experts understanding of some topics and a very weak understanding of other fundamental areas of computer science and software engineering. E.g. 1000+ hours of crafting complex regular expressions but lacking a proper understanding of the relational model and how a database table join works.

I would be really interested to see if there is any literature that examines using SRS in this manner (as opposed to learning simple facts).

You approach looks very interesting. I was thinking how can SRS be used effectively for programming/software development skills. Can you please share your cards or any thing more on your approach?

Even for learning deeply creative skills, having a good scaffold of the vocabulary, concepts, and atomic pieces of knowledge is very important to construct high-level connections later on. Often, the relationship between fact-base memorization and high-level synthesis is a complex balance, but learning any skill or discipline is going to involve a fairly large amount of gruntwork in terms of just memorizing vocabulary and basic facts.

> Using flashcards to learn mathematics, computer programming, physics, or anything requiring more creative or analytical thought is not so useful.

I thought you were wrong for a second, since there are a lot of definitions and theorems that are probably good to know on the fly. But then I thought about learning about the meaning of greek letters; okay, but in what context...

Spaced repetition is great for material which is divided into very small chunks and where you have a ton of time for comparatively little information (Vocabulary, especially in languages you are not learning for a degree).

Problems arise in practical situations when you want to pass an exam. Or a couple of them in a certain period of time.

Proponents of SR will say that it takes at least 6 times the amount of time/effort over learning "short term". So if you are learning for an exam, you better have plenty of time to spare. And you better start early, like 6 months before the exam. All that makes it pretty useless in a lot of situations I've been in.

Short term learning isn't as bad as it looks. If you really "cram" for an exam, and you think you forgot all of it a couple of days or weeks later, that's not what happened. Instead, your brain is now primed for the material, and when you come into contact with it later, a lot of information is still there and re-learning will be easier.

Proponents of SR will say that it takes at least 6 times the amount of time/effort over learning "short term". So if you are learning for an exam, you better have plenty of time to spare. And you better start early, like 6 months before the exam. All that makes it pretty useless in a lot of situations I've been in.

This is a bit of a strawman. If you need to pass an exam quickly, you should and will "cram". Afterward, you can resume your spaced repetition learning.

For me, exam-focused learning has been the norm, not the exception. I don't hear very often that people use these SR systems continuously for years, and if so, it's usually in language learning

I don't think it's a straw man, because then after the exam you have to take new classes. College simply moves too fast to retain everything you've learned. I can't seem to add more than 20-30 cards a day in my Anki deck, or my review load gets bigger and bigger. If college worked more like a semester of classes, then a semester off to focus on retention and shrink the deck load, maybe that would work.

Assuming you're paying to be educated about a certain thing, and you actually want the knowledge and not just the diploma, walking away merely being "primed" to learn it later isn't the best outcome. As an analogy, cramming is better than failing the exam as driving into the ditch is better than a collision. However, best to avoid either scenario in the first place.

For me that would have meant avoiding getting the degrees I wanted to get. Bad option.

This is a bug with exams (and if you're in a situation where you need to cram, your study habits), not with spaced repetition. The fact that it's harder to pass exams while using learning strategies that enable long term retention is one of the main reasons for the disparity between academic achievement and actual competence.

The bug seems to be in my learning efficiency and time management. To learn some material for the long term, I would need about a year or two between having the lecture and the exam. Plus I'd need about six to ten times the amount of time to study. Time I just didn't have to spare.

The big misunderstanding is that an education is not about the amount or ratio of details retained in memory, but rather the mental infrastructure that was built by learning it, even for a short period of time.

I don't even agree that there is such a "disparity between academic achievement and actual competence". Anyone who demands more long term strategies has to demand a big reduction in the amount of information studied or increase the duration of the educational programs.

I think you're conflating latency and bandwidth. The total amount of time spent actually studying a given fact with spaced repetition (to get it in long term memory) is comparable to the amount of time spent studying that fact when cramming for an exam. The difference is that that time is spread over months instead of over a few hours. The bandwidth is roughly equivalent, but the latency is much higher.

So if you use SRS effectively, you can actually spend considerably less time studying. If you cram for each test, you're essentially crunching a bunch of data into short term memory, forgetting it, and then having to crunch a lot of it back in for the next exam. Then, for the final, you end up having to cram all of it in. Essentially, you're paying the cost of short term memorization over and over again, every time you have an exam.

If you use SRS and use it daily over the course of the entire semester, you'll only pay that cost once. And since you'll still have the information in your brain in between exams, your mind will be able to more effectively connect new information to old than if you cram and forget.

Gwern, how long does it take to put together a comprehensive document like this? Also, what motivates you the most in compiling the information? Is it a rabbit hole of fascinating discovery which you joyfully explore until your knowledge is satiated or does something else compel you?

> Gwern, how long does it take to put together a comprehensive document like this?

This particular page is one of the most citation-stuffed pages on my site, and I read everything I link if the fulltext is available (which is currently almost all of them after my last batch of requests)... I know I spent at least 40 hours compiling the core of the literature review for the LW contest, and I feel like that was less than half the time I spent writing the rest of it, updating it with new papers, that sort of thing, so I would guess this took somewhere upwards of 80 hours.

I always like sticking to the very best, so SuperMemo was the choice for me. I remember spending a couple of weeks just reading the whole website (http://www.supermemo.com/) because it contained so much interesting information on learning and memorization. It also took a while to understand how to use the program properly because it probably has one of the worst UIs one can imagine ;-).

Anyways, I ended up dumping SuperMemo because once my item database grew I became too lazy to spend 30 minutes (or more!) each day going through this drill. I might get back to it at some point, though.

Edit: By the way, SuperMemo offers a few more interesting tools.

One is "Incremental Reading", where you can import an article or text file into SM and SM will let you incrementally process it. For example, it might pop up as part of your repetition routine, and you can read a few paragraphs (you read as much as you want), and pick out the important parts of what you've read (parts you want to remember). Once you're done reading, you continue and the whole text will come up again and again, until you finish reading it and then you can dismiss it. The parts you cut out from the text become separate learning items and they too will come up in your daily routines, which let you further work on them until you ultimately reduce them into a Question & Answer form.

Another tool is sleeping monitoring. SM will let you track your sleep data, and it will tell you how your sleep affects your memory performance.

    > Anyways, I ended up dumping SuperMemo because once my item database
    > grew I became too lazy to spend 30 minutes (or more!) each
    > day going through this drill.
Yeah, same happened to me. But I didn't catch on so soon, and I found myself spending 2 hours/day on it before I started to scratch my head.

When I started, it wasn't obvious, but clearly there is an upper limit on the rate of new entries that the system can support. Plus, you have to factor in time spent doing data entry. I think these factors combine to make for a life-time limit on the number of entries. Suppose you use SuperMemo for 20 years (7300 days) at 2 hours per day (14600 hours) and 100 new entries per day. That's only 1.4 million unique entries.. (this analysis is severely lacking and deserves more thought).

Insert here some humbling story about the benefits of improving meta-aspects of how and what you pick to spend time on. Obviously, inserting everything you ever come across into SuperMemo is a bad idea.

Anyway, then I realized this was all stupid because I remembered everything anyway. So now I'm off SuperMemo. Hooray memory!

> I think these factors combine to make for a life-time limit on the number of entries. Suppose you use SuperMemo for 20 years (7300 days) at 2 hours per day (14600 hours) and 100 new entries per day. That's only 1.4 million unique entries.. (this analysis is severely lacking and deserves more thought).

That's fairly similar to Wozniak's estimates: http://www.supermemo.com/articles/theory.htm

In the FAQ he estimates that probably 200k is the max for most people.

> Anyway, then I realized this was all stupid because I remembered everything anyway. So now I'm off SuperMemo. Hooray memory!

Fortunate for you, but the rest of us must make do with our fallible memories. :)

That's different from my review session.

I review all the due items in less than 30 minutes, always. Then I fill it up to 30 minutes with new materials, sometime going a few minutes beyond 30 minutes, because the materials sometime requires some time to learn.

Spaced repetition is not new, nor fancy, and has direct biological evidence. It works. That said having a good tool and motivation are required. I can recommend ankiweb.net / anki desktop for the first, but can't help with the second. The motivation to follow through on doing cards is what I have trouble with.

I managed to hacked myself to build a habit of 30 minutes a day of spaced repetition, building off my momentum to consistently do khanacademy an hour a day(A project which I finished a while ago) in conjunction with my time tracking tool. The key seems to be a bunch of small repeatable goal that allow for small wins. It works for me, though I don't know if it will work for anyone else. Plus, it's only anecdotal, so take it with a grain of salt.

(I wish there is a accessible book that rigorously about which habit building strategies work according to scientific research)

The Power of Habit and http://tinyhabits.com/ are both great.

That sounds like such a scam, the wording is the same as every "With these easy steps, you'll feel better and be more fit!" program I've encountered.

Not saying it is a scam, but I'm weary of clicking through.

If you're talking about Tiny Habits, I can vouch for it since I took it end of March this year. Of course my word's not worth much but BJ Fogg does explain about the method on his blog: http://www.foggmethod.com/#large-media and http://www.behaviormodel.org/ so in that sense it isn't like the "fitness/attract opposite sex" programs (scams) you talk about where everything is hidden and shrouded in mystery.

It's all automated via email, so it can be free. It is quite simple and effective but applying the rules in a general sense is something you'll have to master over time.

Also although BJ Fogg has my email address, he's never tried to sell me anything after the program finished, even though I note from his updated site he's got a new certification to be a coach program going on.

Wow it sure does. Only slightly more plausible than "one weird trick".

Well, I took one for the team and registered. The world hasn't ended yet. I'll let you guys know. :)

I largely credit spaced repetition with my improvement as a chess player from a rating of 1800 to 2000, something pretty unusual for a middle-aged adult. I wrote about about my experience at http://dfan.org/blog/2013/07/07/mnemosyne-part-3/.

This is one of the greatest pieces I've ever read.

The anecdote where Hamming went to Bode's office to talk about Turkey was like a lightning ! Is it possible to see so many great names in one paragraph !

I've always seen some analogies in things with half-life and the charge and discharge of capacitors.

I'll give an example: I've always thought about learning as a capacitor charging through a resistance, and thought about the RC constant. In my mind, it takes relatively a short period of time to get to 63.2%, but it takes a very long time to get from 63.2% (it will never reach that, since it's an exponential).

What this tells me about learning ? Pick a field and then decide how much do you want to master it. Then know that it'll take you longer to get from 63.2% to 100% (which you will never achieve). An example would be the fact that English is my fifth language, which I only started speaking in 2009. Do I want to be an English major and write in a Kipling way, or do I want to have a command good enough to get my message to you (and possibly learn other languages) ?

It depends, but I find it a fascinating subject.

The other funny thing is that I used a mix of cramming and spaced repetition to study: I used to skip college for a year, and then get a month and go hard-core. Five courses. I'd draw a pentagon with the five courses and study two courses a day. 1,2 - 3,4 - 5,1 - 2,3 - 4,5 - 1,2 ...

You'll notice that I go back to each module each other day. The results I got from that are just astounding, as I was capable to memorize entire pages, chapters and I would see them. I would know exactly where I've written my notes on which pages. I aced the final exam of physics and had the best grade with questions that were never asked before (it was theory, and no 'values' or 'calculations' were there). I was writing things I've never written before, but I knew my maths checked out because I just knew. These weren't exercises or something.. I've been accused of cheating many times. A part of me found it insulting, the other part found it flattering as you only accuse someone of cheating if what he did was an accomplishment.

I recently read this study of Halo players from Microsoft, and the authors found that spacing out your play made your gain rating faster per game played, BUT if you space it out too much, then you'll simply be outmatched by players who play a lot more. In other words, if you want to improve at Halo efficiently, space out your play more, but if you want to improve in sheer absolute skill, you will want to simply play more games.


I've been considering making spaced repetition software to help reteach and remind me of technical concepts I've learned over college that I don't recall too well. Stuff like fluid mechanics, microelectronics, thermodynamics, control systems, complex analysis, computer architecture, etc. Basically I have access to a lot of relevant course notes I took over the years (and digitized), but I don't remember it, so I would use SRS on this material to relearn it. I was thinking 20 mins a day would be a good amount of time for this.

But wait, let's take a step back. Does it really matter if I can recall all this engineering material? Supposedly my brain "knows" enough of it such that if there were a big project in a field requiring that knowledge, then I could relearn it all, because I've been trained to think in a way pertinent to that material. Is my time better spent just learning new topics and building things, which is what I currently do? Why dwell on having perfect recall? It only seems to benefit test takers.

I would love to hear counter-arguments for why SRS for engineering materials would be a good idea for those no longer in school, maybe then I'll be motivated to build some gamified spaced repetition software. It'll need to be fun and reward-based, imo.

I've been thinking about this for a while--I created a spaced repetition system to digitize and learn material in college (alas, I don't use it anymore). I have a philosophical approach to the question here (also cited in Gwern's article): http://blog.learnstream.org/2012/08/spaced-repetition-in-nat...

Your distinction between learning facts and learning ways of thinking is interesting. Most of the spaced repetition research and design is around fact recall--what's called "paired associates" because you recall some response paired with some stimulus. I don't have any evidence for it, but I'd say that the schema can be lost just as well as facts can be.

One thing I will say is that spaced repetition is generally known for the repetition--that you retain something over time, and the spacing is more thought of as convenience. But the truth is that spacing is also part of better learning--when recall is difficult it's learned better (called "desirable difficulties"). I thought of that because Robert Bjork does that research with skills as well as facts. (See http://www.wired.com/geekdad/2012/01/everything-about-learni... for an overview.)

Yes it's not obvious to me how to impart knowledge in a SRS format that's more than fact recall. I was thinking of starting with thermodynamics because it's one of the more straight-forward equation courses - but then how do you properly do problem solving? I suppose if you walk through a problem broken up into enough chunks phrased as questions, and somehow give users a toolbox from which to choose equations from, it might work.

Not sure how deep down the rabbit hole you want to go, but here's what I know:

The field of intelligent tutoring systems is essentially how to break up a problem and estimate your knowledge on each piece (called a "knowledge component"), and then how to instruct based on the resulting student model. I got started by reading about Andes physics tutor: http://oli.cmu.edu/wp-oli/wp-content/uploads/2012/05/VanLehn... but there are many examples.

Most of the research doesn't actually incorporate spaced repetition. One exception is Philip Pavlik's work: http://optimallearning.org/

It's still a lot of effort to break things down into knowledge components and figure out the sequencing and instruction. That's why companies like Carnegie Learning and Knewton exist, and even those are only targeting elementary or intro level material for now.

Whether you should use SRS for any given domain is completely dependent on your specific circumstances. The answer could easily be different for you and a classmate of yours who fits the description above.

But I'd say that if there's some are you'd like to work in that requires specific knowledge that fits well within SRS, why would you not? As opposed to trying to learn and recall anything that might possibly be useful, ever.

I have been experimenting with spaced repetition algorithms for around two years now, and I've found that for me, an a simple exponent with base of 2.71828 (e) to be optimal for learning short question/answer pairs.

This means, every time I successfully recall the answer to a question, I increase the interval of when I will be challenged with it again by about 2.71828 times since I had seen it previously.

This is with a recall failure rate of about 5%, which I'm trying to maintain. (If it's nearing 0, then you're probably wasting time being challenged too frequently.)

This is on a data set used daily for 30 minutes, reviewing about 50 older questions and encountering around 3 new ones per day.

Comprehensive, as usual. An obligatory nitpick:

> At least with physical fitness there isn’t a precisely dismaying number indicating how far behind you are!

Many sports are precisely measurable and in such sports, the measurements determine who has won, to various resolutions. Shortest running time over a given distance, heaviest weight snatched, furthest distance jumped and so on.

Athletes in such sports can very accurately track their performance over time. As of yesterday I demonstrated that I can clean & jerk 25kg less than my lifetime record, for example.

I think the better analogy would be if you had some sort of workout schedule and you had missed n workouts. Just bad performance doesn't indicate much: if I score a 2 on all my flashcards today, that's bad, but I'm not 'behind' since the cards' next review will just be moved up as necessary to compensate.

For me it was absolutely surprising that out of the 200 Android apps, only 4 used spaced repetition but beeing an Android developer myself, I took the challenge to make it better: https://play.google.com/store/apps/details?id=de.leowandersl...

FluxCards provides spaced repetition learning with easy to edit and create own cards. You can sneak the answer line by line, you can finger-paint and the stats are fast and meaningful.

It is free to use with 0.5, 1, 2, 4, 8, 16 days intervals. Beyond that you need the premium version but your cards are yours, so if you consider it not worth the fee, you can export your cards to SD and use them in Anki.

And yes, spaced repetition rocks. I can only encourage any flash cards programmer to implement it, as all these apps that implement flash cards but no spaced repetition learning are just not worth your time.

I use SRS (Mnemosyne) for memorising traditional Irish tunes. It feels like each tune is a bit too much of a large chunk to remember at one time for best use of an SRS (you're supposed to chop the information up into the smallest element possible), but the benefits of the system are amazing. Before I started using SRS I'd just practise the tunes I liked and already knew!, with the new tunes, and anything in the middle got forgotten about. Now mnemosyne forces me to practise tunes that I don't know, and the number of tunes I'm fluent in has increased dramatically.

I'm also building a language learning startup that uses SRS. If you're interested in trying the system out in beta, or being notified when it's ready for general consumption, email me at languagebeta@jh.id.au and let me know which language you want to learn.

at kenHub[1], we built an online trainer for anatomy. It uses a game-like sequence of exercises designed to teach and test you at the same time about anatomy structures and their terminology. We've implemented a very basic spaced-repetition algorithm, but there's always more to improve.

One interesting thing we've found early on, was that when we encouraged people to actually "take a break", not to repeat too much, but instead let your brain process and come back to it later, people didn't like it. They wanted the ability to keep training and training. This resonates well with what the article describe about students' cramming tendency. So some time you have to manipulate even a solid algorithm, to fit with reality.


I combined spaced repetition with machine learning to attempt to predict memory recall for my thesis. Turned out fairly effective, correctly predicting recall up to 70% of the time.

Spaced repetition is pretty amazing. Adding records feels like you're entering them into your brain, provided you use it every day. I managed to go from knowing almost nothing to scoring 99% in a Japanese kanji exam once, entirely thanks to Anki.

My thesis is available here: https://github.com/jordwest/Thesis-Report/raw/master/report....

The online SRS I built is also available: https://github.com/jordwest/Membit

SRS software (I mostly used Anki) was invaluable to me when I was learning Japanese, both for vocabulary and the kanji.

I always wanted to make my own SRS, one that had ease of use , stability and data integrity at its core rather than cutting edge features. This year I finally set aside time to make it: feel free to try it out and let me know what you think. It is very simple (there is only one deck), but multiple decks will be the next feature to be added.


On top of that platform is an implementation of a rather interesting way to learn Japanese reading which it would also be great to have feedback on:


iikanji looks good, but have you seen wanikani? The concept seems similar. http://www.wanikani.com/ Some feedback: I know how to read all of the stuff I tried (the first 10 kanji), mostly thanks to wanikani. But I haven't studied much gramar, so I'm not always sure what the sentences mean. You should probably add translations for these in the back of the cards.

Does anyone know of any command line spaced repetition tools? if not could be a fun side project.

i am looking for that as well.. if you use emacs there are org-drill.el and flashcard.el:


I'm not really sure why you would want that. A CLI spaced repetition means you make it much harder to display images and LaTeX, which are going to be used in a lot of cards.

Anki is written in Python. Maybe you could fork it and wrap its core in a command line app.

It's great for learning facts and exploring a subject. Spaced repetition is fascinating. Especially in the realm of language learning. Vocabulary acquisition can be optimised by deciding the right order of words to maximise your ability to read a foreign language text. gwern has a link at the bottom of the article on modelling vocabulary lists as a travelling salesman problem. Feeding this into SRS Flashcard program is powerful. I guess one of the big problems ( if speed is your goal) is the creation of the cards themselves. The time to create the card is much greater than the time to learn the card and can be time a sink if you are trying to learn your first 3000 words of language.

I've been using Anki for learning Japanese. I've particularly appreciated the decks that include audio of complete sentences and have found that it has greatly improved my ability to understand spoken language.

One thing that I've wondered is whether it is a good idea to try to think up mnemonics for tricky words when using spaced repetition. It certainly seems effective in helping recall but I worry that I might train my brain to always rely on going via an indirect path to get to the information and that this will slow down my ability to understand real language.

Has anyone seen any research along these lines?

From my personal experience, using such a method doesn't cause problems. I studied the kana using Heisig's Remembering the Kana, where it connects each character and pronunciation to a mnemonic story. I'm long past the point where I actually need the stories to read kana, the sounds are simply tied to the characters now, with the stories available for recall if i have a brain flub and can't remember one for a bit.

I was working on mobile app that helped with spaced repetition as a side project. I built a pretty good looking user interface but ran out of time to work on it between school and my company. If anybody is excited about spaced repetition and wants to join me on my side project, I'd love to try to make a little bit of extra time to get it live. I really wish that the application already existed - my midterms would be going much, much better :)

Email me at sean@einstein.com

https://play.google.com/store/apps/details?id=de.leowandersl... would need a developer for a browser version. I have big plans and if you convince me, I might pay you for that or we agree on a share. It already has 10k users and a 4.5* rating.

I used anki throughout high school for classes and SAT prep - pretty much the main reason I ended up at an ivy league school (now a junior). It was especially helpful for classes like AP Bio / AP US History where there was a ton of pretty obscure stuff to memorize.

It's not so useful in college now that I'm majoring in applied math, but occasionally I'll use it for a humanities class. (throwaway b/c I always use throwaways)

For those using org-drill in Emacs, I have a write-up of my own card/directory layout and custom elisp to provide a navigation prompt to select which "category" you want to train: http://ixmat.us/articles/2012-12-01_usable-org-drill.html

I don't have anything constructive to say besides that I'm always impressed by your work, Gwern.

Thanks for all the references as always.

I use Anki every day. I think part of the project is in learning how to use it best. I have little interest in learning word-of-the-day or country names. I am more interested in learning new skills, and I shared the concern that it wouldn't work so well for that.

I think it's doing all right though. Here are some of the decks I'm most proud of:

- Using lilypond snippets and a midi-to-wav generator, I wrote a perl script that generates a deck for every (non-accidental) note on a guitar fretboard. It shows the fret position with a question mark, and then when you identify the note, it plays the pitch and shows the note on a treble clef. I'll probably expand it to include accidentals soon.

- Also using lilypond, I wrote a jazz theory deck for major chords. Questions like, "What chord quality and inversion is this?" where it plays a chord, and answer would be m7b5 (half-dim) 2nd inversion. My ear training has improved a ton. And, what is the key scale for bII7/iii in B major? (Plays the chord audio). Answer: B melodic minor or E lydian/mixolydian, playing and showing the scale on treble clef. And an effect like gwern described has happened. I haven't actually memorized the answer to every card in the deck (since the perl script generated about 2500 of them). Instead, I've gotten faster of doing the actual music theory processing in my head to calculate/recognize the right answer almost instantaneously. End result is that it has improved my speed in reading lead sheets. (It has not, however, improved my melodic improvisation abilities. :-) )

The rest of my decks are basically based off of Coursera courses. Unfortunately, I haven't finished any of them yet. I routinely run into the problem where partway through the course, the amount of time I need to spend keeping up with the new cards limits my time available to watch new videos and do new assignments, so I don't finish the course. I am now taking the Scala course again, and I'm hopeful that since I'm fully reviewed up through week 4, I might be able to power through and finish it with a deck that fully reflects the course.

This also means that I am improperly creating a separate deck for each subject matter, when Anki doesn't let you scramble them. I should be putting them all in one deck, but I kinda don't want to before a deck is "complete". I'm also a couple of chapters into "Learn You A Haskell", with the rest of the book on pause.

Finally, I can echo that practicing helps a lot. For instance with the Haskell deck, I got up to fully reviewed, and each card was still kind of hard. Then I spent a couple of hours playing with the first Project Euler problem. After that, the entire deck was much easier and faster. So, retaining familiarity still is somewhat different than learning the concepts. If I have a fully reviewed Anki deck from Odersky's Scala course (on coursera), it won't mean I know Scala. But if I go start a Scala project, the Anki knowledge will help me a ton, and then afterwards, the experience from hacking on the project will make the Anki cards much easier, maybe even to the point that I could delete a bunch of them.

BTW, that's feature I wish I had from the deck - some kind of advice for when it might be appropriate to delete a card. I worry that I'm doing the equivalent of writing pointless unit tests sometimes.

How much of a music theory background did you have before you started making those cards? What do you think is the minimum someone needs to know to be able to benefit from making such a deck?

Well, my college degree was in music and I always had a knack for theory, testing out of college courses etc. But you don't have to have university level music theory knowledge to study jazz theory. I'd say the prerequisites for the more complex jazz deck of mine are knowledge of scales (major, melodic minor, major bebop, minor bebop), chord qualities (M7, 7, m7, m7b5), chord inversions, and treble clef notation.

For the people reading this that do jazz, it's based off of Randy Halberstadt's concepts of planets and harmony. Scale choices differ from the classic guideline of being simply chord based. For instance, dmi7 does not necessarily imply a d dorian scale. If it's in the key of C at that point, then you'd instead play C major bebop.

> I worry that I'm doing the equivalent of writing pointless unit tests sometimes.

Unit tests never go away, though. Eventually a card will get scheduled for years out, and it takes a trivial amount of time to review, so it shouldn't be a problem. I'd imagine.

Any interest in sharing these decks? I'd love to use them.

I'm also interested.

Wow! Have you published those?

No... I got hung up on the idea of whether I wanted to try charging for the music decks, and never fully decided. Plus I want to clean up my git repositories before I consider releasing them. One of those low-priority todo items that nags at me once in a while. :-)

The Coursera decks I would absolutely release but I need to finish a course first, and I need to make sure they don't have assignment answers in them I guess.

Put them out there one way or another, free or not. That sounds too cool to stay cooped up! :)

Derek Sivers wrote a blog post on this back in january[0]

[0] http://sivers.org/srs

Rethinking the exam structure in US schools to better accommodate and motivate spaced learning would be a significant step toward improving education quality.

'cperciva noted this effect from his time at Oxford...

"When I tell North Americans about my time at Oxford University, one aspect of its undergraduate program inevitably surprises them: Final examinations. Rather than writing examinations at the end of each term, students in most subjects write a single set of examinations at the end of their final year, on which their entire degree performance is measured. (In recent years, some subjects have switched to annual examinations.) As terrifying as this might sound, the system has some merits: Whereas many students at institutions with termly examinations study the day before an exam and forget the material the day after the exam, attempting to 'cram' three years worth of material at once is largely an exercise in futility; as such, Oxford's examinations often provide a better measure of how much material a student has learned and retained. Given some of the conversations I've had with — and code I've seen from — people who have received computer science degrees from major universities and are actively working as software developers, I think that last part is very important: There's a lot of people out there who have forgotten some very basic — and very important — material" (http://www.daemonology.net/blog/2012-10-08-software-developm...).

You can obviously practice spaced repetition regardless of your school's exam structure, and Cal Newport specifies a method for doing this in "The Art of Stealth Studying: How To Earn a 4.0 With Only 1.0 Hours of Work" (http://calnewport.com/blog/2007/10/03/the-art-of-stealth-stu...).

However, most students are unaware of this method nor have the disciple to follow it on their own so to realize its benefits on a mass scale, we would need to revamp the education system around these principles.

And as 'qwern noted, sleep is a key factor in spaced repetition learning. Research shows sleep is when the brain prunes itself by separating signal from noise...

"Sleep researchers at the University of Wisconsin-Madison School of Medicine and Public Health believe it is more evidence for their theory of 'synaptic homeostasis.' This is the idea that synapses grow stronger when we're awake as we learn and adapt to an ever-changing the environment, that sleep refreshes the brain by bringing synapses back to a lower level of strength. This is important because larger synapses consume a lot of energy, occupy more space and require more supplies, including the proteins examined in this study."

"Sleep — by allowing synaptic downscaling — saves energy, space and material, and clears away unnecessary 'noise' from the previous day, the researchers believe. The fresh brain is then ready to learn again in the morning" (http://www.sciencedaily.com/releases/2009/04/090402143455.ht...).

Spaced repetition, also known as rote memorization, is a terrible way to "learn" anything. The trouble is, you aren't learning at all, you're hacking your brain to force information into it that has no relevance and really doesn't want to be there. It's not a useful or pleasurable activity. It definitely won't make you smarter. Even Mr. Supermemo himself admitted that after FIVE YEARS of constant daily marathon memorization, 60% of it is gone within the first year of stopping (http://supermemo.com/articles/kowal.htm). The brain does not want to work this way.

I'll propose an alternative to rote memorization for learning: critical thinking.

> Even Mr. Supermemo himself admitted that after FIVE YEARS of constant daily marathon memorization, 60% of it is gone within the first year of stopping (http://supermemo.com/articles/kowal.htm).

From the link:

> This is a translation of a popular scientific article about memory and learning written by J.Kowalski, Poland, for Enter in November 1994. Translator comments are placed in square brackets... > Simulation experiments based on Wozniak's model of learning show that a student who stops repetitions after a 5-year-long work with SuperMemo is likely to forget 60% of the learned material in the first year after the cessation! [this figure has later been proven exaggerated]

Isn't that a rather dishonest description?

> I'll propose an alternative to rote memorization for learning: critical thinking.

Memorization fosters understanding: http://www.gwern.net/Spaced%20repetition#abstraction

Exams don't reward critical thinking, they reward you knowing your stuff. Same goes for life. Understanding a topic isn't enough, and for example medical doctors better know more about the diseases they are treating than they understand on an academical level with lots of critical thinking.

This varies in different subjects of course. In computer science and business degrees you just don't have to be able to pull as many details out of your hat as in biology, medicine or law. The effort and work required to master a math-heavy homework assignment can be enough rote memorization.

I've come to think of rote memorization as actually the most natural way to learn in a time efficient manner. I have yet to witness this approach compromising my understanding in any way.

As a member of various communities dedicated to language learning, spaced repetition software is very widespread. Languages require memorization, you simply need to remember the words so they eventually become second nature.

Sometimes in life the answer can't be reasoned about, you just have to buckle down and memorize it.

Spaced repetition is just choosing the frequency with which you engage in an activity. Whether that activity is memorizing random-facts w/o context or something more in-depth is up to you.

There are a lot of situations where you have to know a lot of facts and there's no way around rote memorization. Critical thinking isn't enough - you need a corpus of information. For example, if you're in a medical field, you'll have to memorize tons of facts about anatomy, physiology, disease states, drugs and other treatments, treatment algorithms, etc. Same with a language - you have to memorize thousands of words, lots of idioms, thousands of characters if it's Japanese or Chinese.

I kind of agree with you. Spaced repetition is raping your brain. School makes you want to do that for stupid stuff. Life sometimes brings along stuff, your brain is more susceptible for.

I use fluxcards that I wrote myself and have some 2600 facts that I'm learning.

The Spanish words work smoothly, cause I actually want to really not use them wrong in my daily life in Chile.

The friends and family birthdays work so so. Yeah, I admit, I don't care so much. I learn them as a favor to my friends that enjoy when I remember their birthday but I find birthdays kind of unimportant.

Then there is Farsi. I was in Iran and for my Iranian friends I learn some 35 words of Farsi. That is the worse. It makes no sense and my brain just won't accept these weird words to stick. Every time I want to use these words in a chat with my Iranian friends, I don't dare cause I just would mix things up and I don't get these words above the one month level in 2 years. If I would want to give up, I would have to delete these cards and acknowledge that I will not learn this stuff. At least not now, but I do it for them, so why not repeat one of these cards per day for them? Costs 3s and eventually I get my * brain to not forget these "important facts".

Critical thinking would only help me to admit that I will not learn this or that now. If I took the conscious decision to learn stuff anyway I know nothing better than spaced repetition.

Critical thinking and memorization are not intrinsically at odds with each other. Rather, they are separate tools that are powerful when used together. If your critical thinking is strong, it can be used instead of memorization, but relying on this is a crutch, and it hurts you in the long run.

I fell into this trap in math classes all the time. My exams would be covered in work where I would do something like reinventing the chain rule from the definition of the derivative. The definition of the derivative is elegant and obvious in its workings, and therefore easy to remember based on critical thinking. The chain rule commonly useful, but it's a special case, and the end result is opaque. So I wouldn't remember it, and I'd just figure it out from scratch on test day.

As it turned out, that got me plenty far. But my grades would have been better if, having understood how to derive it, I had cached it in long term memory. I would have made fewer mistakes, had less work to check, and more time to check it.

These days the same thing happens when I'm working on a math problem for practical purposes. I don't have a lot of these formulas committed to memory, so instead of spending time on actually solving the problem I have, I have to spend time rebuilding math from scratch.

It's the same basic concept as caching and memoization in programming. It would be a mistake to write every function as a giant look up table to handle each parameter combination. But if your code keeps calling the same expensive pure function with the same parameters, you can greatly speed it up by caching the result.

That's funny because you can write a deck to help you learn concepts of critical thinking. Necessity, Sufficiency, Contributory Causes; Propositional Logic, Predicate/First-Order, Modal... the deck won't TEACH you critical thinking, but it will make it much easier for you to learn it.

Learning the names of things is not learning those things. This is particularly true when you're learning ways of thinking.

What is important is not that you can identify and name methods of reasoning. It is important that you can apply the appropriate reasoning method to problems in your life.

Applications are open for YC Winter 2022

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact