
Bloom's 2 sigma problem - gballan
https://en.wikipedia.org/wiki/Bloom%27s_2_sigma_problem
======
austenallred
We have implemented this methodology at Lambda School (YC S17 -
[https://lambdaschool.com](https://lambdaschool.com)) with _staggering_
results. We only get paid if our students are hired making more than $50k/yr
as a software engineer or data scientist, so we’re more motivated than most to
try things like this, even if they’re expensive.

First, mastery-based progression. At the end of every week we have a “sprint
challenge,” where a student builds a project using everything that was learned
that week. If they didn’t grasp the material, instead of pushing them forward,
they stay back and repeat that week until they have mastered it.

Of course, this is very expensive to do, and therefore will never be done in
most classroom environments (you erode economies of scale), but we find that
it’s worth it.

The results from that alone have been astounding. We don’t have a
scientifically-designed study that shows a standard deviation increase (too
many other variables are simultaneously changing), but the difference in
student competence and ability at graduation was jaw-dropping.

The next aspect is probably an order of magnitude more expensive to implement
- individual tutoring. We obviously can’t pair every student with a tutor all
day (our instructors are experienced engineers and therefore very expensive),
but we keep every student in a group of 8 students with 1 TA (a student who is
at least 4 months ahead, performed well, and stays on for a short contract).
Importantly those students are happy to be paid to review the material, and
cost about 1/5 of what an experienced instructor does.

In such a small setting a TA can be there to individually work with a student
(though for a little less than an hour a day).

Again, I don’t have any good way to scientifically prove this, but the level
of ability our students have now at the end of our 30-week curriculum (which
can become longer due to mastery-based progression) is incredible.

If you’re an engineering or hiring manager and want to try it out shoot an
email to hiringpartners@lambdaschool.com and ask to interview a student who
had never programmed before the 30 weeks to see what they can do.

I thoroughly believe Bloom’s 2 Sigma Problem is the most important study done
in education... ever.

~~~
robbiep
How can you have great results but not be able to prove it? I see this
perception coming up again and again, and it has, in my experience, always
been due to 2 factors: the time has not been taken to determine a study
design; or the improvement doesn't exist.

I'm not saying your method doesn't work, but surely if it is your business you
could take the time and money to prove it (i'm in a similar situation with
mine, although in the health space no-one trusts you unless you can publish,
which raises a whole other set of problems - ie shitty studies)

~~~
austenallred
We have had a huge swing in results, we can show that, but we’re
simultaneously improving and testing a lot of things across the entire company
(we’re 18 months old and are iterating in everything quickly), so it’s not
like we have a well-designed study that isolates this as the only cause.

We could do it now I suppose, but you begin to run into ethical questions that
don’t mesh with our vision: “let’s give n students a subpar experience to
prove a point” would be a tough sell.

~~~
robbiep
have you thought of doing a crossover study and reducing fees for a portion of
a cohort? If your method has results, then you should be able to catch them
up, and then you can go to market with real numbers (alternatively, it might
hint towards class sizes or other changes you can make to maximise efficiency)

~~~
austenallred
I’m sure you could find a way to prove it’s better; I’d rather just do a great
job for everyone and let the results speak for themselves

~~~
robbiep
Not to harp on about this too much, but that is exactly the problem: results
that speak for themselves but don’t have any rigour are sham results and leave
you open to accusations that you are simply using marketing flash or selective
statistics.

Now, a real study can open you to this as wel, but it is harder when you have
some ostensibly objective results.

There is no such thing as achieving results unless you can prove it, it
doesn’t make sense

~~~
austenallred
> there is no such thing as achieving results unless you can prove it

That’s not true. Things happen regardless of whether or not they happen within
the guidelines of a scientifically designed/valid study.

Students are getting better. Was that correlation or causation? I’m not going
to force students to have a mediocre experience to prove it.

~~~
robbiep
In other words, ‘just trust me’. Your business is not self evident

------
andres0l1
Bloom’s 2 sigma problem was a fascinating study. Regarding the tutoring
aspect, at College Connections we want to take this a step further. We use our
in-house-designed software MentorMatch to connect students and tutors based on
the strength of their personality compatibility. It has been said that “The
mind is not a vessel to be filled, but a fire to be kindled.” This echoes
anecdotal evidence that students enjoy their work more when they get on with
their teacher. Furthermore, we will reduce the cost of private tutoring by
targeting schools recent alumni who are happy to tutor students in their
schools for a lower rate than professional private tutors. We are currently
hoping to conduct a large scale study with few schools to see whether
personality-based tutoring matching can actually yield results beyond two
sigma! If you want to know more, check this out:
[https://medium.com/@maxdunhill/matchtech-white-
paper-4a6e4e0...](https://medium.com/@maxdunhill/matchtech-white-
paper-4a6e4e05a93e)

------
hef19898
That rings true. At one point I came to the conclusion that high attrition
rates are first and for most an indication for lacking training / teaching
quality and not difficulty of the course or trainjng program. As one (former?)
Pararescue instructor stated, 30% always drop out, 10% always pass and the
rest is up to the instructor (unfortunately I lost the source for that...).

------
imranq
The key thing that self studiers don’t realize is that you don’t know what you
don’t know. And frequently those things are very subtle but build up over
time. Luckily with the internet, any motivated student can find great teachers
to help them.

~~~
hef19898
That's true, and the resulting basic mistakes can be extremely hard to get rid
off. Like bad foot work in boxing, once you do it you have ahrd time
correcting it later.

~~~
imranq
That’s a good example. Piano is also a classic foundational problem

------
acjohnson55
I was a teacher. Before doing my teacher prep program, I tried to gain
relevant experience tutoring after school. When I actually started teaching, I
realized very quickly that it's a completely different ballgame.

Teaching isn't just a one way upload of knowledge. It's an interactive process
and feedback from the learner is extremely important. Not just for correcting
misconceptions, but also for motivation, partnership, and dialog. One-on-one,
the loop is very tight.

That's one reason I'm skeptical about technology making a huge impact any time
soon. Our technology is just really, really far away from being able to
stimulate tutoring.

~~~
sitkack
Technology as a replacement, no; as an augmentation, definitely. A computer
could be a powerful tool in improving tutors effectiveness. I just saw a paper
on using DNN for generating questions from text which could be used for
interactive flash cards (spaced repetition, quizes, etc).

~~~
acjohnson55
Even for augmentation, I'm bearish, short term. We'll eventually get it right,
but not before any kids I have are through schooling.

------
ulucs
I think the more exciting takeaway from this is the Mastery Learning method
causes a one sigma gain (from 50th to 84th percentile) _for free_ whereas
private tutoring lets us gain another sigma (from 84th to 98th) for 10x-15x
the tutoring cost.

There’s also something else that’s driving my skepticism further. We know that
deliberate practice _works_ [1], and the possible effect size is much larger
than two sigmas (think more like from a good chess player to grandmaster [2]).
However, at this point the effects of both deliberate practice and tutoring
start to blend in too much. The mastery learning aspect already seems like a
methodology similar to deliberate practice. Does tutoring cause the additional
sigma by letting more deliberate practice happen, or by some other vices? If
we’re stuck with a 30-pupil classroom as a tutor, is it possible to extract
those vices from private tutoring to help our class succeed?

[1]
[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3102783/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3102783/)

[2] [http://slatestarcodex.com/2017/07/31/book-review-raise-a-
gen...](http://slatestarcodex.com/2017/07/31/book-review-raise-a-genius/)

~~~
pmiller2
Mastery learning is not costless. The cost is time: time for the student and
time for the teacher.

