
Selection bias is the most powerful force in education - gfredtech
https://fredrikdeboer.com/2017/03/29/why-selection-bias-is-the-most-powerful-force-in-education/
======
alejohausner
A while ago I read a review of a book called "The trouble with diversity" by
Walter Benn Michaels. I was struck by his claim that universities serve to
"launder privilege into qualifications".

To me, this helps to emphasize the importance of class in America. Rich kids
go to Harvard, after graduating from wonderful high schools, helped by private
tutors, and, _most importantly_ , knowing that their wealth and their parents'
connections will guarantee them success. But since America is supposedly the
land of equal opportunity, those kids must put on a facade to the world, which
tells other people a lie: I succeeded because I'm smart, not because I'm rich.

~~~
vostok
> I succeeded because I'm smart, not because I'm rich.

This is a fair point, but in many cases you need to be rich, hard working, and
smart. This is a major improvement over the old system where you just need to
be rich.

This is not a uniquely American phenomenon. Top universities in the UK and top
grandes ecoles in France are also dominated by the middle class. I'm sure
other countries are like that too if not worse.

I think any system with inequality will have this issue. Parents want to help
their children and some parents will have a greater ability to do so.

This is not unique to capitalism either. I'm sure those of us from the Soviet
Union will be intimately familiar with the concept of блат, which is
significantly more pernicious than wealthy children doing better because they
have tutors.

~~~
frubar
>This is a fair point, but in many cases you need to be rich, hard working,
and smart.

Like Paris Hilton, Kim Kardasian, etc.? No, rich is still the only
requirement. Those other things just give you more options.

~~~
haukilup
> >This is a fair point, but in many cases you need to be rich, hard working,
> and smart.

> Like Paris Hilton, Kim Kardasian, etc.? No, rich is still the only
> requirement. Those other things just give you more options.

For every person you cite as being wildly successful only because of the money
they were born into, you could also cite people who came from nothing and
became multi-millionaires. Or people who were born rich and didn't succeed.

You're not wrong - there are people who only needed to be rich to begin with.
But giving examples like you have doesn't counter the above point.

~~~
Retric
If you have 10 rich people with a 50% shot of making it : 99,990 people with a
0.005% shot of making it then the number of people with both backgrounds who
made it is reasonably balanced. But, in no way is this a meritocracy.

------
calebl
Loved this article. It presents a compelling case that most colleges are
providing near-equivalent _education_ , and that the variance is mostly in
selection bias. My only objection: at the end, the author takes issue with the
fact that it's nearly impossible to get people to take selection bias into
account when choosing where to send their kids. However, there's another
possible factor that I think parents might be considering: the network of
connections that I build when I go to school X. Even if Harvard gives my
daughter the same educational quality that, say, Florida State gives her... if
she's likely to form a network of friends that are all approximately 100x
higher in net worth, then if I care about her long-term economic outcomes, I'd
still try hard to send her to Harvard, no?

I'm not saying that's ideal, and it's most certainly an outcome of the exact
selection bias you're talking about... just that it's a semi-rational choice
for the parents, because school choices can often be about non-educational
effects.

Just my two cents, though... I could be way off. And regardless, thanks for
the article.. definitely made me think about the topic more than I had.

~~~
fragsworth
I think the network is overrated. Everyone I've known has had the same
experience - immediately after graduating, everyone gets a job, moves far away
from one another, and slowly drifts apart from the friends they met in
college.

You'll get a similar network working at your first job. And then your second
job. The group you met in college is of little consequence in the end.

~~~
taneq
To give an opposing anecdote, I've landed more than half of my full time
permanent positions through personal recommendations or references from people
I know from university. And these jobs have by far been more interesting
and/or better for my career than ones I found through recruiters or vacancy
ads.

~~~
wott
So, recommendations and references from people who never saw you at work.
There's something wrong about that. I am not saying it doesn't happen, it
does, but it still makes very little sense.

------
jstewartmobile
Parents, teachers, and students are so focused on "education" as a pedigree
and job pipeline that if they don't see an immediate connection between a
topic and a paycheck they're like "Who cares?! Why are you wasting my kid's
time on that?"

When the whole thing is an exercise in competitive pedigreeing, of course it's
going to be gamed. If it were more about human development, we'd be focusing
on the deltas instead, and they'd be harder to fudge.

~~~
gnaritas
They're not wrong to worry about that, it's a rough world out there and
getting a good job is vastly more important to most people than "human
development". Parents don't send their kids to college to become better
people, they do it to make sure they have a skill that can lead to a career so
they can survive on their own. Only the rich have the luxury of being
concerned about something other than a job.

~~~
banach
On the contrary. The less you have, the more you stand to benefit from
appreciating other things than possessions.

~~~
bpodgursky
Yes, I'm sure a generation trying to save for a house under burden of student
loans appreciates your sage bullshit advice.

Must really appreciate the lack of affordable health care and never being able
to afford a family. But at least you can appreciate it!

~~~
banach
Philosophies of life like Buddhism and Stoicism came about in times that were
much less comfy than even the sad reality we have to endure. I am not saying
people shouldn't demand things of society (quite the opposite, I think the
level of organizing among the poor is the root of many of the issues we are
seeing today), but I am saying that they can benefit from separating their
happiness from the fulfillment of those demands.

------
glangdale
This is a good article. I wish I could force everyone who pontificates on
schooling in Australia to read it.

We have an enormous private school system (part govt-funded, which is gross)
that can get rid of students as they please. Then they go on to verbally dump
shit on the public system, where they get to dump their problem students.
Nice.

We also have a real fetish for selective schools, which are essentially
selection bias driven to a huge extent. Any suggestion that this isn't a good
idea is met by an chorus of "you must hate smart people".

~~~
sien
The private school system in Australia saves the government money. Per student
funding in the private system is lower than per student funding in the public
system. This money is then available to spend more on government schools.

It also keeps the government system honest because people can and do leave
schools that don't work such as many of those in poor areas in the outer
suburbs of the big cities and regional areas. If there were no private system
the only way to get into better schools would be to own very expensive houses.

Meanwhile, the selective schools show that the government can provide a good
education for highly able students as well. In NSW ~18 of the top 20 high
schools are government schools.

Personally I think most people are unwise to send their kids to expensive
private schools and they would help their kids much more if they put the money
toward investments. But they are free to waste, in my mind, their money to
reduce the cost to the tax payer of education.

~~~
Paul-ish
> The private school system in Australia saves the government money. Per
> student funding in the private system is lower than per student funding in
> the public system. This money is then available to spend more on government
> schools.

Isn't this selection bias again? More difficult (expensive) students may get
shunted into the public system.

~~~
sien
A number of private schools roughly kick out people in the bottom 10-20% of
results. They aren't necessarily that more expensive to teach, the private
schools just don't want them.

The really expensive kids (say 20K+ per year) are those that are seriously
disabled and they almost all go into the government system. But, if the
government only has to pay ~1/3 of the cost of about 1/3 of students who are
in the private system there is more money available for them.

~~~
stevenwoo
There is a weird situation in Texas in the USA. The state is limiting the
amount of funding for each local district to 8.5 percent for special education
with the end result that families end up going to private schools to get
special education. The one family in this school ended up paying 50,000 per
year for private school for two kids with a readi ng disability.
[http://www.houstonchronicle.com/denied/7/](http://www.houstonchronicle.com/denied/7/)

------
bluGill
Sending a troubled kid to a great private school probably will cause the kid
to do better: because all the other students are doing well the bad students
peers are working hard so the bad student sees that example and is likely to
succumb to peer pressure and do better.

For this to work the troubled kid needs to be kept out of the gangs and
whatever else outside of school will lead to bad examples. However selection
bias already does this: parents who care enough to send their kid to a private
school are involved enough with their kid that they would probably do this
anyway. In short these are kids that might have been troubled, but they still
would have been at the top of the troubled group.

~~~
sololipsist
> Sending a troubled kid to a great private school probably will cause the kid
> to do better ... However selection bias already does this

Funny thing, though. We've done a bunch of studies on this and it doesn't
actually help. The people who care about their kids more getting them into
better schools, yes - but placing the kids of parents who don't care into
better schools, no.

This has been shown time and time again. The best results to the contrary
managed this by ... drumroll ... designing selection bias into the study.

~~~
throwaway613834
> Funny thing, though. We've done a bunch of studies on this and it doesn't
> actually help.

> This has been shown time and time again.

One would think a link to at least _one_ study would be appropriate here?

~~~
simonh
The original article already provided evidence, based on analysis of the
performance of similar students who do and do not get into a selective school.
As it says...

"Except that attending those high schools simply doesn’t matter in terms of
conventional educational outcomes. When you look at the edge cases – when you
restrict your analysis to those students who are among the last let into such
schools and those who are among the last left out – you find no statistically
meaningful differences between them."

So unless you're calling out the original article, how about offering counter-
evidence?

------
spunker540
I think the real problem described is that people are judging schools based on
the performance of their students. There are other objective factors that
won't be as affected by selection bias such as: breadth of extra-curricular
options in sports and in the arts. Number of AP classes offered. Teacher to
student ratio. Budget per student. Does the school have newer computers,
working instruments, modern facilities?

~~~
richdougherty
Are you saying that people should judge schools based on the inputs that go
into the schools rather than on any effects the schools produce in their
students?

~~~
spunker540
The article points out the high degree of selection bias with schools and how
that can make it very difficult to make apples-to-apples comparisons between
schools.

I was just trying to mention ways to compare schools that won't be affected by
the selection bias described in the article.

~~~
AstralStorm
A proper analysis will attempt to identify such factors and how important they
are. Since the sample size is big enough it is not impossible to determine how
much of the effect is those other measurable factors.

Selection bias might be measured by the rejection rates on admission (esp.
scores rejected)

------
hyperion2010
That tricky thing when dealing with humans: are they learning because of our
efforts to teach them or in spite of them?

------
guyzero
This has long been my assertion about elite undergraduate engineering schools.
The teaching is usually just OK but their main advantage is that they get to
"skim the cream" from the applicant pool.

------
leongrado
I'm from the midwest and most people who weren't aiming for Ivy Leagues just
took the ACT.

~~~
jblumstein
Yah. It's weird that de Boer argues selection bias is in play in SAT
participation between MS (3%) and CT (88%) when 100% of MS high schoolers take
the ACT. That seems to be what's happening here: the kids in MS taking the SAT
in addition to the ACT are the ones aiming for elite schools.

~~~
boomboomsubban
>the kids in MS taking the SAT in addition to the ACT are the ones aiming for
elite schools

Which is an extreme selection bias. The ACT should have been mentioned to
provide more context, but the actual argument works without it.

~~~
jblumstein
You say it better than I do. Agreed it's an extreme selection bias, just want
to point out that it's slightly different than the one de Boer seemed to be
pointing out.

~~~
boomboomsubban
It's still the same bias he argues, the three percent are the most motivated
and prepared. Why the other 97% skip it isn't relevant in this argument, but
the number is independently shocking without the context. I fell for it until
the ND mention, and I took the ACT.

------
terrahutte
My school has a large Chinese international student body. The school is
marketed as being diverse, but really those students are from wealthy families
and can easily afford the 40k+ tuition. In the end, the campus isn't diverse,
it is segregated. But at least the uni has more money for research.

~~~
sabujp
majority of that mooney doesn't go to research, it goes to admin

------
WalterBright
"People involved with the private high schools liked to brag about the high
scores their students scored on standardized tests – without bothering to
mention that you had to score well on such a test to get into them in the
first place."

Few private schools have entrance exams. Most only care if the tuition check
doesn't bounce.

------
LiweiZ
Education is about doing things at scale. Learning is about individual gain on
delta. I believe better personal tools for learning could do a better job for
individuals on average.

------
jgalt212
He was wearing my Harvard tie. Can you believe it? My Harvard tie. Like oh,
sure he went to Harvard.

[http://www.imdb.com/title/tt0086465/quotes](http://www.imdb.com/title/tt0086465/quotes)

------
ngrilly
This is simply the best article I've ever read on this topic.

------
ajuc
> People involved with the private high schools liked to brag about the high
> scores their students scored on standardized tests – without bothering to
> mention that you had to score well on such a test to get into them in the
> first place. This is, as I’ve said before, akin to having a height
> requirement for your school and then bragging about how tall your student
> body is.

Of course, that's a bad analogy, because average height isn't correlated with
the amount of time teachers have to spend on OTHER kids, but average grades of
the kids - is (inversely), so parents still prefer schools where other kids
are as "smart" as possible, because that indirectly benefits their kids (more
attention of teachers for their kid, higher expectations, peer pressure to
perform, etc).

So - selection bias is one of the reasons kids do better, but that doesn't
mean it doesn't also make teachers do better.

By the way, in my country (Poland) public universities are considered better,
than the private ones (because of historical reasons, and the way free
university education works). There's still selection bias (best students go to
public universities and don't have to pay for education, so these universities
have the best students and teachers, and the best results), but it's mostly
unrelated to students' wealth (apart from extremes where kids don't have food
or time to do homework in primary school, etc).

Do you still consider selection bias in education a bad thing, if it's only
about results, not about your family, connections, wealth, etc.? It's easier
to teach a group of people on similar skill level, than a group of people
where a third is bored, a third is learning, and the rest don't get what's
going on.

~~~
AstralStorm
Selection bias makes it also easier to implement special measures. You know
where the hot spots are and they are not diffuse.

(FWIW I live in Poland and had a short run with our private education system
on both ends.)

What has to be fought is discrimination in employment based on the origin of
degrees. All too common. There is enough of this based on connections already.

------
nichtich
I've read the research about Stuyvesant, and several other articles with
similar theme. The data and reasoning seems quite convincing. It's just I
can't reconcile it with my personal experience.

All these studies seems to suggest it doesn't matter much where you go to
school, everything is already decided. But I _know_ I've learned things in
school, things that I can't expect to learn quite systematically should I just
teach myself. And I also know there are schools that don't teach much, where
teachers are exhausted for just maintaining order and keep violence to an
acceptable level. It seems pretty obvious that I wouldn't learn much in this
kind of environment. Maybe I can still get to a similar level as I am today
but it be would much much harder.

------
lifeisstillgood
""" You believe that making your state’s high school graduates more
competitive in college admissions is a key aspect of improving the economy of
the state. """

Well there's your problem. The link between educational attainment of a
country and productivity is weak at best. (Switzerland being a good example of
low university attendance and outrageously good economy)

Ha Joon Chang was quite persuasive on this - and suggests that the imaginary
governor of the article should look to fix his states economy with focusing on
improving the eco system of the economy - longer term more patient capital,
regulatory regiemes that force longer term investment and behaviour, picking
winners and protecting nascent growth industries. All the bad things.

------
gnicholas
> _People involved with the private high schools liked to brag about the high
> scores their students scored on standardized tests – without bothering to
> mention that you had to score well on such a test to get into them in the
> first place. This is, as I’ve said before, akin to having a height
> requirement for your school and then bragging about how tall your student
> body is._

Not quite — kids don't get shorter over time, but they can easily get worse
grades/scores. It's true that having a screening test makes it more likely
that your students will score well on other tests, but it's not a guarantee
(as it is with the height example).

~~~
aidenn0
Yes quite; for example, kids who are in the 95th percentile of height at 9th
grade might be only in the 60th percentile of height at 12th grade.

~~~
gnicholas
The example wasn't about height percentiles, which can change. it was about
raw height. If his example had been about percentiles, then it wouldn't have
been such a stark no-brainer. Because a high school that admits only taller
students would be mostly girls, and by 12th grade they would be shorter than
many/most boys.

I was taking issue with the example as given. If it were framed the way you
suggested, it would not have had the rhetorical power that it did (at least at
first glance).

~~~
aidenn0
It is assumed that schools bragging about how tall their students are are not
doing so in a vacuum, and that they aren't bragging that their 12th graders
are taller than the other school's 9th graders, no?

------
dalbasal
In education this is particularly bad. Students and parents also steer into
selection bias under the assumption that peer groups are important.

But, to generalize the point into public discourse broadly: "Statistics are a
problem."

A lot of public discourse revolve around quantitative arguments like this.
Police shootings, women's 83 cents on the dollar, British Muslims' on Sharia,
unemployment rates...

Even in the more intellectual branches of public discourse, bad arguments with
a similar flavour can be the dominant ones.

------
thisrod
I'm amazed by the range of completion rates in that first scatter plot. Here
are the Australian figures, for contrast. Note that living in the NT is
strongly correlated with being Indigenous. (Indigenous Australians are the
people formerly known as Aborigines.)

[http://www.mitchellinstitute.org.au/fact-sheets/senior-
schoo...](http://www.mitchellinstitute.org.au/fact-sheets/senior-school-years-
school-completion-uneven-across-australia/)

~~~
bwanab
I assume your amazement is because some states have such low rates. If so,
this is very misleading since there are two competing tests in the US (ACT and
SAT). The states that have low rates for SAT almost all have very high rates
for ACT and vice-versa.

In fact, looking at the chart you can see that it is a split distribution.

------
Nelkins
A professor at the University of Rochester (my alma mater) has begun a project
to better evaluate the value of a liberal arts education, and to try and
quantify what impact it truly has on the career trajectories and future
earnings of graduates[1].

[1]
[https://www.rochester.edu/pr/Review/V79N6/0304_lennie.html](https://www.rochester.edu/pr/Review/V79N6/0304_lennie.html)

~~~
mulmen
The benefits of a higher education are more than just career path and
earnings.

~~~
mmirate
Perhaps they are, perhaps they aren't; I'm not a Doctor of Education, so I
can't and won't claim that I know the answer to that question.

 _My_ question to you, is applicable regardless:

As an industry-destined BSCS student, in what ways does my future improve as a
result of my education, except for career path and earnings? And if there are
such (nontrivial) ways, then why should I _care_ about them?

~~~
mulmen
I can only speak from personal experience but I grew a great deal as a person
while pursuing my B.S. I had experiences and met people I never would have
otherwise. I very nearly stayed in my small rural town working retail and
farming jobs but my education opened doors to new people and experiences. My
worldview is much different now than it was before college and my life is much
more fulfilling on a personal level.

While I was fortunate to select a degree that had positive economic benefits
for me that is far from the only benefit. My B.S. was far from just a job
training course. I studied a variety of topics and was exposed to ideas that I
never considered before. I learned to think much more critically and to be
more open to the world.

You should _care_ about this because money and a job are not everything in
life. I know that sounds cliche but I am finally reaching a point in my life
where I understand that. My education helped me reach this point.

I really cannot say enough about how much my education means to me. Even if I
did not leverage that education to start my career I would still consider it
to be completely worth it.

~~~
mmirate
> I very nearly stayed in my small rural town working retail and farming jobs
> but my education opened doors to new people and experiences.

So, that still falls under career path and earnings.

~~~
mulmen
It doesn't, please read the rest of my post.

------
RhysU
A simple thought experiment.

Consider bussing kids from 'good' places to bad and from 'bad' to good. See
after how long their outcomes become equivalent. Do so at different grade
levels to measure the convergence time versus age of displacement.

And now for the actual experiment: how do you impactfully present such
results? I assume no outcome. Just the existence of an outcome.

I posit that priceless data would be worthless in American society.

~~~
Analemma_
How many parents, who probably paid a lot of money and work two full-time jobs
to live in the "good" place so their kids could go to the good school, are
going to be OK with this experiment? Do you plan to force them, if there
aren't enough volunteers to get statistically significant results?

~~~
santaclaus
San Francisco, some of America's most expensive real estate, doesn't tie
school assignments to geography. The system hasn't been voted out yet.

~~~
haihaibye
30 percent of children in San Francisco attend private schools — the highest
rate of private-school attendance in California, and the third-highest in the
nation

------
novaleaf
The article's hypothetical gubernatorial scenario is a big turn off. It states
that Mississippi has higher SAT scores than Connecticut so Connecticut's
Teacher Unions are bad.

As the article doesn't at all remark on the suspicious causality of that, it
makes me more skeptical of the rest of the article.

~~~
cefstat
That’s not at all what the article claims. It presents a hypothetical
scenario, based on real SAT average scores, where somebody can reach a very
wrong conclusion (teacher unions are bad) based on this data. Then the article
proceeds to explain how the difference in the average scores can be attributed
to self-selection among students who take the SAT rather than unions. So, the
article not only remarks on the suspicious causality but the suspicious
causality is the article’s main point.

~~~
novaleaf
I understand what you are saying, but really don't think that suspicious
causality is what the article is about at all. It's about selection bias. (not
the same as causality)

Indeed after reading the article I feel it is generally well written and
compelling. However the contrived teacher union scenario is a big distraction
for me as I was reading the article hoping that issue would be addressed and
it wasn't. It would have been much stronger without this gaping logic hole
introduced in the beginning.

------
mbillie1
Nothing of substance to contribute but I'm immensely pleased to see Freddie
DeBoer writing again.

------
grandalf
I think a corollary of the author's point is that _by the time someone is
college age_ , selection bias is the most powerful force in education.

I'd be curious to see a study of younger people to see if perhaps there are
some assimilation effects.

------
programminggeek
Someday, people will wise up to the fact that success has more to do with
psychological programming than resources. Actually they won't because they
won't ever get enough of the programming themselves to realize it.

------
neves
I like the way the best public schools in Brazil select: randomly. In the
first years is just a draw, in the advanced ones they have just a proficiency
test, and then draw among who not failed.

