Hacker News new | past | comments | ask | show | jobs | submit login
Selection bias is the most powerful force in education (fredrikdeboer.com)
385 points by gfredtech on Aug 15, 2017 | hide | past | web | favorite | 178 comments

A while ago I read a review of a book called "The trouble with diversity" by Walter Benn Michaels. I was struck by his claim that universities serve to "launder privilege into qualifications".

To me, this helps to emphasize the importance of class in America. Rich kids go to Harvard, after graduating from wonderful high schools, helped by private tutors, and, most importantly, knowing that their wealth and their parents' connections will guarantee them success. But since America is supposedly the land of equal opportunity, those kids must put on a facade to the world, which tells other people a lie: I succeeded because I'm smart, not because I'm rich.

> I succeeded because I'm smart, not because I'm rich.

This is a fair point, but in many cases you need to be rich, hard working, and smart. This is a major improvement over the old system where you just need to be rich.

This is not a uniquely American phenomenon. Top universities in the UK and top grandes ecoles in France are also dominated by the middle class. I'm sure other countries are like that too if not worse.

I think any system with inequality will have this issue. Parents want to help their children and some parents will have a greater ability to do so.

This is not unique to capitalism either. I'm sure those of us from the Soviet Union will be intimately familiar with the concept of блат, which is significantly more pernicious than wealthy children doing better because they have tutors.

concept of блат

Can you explain that?

блат (blaht) means "nepotism", but used in a much more widespread way, to refer not just to jobs obtained through relations and connections, but also buying things you normally can't get in stores, getting invitations to selective summer caps for kids, etc. etc.

In the Soviet Union, things were typically inexpensive but unobtainable. So for example if you wanted a fully automatic washing machine that didn't break down after a few months, the problem wasn't that you couldn't afford it, the problem was that too few of them were imported or produced compared to the number of people who wanted them. In theory, these fancy washing machines went straight from producer/importer to distribution centers and then to regular stores to be bought by regular people. In practice, every chain in the distribution link was highly corrupt, and desirable items never reached regular stores (or, in some cases they did, but the knowledge of when exactly and in what store they'll be giving out something "good" was by itself desirable). Everybody worked intensely at cultivating connections with the right kind of people who were gatekeepers in some distribution chains, or officials in charge of those chains. Now repeat that for every kind of expensive item: furniture, cars, good clothes, shoes, some kinds of high-quality food (from caviar down to sausage), getting your kids into a good school, summer camps, vacation packages, a work trip abroad... all could be and were obtained through nepotism.

>This is a fair point, but in many cases you need to be rich, hard working, and smart.

Like Paris Hilton, Kim Kardasian, etc.? No, rich is still the only requirement. Those other things just give you more options.

How rich?

The only requirement for what? Financial success? Celebrity?

Seems your example only applies to a very narrow demographic: wealthy socialites. Does it apply to doctors, lawyers and software engineers? How often does it apply and how often does it not?

I think these nuances are important to answer before generalizing.

> >This is a fair point, but in many cases you need to be rich, hard working, and smart.

> Like Paris Hilton, Kim Kardasian, etc.? No, rich is still the only requirement. Those other things just give you more options.

For every person you cite as being wildly successful only because of the money they were born into, you could also cite people who came from nothing and became multi-millionaires. Or people who were born rich and didn't succeed.

You're not wrong - there are people who only needed to be rich to begin with. But giving examples like you have doesn't counter the above point.

If you have 10 rich people with a 50% shot of making it : 99,990 people with a 0.005% shot of making it then the number of people with both backgrounds who made it is reasonably balanced. But, in no way is this a meritocracy.

There are probably 10,000 who only needed to be rich for every self made multi-milli)onaire who came from nothing.

I'd lump beautiful women and gifted athletes into the "smart" category. While not necessarily the same in practice the effect of being either is basically the same as being incredibly smart except there's more luck involved. Neither are common enough to get their own category IMO

Being gifted is being gifted, yes, the area is important but secondary. And there is no more luck involved. It's probably the same mechanism, too (low mutational load).

I'm not saying those women aren't "smart", I don't know them personally. I'm saying it doesn't matter if they are or not.

The problem with this is how class affects the WILL to be good at school, SAT, college and so on.

If you're high up in the middle class, you have a high-paying job. That means one of three things:

1) medical degree

2) high-end engineer

3) varia (like climbed-up MBAs) (but this is a small group)

Now when it comes to groups in society, the picture looks very different:

Largest groups are non-working people, and workers and tradesmen (from factory workers and construction workers to truck drivers) (total of these groups is like 90% of America)

Both of these groups are famous for disrespecting education, telling anyone who'll listen how college and university degrees are worthless (and for their positions in the world, I'm sure that's true).

And ... surprise ! Their kids mostly don't care about education, and don't score well on SATs (except the ones that figure out they really DO want to in time). Could this be ... because kids believe their parents ?

The "problem" is the entire feedback loop running through society, which mostly is the people itself, not that "rich" people tell their kids to achieve things. You cannot fix this without separating kids from their parents somehow, and you don't want to do that.

The only thing you can do is to prevent the middle class from achieving anything, or anything worthwhile. Which is exactly what is mostly happening in these diversity efforts.

Even that is disregarding the fact that there are already too many college and university graduates to make sure such a degree is definitely worth it. This is a problem akin to "let's make the lowest 50% of the population better off than average".

I think this depends on what you frame what an opportunity is. Giving everyone the opportunity to try and meet a certain criteria in order for you to accept them into whatever job/institution you represent is a binary opportunity in theory. They'll either meet the criteria or they won't. If you frame being able to cultivate the skills to meet your criteria as being an opportunity in itself then that negates your offering as being an 'opportunity'.

Loved this article. It presents a compelling case that most colleges are providing near-equivalent education, and that the variance is mostly in selection bias. My only objection: at the end, the author takes issue with the fact that it's nearly impossible to get people to take selection bias into account when choosing where to send their kids. However, there's another possible factor that I think parents might be considering: the network of connections that I build when I go to school X. Even if Harvard gives my daughter the same educational quality that, say, Florida State gives her... if she's likely to form a network of friends that are all approximately 100x higher in net worth, then if I care about her long-term economic outcomes, I'd still try hard to send her to Harvard, no?

I'm not saying that's ideal, and it's most certainly an outcome of the exact selection bias you're talking about... just that it's a semi-rational choice for the parents, because school choices can often be about non-educational effects.

Just my two cents, though... I could be way off. And regardless, thanks for the article.. definitely made me think about the topic more than I had.

I think the network is overrated. Everyone I've known has had the same experience - immediately after graduating, everyone gets a job, moves far away from one another, and slowly drifts apart from the friends they met in college.

You'll get a similar network working at your first job. And then your second job. The group you met in college is of little consequence in the end.

Speak for yourself. My current career in systems administration, a pleasant improvement over my retail start during college, was gained through my network of college friends. I don't keep in touch with them often, but we went to the same programming classes, and know each other's relative strengths. One of my buddies took me to lunch with his friends that work here, and one thing led to another. I still had to apply and go through the technical screening just like anyone else, but just like that I had an "in" and was able to perform a class jump.

Is my experience anecdotal? Absolutely! I didn't even finish my degree, which makes this a weird exception story. On paper, college got me nothing, but in reality, the network of contacts and the life experiences I earned were far more useful to me than the degree I never obtained.

To give an opposing anecdote, I've landed more than half of my full time permanent positions through personal recommendations or references from people I know from university. And these jobs have by far been more interesting and/or better for my career than ones I found through recruiters or vacancy ads.

So, recommendations and references from people who never saw you at work. There's something wrong about that. I am not saying it doesn't happen, it does, but it still makes very little sense.

I generally disagree that the university network is unimportant.

"85% of All Jobs are Filled Via Networking"[1]; the network in most cases (certainly mine) will consist of people they met at or through university.

In this case, I would argue that the "through" university section is highly important. If you go to uni, land a couple of internships at large companies or go on exchanges you probably have a very large network of people you otherwise wouldn't have.

I do, however, agree that it takes effort (on both sides) to maintain the friendships and connections made at university.

[1] https://www.linkedin.com/pulse/new-survey-reveals-85-all-job...

One's choice of first job is of great consequence, however, and it is greatly influenced by social group.

Not always True. I'm in Boston and the Harvard alumni network is strong. You see connections all the time throughout various events and industries. For example, I'll meet someone in biotech (our state's largest industry) and talk about venture funding for his startup when he'll mention his friends from Harvard that can help get the right introductions.

Not to question the authenticity of your comment but it feels handcrafted to piss off about 75% of HN (not me, necessarily!)

What is offensive about this? The fact that someone on HN went to Harvard?

Envy I'd guess? There are probably a lot of reasons that some people resent Ivy League grads, probably the whole "legacy" model. I get that, but I know plenty that were just talented and worked very hard to be there. Haters gonna hate!

Just an odd take to me. HN was started by a Harvard alum based on experiences in the Harvard Computer Society, so this should be the last place that's an issue...

I feel like a lot of people think of the worst type of legacy freeloader when they hear 'Harvard Grad', not that it's right.

I think that the main added value of prestigious universities is networking - with peers and professors alike (almost in its naked form for many business jobs).

If Mark Zuckerberg had attended some university in Slovakia (being as smart and skilled) there is no way he would have started Facebook (and it goes for every networking effect - from co-workers, through financing to reputation of place that allowed this network to grow).

Parents, teachers, and students are so focused on "education" as a pedigree and job pipeline that if they don't see an immediate connection between a topic and a paycheck they're like "Who cares?! Why are you wasting my kid's time on that?"

When the whole thing is an exercise in competitive pedigreeing, of course it's going to be gamed. If it were more about human development, we'd be focusing on the deltas instead, and they'd be harder to fudge.

As a student, I agree with many of the words you said but for different reasons. I've had a middle school teacher who told us about her own sex life during class. I've had a high school teacher who droned on and on about her divorce (to the point that everyone learned so little, almost no one passed the AP exams she was supposed to prepare us for) and put down religions - again, during class. College isn't a whole lot better, and elementary probably had similar stories that I can't remember.

We (students) all hated those teachers who wasted our time with things that wouldn't help us get further in the job pipeline. I'd like a little more gamification, because I'm in college and still dealing with too many profs who teach less and less about what people need to do their jobs and more and more stuff no one asked for.

There doesn't have to be an immediate connection between what a teacher says and the subject they are supposed to teach. Many fantastic teachers have inspired me and made me who I am by giving little pieces of unrelated advice. But the advice of a teacher is really, well, just the advice of some teacher. They aren't gods. Plenty of them give really bad advice. I just care about students learning to learn and developing critical thinking. Students can read literature for opinions on things unrelated to school subjects, and ponder on their own. They can watch movies, listen to songs, google things, or even talk to other students. But I don't think teachers should go even further out of class discussions than they already do.

Having been a student with good and bad teachers, I can see your point.

However, many of the things that you are being taught (at least I hope this is still the case) that don't appear to be needed for the "job" are actually important because they are needed for the understanding of the background for why and how the "job". The failure of teachers that we faced as students was that the reasons for learning something were often not explained and only became obvious when learning about other subjects in later years. Having the relevance of the knowledge explained to us would have made it easier for us to learn.

One of my favourite examples was the subject of Karnaugh maps (or K-maps) which is used in logic circuit minimisation for small circuits. We learned all about and didn't use it again for the rest of my engineering undergraduate days.

However, a number of years working with a colleague on trying to solve a particularly pernicious software problem, I asked him why he hadn't used the concept of "don't cares" in short-circuiting the logic mess that was required to solve the software dilemma we were facing. He had never heard of the concept as he had never been taught about K-maps in his undergraduate programming course.

One he understood the concept and it didn't take him long, he was able to write 5 lines that solved the problem, instead of the hundred's that he had already had written (with many more to follow) to test for the problem throughout the program.

I have been surprised at how often over the years, where theoretical knowledge or other knowledge has come to the rescue in finding a solution for specific real-world problems, even though it has (on the surface) no relevance to the job at hand.

I have worked with some very smart people over the years (much smarter than me) who have been stuck for a solution because they didn't have that "irrelevant" piece of knowledge from their undergraduate days, but I did and so a problem was solved.

The one piece of advice I will give you is to learn as widely as you can so that you basic knowledge base will be broad enough for you to keep adding to as the years go by.

Likewise, as an EE I studied differential equations, La Place transforms, z-transforms, s-transforms, the fourier transform, and the dft transform. I'm a digital logic designer and have never had to directly do any of that stuff, but understanding the duality of time domain and frequency domain has been invaluable to me over the years.

Theoretical knowledge is absolutely extremely useful, but the way we teach it is pretty much a non-starter.

We need to humanize our education systems. The concept that we all sit around in school for 20+ years and then go out and find something to do with all of that knowledge is based more on maintaining rigid financial/power structures than it is based on effective information discovery and utilization, and there's no reason that a useful piece of information recovered partially-intact from the caverns of memory here or there should continue to justify such a lumbering, monolithic system.

There are no guarantees that the system will provide even what you've enjoyed; as you said, a peer with similar education had either not been taught or had not remembered the needed concept.

Humans need it "on-demand". We learn and master a trade experimentally, most naturally in the context of an apprenticeship. We can't compartmentalize book learning and practical employment, putting years-long gaps between them, and expect the eviction algorithm in peoples' brains to have not pruned much of the relevant information. We need it cemented with immediate, meaningful, recurring experience, and we need systems that will make it easy for us to remember and reference as necessary.

The existing educational apparatus is none of this. It is just a cash cow for the banks and universities, enslaving most of its "graduates" to decades of debt servicing. That, IMO, is the primary reason it continues to be enforced, because there is very little that is rational about it.

It's also childcare for working parents. I know it's not supposed to be but it's a side effect that's widely taken for granted.

I believe you're correct about young people being turned into debt slaves. The rich and rich-aspirational in the UK are encouraging/implementing this here as quickly as they are able.

I definitely agree that learning things that seem irrelevant to your job pipeline can be good, just like your K-maps example. But I also think there are plenty of seemingly irrelevant things that end up not helping you at all, ever.

That's why I don't think teachers ought to be in charge of teaching those extra things like K-maps, but rather, students should be asked to study things on their own (both inside and outside their comfort zone), in hopes that students will "learn to learn" and be able to teach themselves.

You're right, this method would not have given you K-maps in school, but it also would have wasted less of your time in school, and empowered you to learn about things on your own (such as don't cares in K-maps, or even just the general idea that you can assign whatever value you like to outputs you never expect to see).

Like you said, most people know who's done them a service and who hasn't. If only that was what they were selling!

Just pretend to care while they're rattling on about their kids and their sex lives, meticulously regurgitate their opinions--verbatim--on the test, and watch the circle of life remain unbroken as one generation of over-credentialed under-performers passes the torch to the next.

I mean, hell, it's only four years of your life. Who's going to miss that?


They're not wrong to worry about that, it's a rough world out there and getting a good job is vastly more important to most people than "human development". Parents don't send their kids to college to become better people, they do it to make sure they have a skill that can lead to a career so they can survive on their own. Only the rich have the luxury of being concerned about something other than a job.

On the contrary. The less you have, the more you stand to benefit from appreciating other things than possessions.

Do you really need a certificate from the Board of Regents to "appreciate things other than possessions", especially when said certificate incurs an immediate cost of tens of thousands of dollars, which most students have to finance and service for decades (thereby depriving them of at least the time, if not the financial freedom, to truly enjoy their appreciation of non-possessions)? Come off it.

No one said anything about possessions, you're projecting. We're talking about survival here, not consumerism.

That's probably a little over-dramatic. In the west, survival is a pretty low bar.

That, and "human development" in terms of an actual broad education will probably get you further in the long run than circling up around the latest job credential.

I remember reading somewhere about how learning certain instruments makes changes to the brain's surface that are large enough to see with the naked eye. From many accounts, playing the violin was no small part of Einstein's life and research. Who knows if he'd have been as productive if he'd just been all-physics-all-the-time? It could have served as a sort of "backpropagation" to his theoretical work.

From my own experience, many of these seemingly unrelated intellectual pursuits open so many doors, and from this article it isn't just my own experience: https://medium.com/personal-growth/einsteins-violin-the-hidd...

> In the west, survival is a pretty low bar.

You've never been poor I see, or really truly known anyone who was. Poverty is a real problem, here, in the good old USA still. Survival is all many people worry about. I'm relaying a reality to you that you clearly have no concept of. I didn't have indoor plumbing until I was a teenager. We lived off government handouts and what we could kill and bathed outdoors in rainwater we had to strain the misquote larvae out of with a towel. Don't tell me I'm being overly dramatic, you don't know what you're talking about.

Yea, music is good, but when you're worried about how you're going to eat today you don't think about such things, you don't plan long term, you live day by day.

My family was on public assistance when I was younger, and I went to public schools where over half the students families were on some form of public assistance. None of us had to bathe in stagnant water...

Then count yourself lucky but don't presume to know what it is to be actually poor nor pretend it doesn't exist in the west; it does.

I presumed nothing about being "actually poor". I wrote that "survival is a pretty low bar" in the west.

Despite whatever hardships you have endured, here you are.

That can be obtained in a much more cost effective manner.

Yes, I'm sure a generation trying to save for a house under burden of student loans appreciates your sage bullshit advice.

Must really appreciate the lack of affordable health care and never being able to afford a family. But at least you can appreciate it!

Philosophies of life like Buddhism and Stoicism came about in times that were much less comfy than even the sad reality we have to endure. I am not saying people shouldn't demand things of society (quite the opposite, I think the level of organizing among the poor is the root of many of the issues we are seeing today), but I am saying that they can benefit from separating their happiness from the fulfillment of those demands.

at least you can ponder the mysteries of existence while working your two mcjobs to pay rent.

But the jobs to come have much less to do with your education as they will more and be be about talent and uniqueness, the exact opposite of what education does to people.

There is a world of difference between being educated and getting an education.

> Parents don't send their kids to college to become better people, they do it to make sure they have a skill that can lead to a career so they can survive on their own.

That's correct, yes, but it's not the whole story.

(Some of) us students hold the same opinions, regardless of whether or not our parents do.

Indeed, I would gladly shed any of the "good person" parts of my personality, if doing so gave me better aptitude in the field that I'm studying.

(Sadly, genetic engineering is still a pipe dream, and won't work on already-living humans anyway.)

They are wrong to worry about that. The world is dynamic, and human development, the way I interpret it, will help the students deal with the world they go out to.

Better people will deal with what comes their way. Maybe today it's a job, but tomorrow it'll be something else.

Students who were only thought how to build software, will have a hard time dealing with life when robots starts building software instead of them.

First have better people, then let better people be better and bring themselves to prosperity.

If "being better" means better communication, better critical thinking, better learning skills, better time management skills, better understanding of oneself — yes, please! These are the universal skills important most everywhere.

These are also skills that are hard to teach as a specific subject; they mostly come with experience in some other, more practical subject. Some of these, like critical thinking, can go against the modern prevailing religion (political correctness), which makes teaching them harder still.

Again, all easy things to worry about when you're not trying to just survive as most people are. Most people are worried about far more practical things that being "better people".

I think the real issue here is that our schools are somewhere between liberal arts and vocational, so they do a pretty shitty job at both

some people get to work in a creative job (I think being successful in STEM is more creative than people generally account for). in which case all that human development is pretty important.

> If it were more about human development, we'd be focusing on the deltas instead, and they'd be harder to fudge.

If you were scored based on delta, gaming it would be laughably easy. Just completely fail your first exam, and then steadily "improve" to your real skill level. That's basically incentivizing students to do what Intel does with processors :)

Your thinking here is still in job-credential-land. If the goal is still some kind of 3rd-party pedigree, that is going to be infinitely gameable no matter how you measure it--but that wasn't what I argued!

I'm not talking about getting graded USDA Prime so I can fetch the best price in the meat case. I'm talking about personal development. When you go to a personal trainer, you're not going to fudge your delta because you can't fudge your delta--the degree is your body, and everyone will know if you passed or failed.

But if you want personal development then gaming isn't an issue (why lie to yourself?), and even if you do - changing the metric isn't going to stop you.

And grading in absolute values is still more useful to asses your performance than deltas anyway: "you can lift X kg today" is more useful than "you can lift 2 kg more today"?

If we're trying to assess the training instead of the trainee, which is what the article was touching on, I wouldn't be so sure. You've got to take the long game into account.

If we keep with the lifting analogy, I know a lot of guys who could lift some serious weights in high school--but with terrible form. Ten years later they're all messed up with joint and muscle problems due to bad training. They've dropped out of the fitness lifestyle, and aren't about to recommend their HS football coach for anything.

But then you look at guys like Arnold or Scooby who've maintained a high level of fitness throughout their lives without a debilitating injury, people tend to take them more seriously. Of course, then we're talking more about integrals--which would be even harder to fudge than deltas.

I think the worst part of the whole sorry situation is that our modern economy is becoming so detached from reality that the credential has more value than actual performance. I've seen so many nutritionists who are fat and tired and will probably not live that long, but they bought a certification from state U, so now they have a job and people listen to them. Or hairdressers! So many can't even give a decent haircut or keep their tools clean, but they got their state cert, so they can go out and charge for haircuts while mom can't. Or even people with 4.0 GPAs and a masters in CS. I can't tell you how many of those I've had to let go because a high schooler ran circles around them.

Computing is one of the few fields, like music or acting, where you can still just go out there, perform, and get paid for it. I wonder how long that will remain the case...

This is a good article. I wish I could force everyone who pontificates on schooling in Australia to read it.

We have an enormous private school system (part govt-funded, which is gross) that can get rid of students as they please. Then they go on to verbally dump shit on the public system, where they get to dump their problem students. Nice.

We also have a real fetish for selective schools, which are essentially selection bias driven to a huge extent. Any suggestion that this isn't a good idea is met by an chorus of "you must hate smart people".

The private school system in Australia saves the government money. Per student funding in the private system is lower than per student funding in the public system. This money is then available to spend more on government schools.

It also keeps the government system honest because people can and do leave schools that don't work such as many of those in poor areas in the outer suburbs of the big cities and regional areas. If there were no private system the only way to get into better schools would be to own very expensive houses.

Meanwhile, the selective schools show that the government can provide a good education for highly able students as well. In NSW ~18 of the top 20 high schools are government schools.

Personally I think most people are unwise to send their kids to expensive private schools and they would help their kids much more if they put the money toward investments. But they are free to waste, in my mind, their money to reduce the cost to the tax payer of education.

> The private school system in Australia saves the government money. Per student funding in the private system is lower than per student funding in the public system. This money is then available to spend more on government schools.

Isn't this selection bias again? More difficult (expensive) students may get shunted into the public system.

A number of private schools roughly kick out people in the bottom 10-20% of results. They aren't necessarily that more expensive to teach, the private schools just don't want them.

The really expensive kids (say 20K+ per year) are those that are seriously disabled and they almost all go into the government system. But, if the government only has to pay ~1/3 of the cost of about 1/3 of students who are in the private system there is more money available for them.

There is a weird situation in Texas in the USA. The state is limiting the amount of funding for each local district to 8.5 percent for special education with the end result that families end up going to private schools to get special education. The one family in this school ended up paying 50,000 per year for private school for two kids with a readi ng disability. http://www.houstonchronicle.com/denied/7/

This may be true that the private system as a whole saves the government money. However, I'm not sure that the last dollar of subsidies spent on the private system saves the government any money at all - i.e. would someone switch to a public school if that dollar was taken away, or would the private schools just move the cost to the consumer who would bear it, or perhaps they might cut back on the indoor shooting range or second Olympic pool (in the case that the dollar is going to an elite private school)?

The argument about that last dollar obviously extends backward quite a way before becoming hugely disruptive or that big a deal. Obviously cutting all subsidies to the private system overnight would be an enormous disruption, but it's far from clear to me that the policy of subsidizing private schools at the current level saves the government any money at all.

As for keeping the government system honest - I think this works both ways. I think having people exercising "feet" rather than "voice" harms the public system by taking out people from participation. The argument is similar about selective public schools (I went to one). I see a lot of flaws in the public system too - tons! - and any strong movement back towards public education in Australia would have to be accompanied by some major changes in how they are run (IMO way too much for the convenience of mediocre teachers and even-more-mediocre bureaucrats).

That's almost certainly true about the marginal value of the last bit of the subsidy and the crazy games around the formula for school funding. A simple, flat formula might be better.

Having had friends who were in bad schools but who had parents who did care it's not clear that the public school system does listen much to people when they have an almost captive audience. Having had a parent and friends who are teachers in the public school system doesn't inspire much faith that where they are broken they are easy to improve either.

The current system with all it's flaws but with distributed control and choice that saves the government money has some advantages.

>Per student funding in the private system is lower than per student funding in the public system.

If you dump all of the special needs kids and the kids that require a ton of effort to learn into a single group then OF COURSE these kids will be more expensive to teach per student than the most motivated students who can afford outside tutors etc.

The whole premise of the article is that selection bias is a big deal.

Very similar in the US, actually, although maybe not with quite as large of a private system (although some would like to change this). Very similar problems. I've had friends who work in the US school system who complain about this same issue--that private schools are immune to many problems because they can just pass the buck onto someone else.

The linked piece is very nice, and illustrates the selection bias problem extremely well.

One additional comment I might have is about variance. That is, in the figure with red and blue plots, those means are hiding within-school variance. In data I've seen, what often happens is that the lower-status schools often have more variance in them. So in some ways the problem with selection bias is maybe even worse than what the author suggests, because people not only tend to attribute to schools what was pre-existing, but they also tend to stereotype. E.g., you went to a school on the left side of that figure, so you're lower in ability and/or learning than on the right, even though you might be at the upper end of the distribution of the whole figure. This pertains to vocation too: people tend to assume certain vocations are signals of lower cognitive ability than others, when the reality is that uncertainly only increases for some.

The other thing I might mention is some social benefits to attending selective schools. The author at times seems to imply that "talented people will end up doing well regardless of where they go" somewhat dismissively. I think this was probably unintentional, but I think that's not exactly true in the way that people care about, because the studies that are done are kind of coarse. That is, if you study outcomes, they tend to be things like general income, or terminal degree attained (undergrad, master's, doctoral), or whatever. In that sense, things are fine. But because of social connections, people might not get into the particular job they want, whereas another person is given a free pass, or might be at a state-level position rather than a federal-level position, or whatever. That is, there are real consequences to overlooking selection bias, that have real importance for people, but tend to get fuzzed over in studies because it's difficult to study.

What's the problem with selective schools? I think mostly everyone is aware that selective schools are successful because of selection bias.

I went to Melbourne High in Victoria (all boys academic selective), and think my peers and I benefitted from leaving our original (mostly public) schools to go there.

Do you think the education system overall suffers from the 'extracting' of these students out of their original schools?

I went to North Sydney Boys High (all boys academic selective, like you). I think we benefited. I think the schools we were taken out of did not. Back in the era of narrower catchments for selective schools, it was more like losing the top 10-20% for a bunch of nearby schools rather than skimming off a few people from every school in Sydney.

As I read this article, it didn't suggest that selective schools are bad or improper, but that they may greatly mislead us about causality or what particular institutions and methods are or aren't accomplishing.

Having been through the U.S. public school system (plus an IB diploma; too little, too late) ... your description of the Australian system makes me very much regret that I was not born+raised in Australia, as I am probably dumber for it.

First, let me ask you a question - Are you a teacher in any Australian school?

Secondly, you have just pontificated on schooling in Australia - so, my second question is, if an article came out with different conclusions (with which you disagreed) would you be amenable to being force to read it as you are in the class of people who pontificates on schooling in Australia.

There are good private schools and there are good public schools. There are bad private schools and there are bad public schools. What makes them good or bad is a function of the school administration, teachers and families involved in those schools.

All are interrelated and how they handle "problem" students is very dependent on the three groups. I have seen both private and public schools that do not handle such matters well and the result is that everything (in time) goes to hell in a handcart.

But I have also seen schools completely turned around by a change of administration, change of teachers and a more involved family community. This has affected both public and private schools.

You negativity toward private schools (comment about partial government funding) doesn't seem to recognise that if private schools closed down the public school system would be so overwhelmed that it would no longer have any hope of coping.

My own children have been at both public and private schools. I have even been on the school committee for the public school so I got a direct management insight into the operation of the school and the processes required for dealing with "problem" children. At times, it was more problems with parents than children, at other times it was problems in the families which could be mitigated by giving them appropriate help and at other times it was just related to recalcitrant and stubborn children that their own families had difficulty with and the families were supporting the school actions in relation to their children and finally there were problems with the teachers themselves and their interactions with specific children.

If there is no cooperation between schools and the families involved in wanting to develop the children to be their best, then failures will increasingly occur.

Your comment about selective schools misses one salient point, children are not all the same. In Victoria, we used to have trade oriented schools, which were done away with. We now have little opportunity to have students who are interested in practical fields to be specifically educated in those areas. The education model is to have all students head for higher education and big debts.

I have a grandson who is autistic and is being educated in the private school sector where there is facilities for him to progress. The local public schools don't have those facilities and hence he would be on the scrapheap. However, there are public schools around who only take the "best" students and pride themselves on having great education standards. If you don't qualify, you'll have problems getting in.

What the solutions are is contentious and many just blame one sector or another for the problems (private/public) without thinking about the broader community aspects and the benefit to students. Teaching is not a highly regarded profession and that in itself brings about structural problems of all sorts. We then have various political interferences that want to control the specific social education across all schools, irrespective of any school community feedback.

I don’t think it’s negative to private schools to say the statistics reflect selection bias rather than apples-to-apples comparisons. I generally take people saying ‘I wish I could force people to read X’ as ‘I’m frustrated I always have to deal with people who don’t understand important concept from X’ and not ‘I actually want to hold a gun to someone’s head until they finish reading this.’ I agree on good and bad public and private schools, but I think the key point is that they aren’t the order of the schools ranking on some standardized test or statistical measure.

The negative bias was the apparent opposition to some government funding. Whether government funding is appropriate is a matter for discussion, but for another day.

When I see or hear people saying "I wish I could force people to do blah blah blah", it has always been an indicator to mean that they only regard their own opinion as having any worth. If on the other hand, they say something like "this is well worth reading for the following reasons, blah blah blah", then its an indicator to me that there is a viewpoint that can be looked even if you don't agree.

As I do on a daily basis, I may encourage people to look at specific things and I can argue with the best/worst of them. But if someone has no inclination or desire (for whatever reason) to look at specific subjects that is their choice. I'll state my piece and if they are not interested then that's where it ends.

Ranking of schools has been attempted before and as far as I can tell, it hasn't really amounted to any valid standard test or measure. When I was in senior secondary school (many decades ago now), our deputy principal was heavily involved in the development of a standardised testing regime for all schools (private and public) in the state. As students, we had the opportunity to pin him to the wall and find out how it was supposed to operate. The upshot was that the system could be gamed in both positive and negative ways and that its sole purpose was to provide a measure by which students in the state could be culled from the higher education market. The comparison he gave was to compare what was happening in the USA at that time and how to prevent the worst aspects of the USA education from ever occurring here. In some senses that has been successful, but only in some areas.

Your remarks seem like a collection of things you'd like to say, a bit of abuse ("pontificated", reiterates the guy who replied to my 5-liner with a rambling mini-essay about a large array of random education issues not connected to my post), and an appeal to authority (I'm not a teacher -so what?)

I am not proposing the private schools shut down. I would like to see them funded less (and suspect that we would continue to have a private school sector if they were not funded at all). I would like people to stop attributing the success of privates/selectives to the fact that they can admit who they like and kick out their problems.

I am not appealing to "authority" but to experience. How much do you know about the working of either private or public schools. Public schools are not free - parents pay all sorts of "education" fees, been there done that. Similarly in the private schools system, there are possibilities of getting financial help for school fees if you me certain criteria, been there done that.

A discussion also includes sharing of "random" stuff since this is generally experience and gives some input into various matters.

In regards to your very last comment about kicking out their problems is a feature of both public and private schools. Both can do it and both do do it.

Sending a troubled kid to a great private school probably will cause the kid to do better: because all the other students are doing well the bad students peers are working hard so the bad student sees that example and is likely to succumb to peer pressure and do better.

For this to work the troubled kid needs to be kept out of the gangs and whatever else outside of school will lead to bad examples. However selection bias already does this: parents who care enough to send their kid to a private school are involved enough with their kid that they would probably do this anyway. In short these are kids that might have been troubled, but they still would have been at the top of the troubled group.

You can almost postulate a sort of "memetic herd-immunity" where if a large enough proportion of the student population is inoculated against bad study habits, a few stragglers will be ok. But if the balance tips too far, the stragglers start joining up and reinforcing each others' bad study habits.

> Sending a troubled kid to a great private school probably will cause the kid to do better ... However selection bias already does this

Funny thing, though. We've done a bunch of studies on this and it doesn't actually help. The people who care about their kids more getting them into better schools, yes - but placing the kids of parents who don't care into better schools, no.

This has been shown time and time again. The best results to the contrary managed this by ... drumroll ... designing selection bias into the study.

> Funny thing, though. We've done a bunch of studies on this and it doesn't actually help.

> This has been shown time and time again.

One would think a link to at least one study would be appropriate here?

The original article already provided evidence, based on analysis of the performance of similar students who do and do not get into a selective school. As it says...

"Except that attending those high schools simply doesn’t matter in terms of conventional educational outcomes. When you look at the edge cases – when you restrict your analysis to those students who are among the last let into such schools and those who are among the last left out – you find no statistically meaningful differences between them."

So unless you're calling out the original article, how about offering counter-evidence?

Here you go: http://pricetheory.uchicago.edu/levitt/Papers/CullenJacobLev...

The upshot, attempting to get into a better school is correlated with doing better. Actually getting to go to the school of your choice is not.

Eh, not really. I've learned over the years to avoid providing sources because more often than not the person won't believe you either way and will only use the source as a tangible target for disingenuous attack through motivated misinterpretation. (https://en.wikipedia.org/wiki/Motivated_reasoning)

I typically provide sources in response to replies like: "That's interesting, but I can't really find anything on the matter; can you point me to something?" But if there is no indication that the person actually tried to find corroboration independently I take it as a sign that he is not actually interested in corroboration. I'm sure this is inaccurate sometimes, but it's the best approximation I can come up with.

I went to five high schools growing up and this is not what I experienced. I personally and through others found the exact opposite to be true. Yes you can send students(whether bad or good) to both bad both public and private schools but sending bad students to good public or private schools doesn't make them a good student.

I also doubt that it's meaningfully true. I think the troubled kid would do better in a private school because there are better resources available, and the teachers would not be overburdened with large class sizes filled with similarly troubled students like in a public school.

I worked as a reading tutor at an elementary school for a year in the US. It was a terrible, terrible job because of how strictly prescribed the methodology was. We tested every kid in the school, and the test was to time the students for a minute and count how many words of a text they could read (an incredibly dumb metric). We did this twice a year and then tutored the kids that were at a slightly below average reading level, meaning that I was not even seeing the particularly troubled / low-performing kids. I had about 25 students that year and got to know a lot of them. Some of the kids didn't need my help at all, they were extremely smart but had read slowly and carefully on the test and so ended up embarrassed by having to be pulled out of regular class.

But the kids that were actually struggling to read at grade level almost all had bad situations at home. There was a boy who had to move in with his aunt mid-year because someone shot up their house while looking for the kid's older brother, a girl who had four other siblings and was frequently hungry because of her father's gambling and drinking, a kid who changed schools because his father was arrested for beating the kid's little sister, a recent immigrant girl who had 15+ siblings and step siblings that her mother somehow ended up taking care of, a very strange boy with a morbidly obese mother and no father, and probably some more I'm forgetting about. My co-worker had a student whose parents were both serving life sentences for murder. This was not even in a particularly rough city or a particularly rough school (the school is maybe 60% Hmong, so many of the kids needed ESL much more than "how many words can you read in a minute?"). The point being, if you don't change home situations like that then there's not much hope for a student's education at either a public or private school.

I won't even go into the daily prescribed routine of exactly how I was supposed to tutor the kids, and the constant evaluations to make sure I was, for example, counting down the timer with the correct fingers and correct hand. That seriously is / was an official Reading Corps policy and it was among many other inane policies in a broader system that was also deeply misguided.

In a country with actual social services, there should be a feedback channel to identify such cases and help them directly (not necessarily funding but sometimes too).

Of course this won't work in some countries because both socialism is frowned upon and individualism is glorified. You fall because you're a failure not because you were dealt a bad hand by life.

> because all the other students are doing well the bad students peers are working hard so the bad student sees that example and is likely to succumb to peer pressure and do better

What about the non-troubled kids? How much of a negative effect can a few troubled kids have on non-troubled kids?

One kid acting out in a class can ruin the learning atmosphere for the entire class.

I think the real problem described is that people are judging schools based on the performance of their students. There are other objective factors that won't be as affected by selection bias such as: breadth of extra-curricular options in sports and in the arts. Number of AP classes offered. Teacher to student ratio. Budget per student. Does the school have newer computers, working instruments, modern facilities?

Are you saying that people should judge schools based on the inputs that go into the schools rather than on any effects the schools produce in their students?

The article points out the high degree of selection bias with schools and how that can make it very difficult to make apples-to-apples comparisons between schools.

I was just trying to mention ways to compare schools that won't be affected by the selection bias described in the article.

A proper analysis will attempt to identify such factors and how important they are. Since the sample size is big enough it is not impossible to determine how much of the effect is those other measurable factors.

Selection bias might be measured by the rejection rates on admission (esp. scores rejected)

Since I cannot edit it anymore, also the self selection bias can be measured (less adequately because questionnaires bring their own problems but still).

That tricky thing when dealing with humans: are they learning because of our efforts to teach them or in spite of them?

This has long been my assertion about elite undergraduate engineering schools. The teaching is usually just OK but their main advantage is that they get to "skim the cream" from the applicant pool.

I'm from the midwest and most people who weren't aiming for Ivy Leagues just took the ACT.

Yah. It's weird that de Boer argues selection bias is in play in SAT participation between MS (3%) and CT (88%) when 100% of MS high schoolers take the ACT. That seems to be what's happening here: the kids in MS taking the SAT in addition to the ACT are the ones aiming for elite schools.

>the kids in MS taking the SAT in addition to the ACT are the ones aiming for elite schools

Which is an extreme selection bias. The ACT should have been mentioned to provide more context, but the actual argument works without it.

You say it better than I do. Agreed it's an extreme selection bias, just want to point out that it's slightly different than the one de Boer seemed to be pointing out.

It's still the same bias he argues, the three percent are the most motivated and prepared. Why the other 97% skip it isn't relevant in this argument, but the number is independently shocking without the context. I fell for it until the ND mention, and I took the ACT.

That's... exactly what he said.

Yes, but without mentioning the ACT.

You seem to be taking issue with the fact that the piece could be interpreted as implying 97% of MS students didn’t want to go to college. Obviously, if that was his point, he was omitting the most important detail.

His point was that if you bias the data source, you get results that you shouldn’t project onto the general population. The specific explanation of the 97% disinterest in the SAT is well and truly irrelevant.

My issue was more the opposite. If the author is saying, hey, only 3% of kids take the SAT which creates selection bias, most people will immediately wonder why the low percentage. The ACT info adds support and explanation to his own argument, so it should have been included.

I'm from the East Coast and I took both, submitted both on college applications (I actually did slightly better on the ACT, not sure if it helped) but yeah SAT was given priority in school/training programs. I had to seek out the ACT.

Same here. I didn't even know it was an option (obviously didn't go to an Ivy League university)...

I’m from the Midwest and went to a majority black public high school.

In all seriousness, I had never even heard of the ACT until college. The SAT and AP exams were the only tests ever discussed with us.

I suspect there may be more variation in this than geography.

My school has a large Chinese international student body. The school is marketed as being diverse, but really those students are from wealthy families and can easily afford the 40k+ tuition. In the end, the campus isn't diverse, it is segregated. But at least the uni has more money for research.

majority of that mooney doesn't go to research, it goes to admin

"People involved with the private high schools liked to brag about the high scores their students scored on standardized tests – without bothering to mention that you had to score well on such a test to get into them in the first place."

Few private schools have entrance exams. Most only care if the tuition check doesn't bounce.

Education is about doing things at scale. Learning is about individual gain on delta. I believe better personal tools for learning could do a better job for individuals on average.

He was wearing my Harvard tie. Can you believe it? My Harvard tie. Like oh, sure he went to Harvard.


This is simply the best article I've ever read on this topic.

> People involved with the private high schools liked to brag about the high scores their students scored on standardized tests – without bothering to mention that you had to score well on such a test to get into them in the first place. This is, as I’ve said before, akin to having a height requirement for your school and then bragging about how tall your student body is.

Of course, that's a bad analogy, because average height isn't correlated with the amount of time teachers have to spend on OTHER kids, but average grades of the kids - is (inversely), so parents still prefer schools where other kids are as "smart" as possible, because that indirectly benefits their kids (more attention of teachers for their kid, higher expectations, peer pressure to perform, etc).

So - selection bias is one of the reasons kids do better, but that doesn't mean it doesn't also make teachers do better.

By the way, in my country (Poland) public universities are considered better, than the private ones (because of historical reasons, and the way free university education works). There's still selection bias (best students go to public universities and don't have to pay for education, so these universities have the best students and teachers, and the best results), but it's mostly unrelated to students' wealth (apart from extremes where kids don't have food or time to do homework in primary school, etc).

Do you still consider selection bias in education a bad thing, if it's only about results, not about your family, connections, wealth, etc.? It's easier to teach a group of people on similar skill level, than a group of people where a third is bored, a third is learning, and the rest don't get what's going on.

Selection bias makes it also easier to implement special measures. You know where the hot spots are and they are not diffuse.

(FWIW I live in Poland and had a short run with our private education system on both ends.)

What has to be fought is discrimination in employment based on the origin of degrees. All too common. There is enough of this based on connections already.

I've read the research about Stuyvesant, and several other articles with similar theme. The data and reasoning seems quite convincing. It's just I can't reconcile it with my personal experience.

All these studies seems to suggest it doesn't matter much where you go to school, everything is already decided. But I know I've learned things in school, things that I can't expect to learn quite systematically should I just teach myself. And I also know there are schools that don't teach much, where teachers are exhausted for just maintaining order and keep violence to an acceptable level. It seems pretty obvious that I wouldn't learn much in this kind of environment. Maybe I can still get to a similar level as I am today but it be would much much harder.

""" You believe that making your state’s high school graduates more competitive in college admissions is a key aspect of improving the economy of the state. """

Well there's your problem. The link between educational attainment of a country and productivity is weak at best. (Switzerland being a good example of low university attendance and outrageously good economy)

Ha Joon Chang was quite persuasive on this - and suggests that the imaginary governor of the article should look to fix his states economy with focusing on improving the eco system of the economy - longer term more patient capital, regulatory regiemes that force longer term investment and behaviour, picking winners and protecting nascent growth industries. All the bad things.

> People involved with the private high schools liked to brag about the high scores their students scored on standardized tests – without bothering to mention that you had to score well on such a test to get into them in the first place. This is, as I’ve said before, akin to having a height requirement for your school and then bragging about how tall your student body is.

Not quite — kids don't get shorter over time, but they can easily get worse grades/scores. It's true that having a screening test makes it more likely that your students will score well on other tests, but it's not a guarantee (as it is with the height example).

Yes quite; for example, kids who are in the 95th percentile of height at 9th grade might be only in the 60th percentile of height at 12th grade.

The example wasn't about height percentiles, which can change. it was about raw height. If his example had been about percentiles, then it wouldn't have been such a stark no-brainer. Because a high school that admits only taller students would be mostly girls, and by 12th grade they would be shorter than many/most boys.

I was taking issue with the example as given. If it were framed the way you suggested, it would not have had the rhetorical power that it did (at least at first glance).

It is assumed that schools bragging about how tall their students are are not doing so in a vacuum, and that they aren't bragging that their 12th graders are taller than the other school's 9th graders, no?

In education this is particularly bad. Students and parents also steer into selection bias under the assumption that peer groups are important.

But, to generalize the point into public discourse broadly: "Statistics are a problem."

A lot of public discourse revolve around quantitative arguments like this. Police shootings, women's 83 cents on the dollar, British Muslims' on Sharia, unemployment rates...

Even in the more intellectual branches of public discourse, bad arguments with a similar flavour can be the dominant ones.

I'm amazed by the range of completion rates in that first scatter plot. Here are the Australian figures, for contrast. Note that living in the NT is strongly correlated with being Indigenous. (Indigenous Australians are the people formerly known as Aborigines.)


I assume your amazement is because some states have such low rates. If so, this is very misleading since there are two competing tests in the US (ACT and SAT). The states that have low rates for SAT almost all have very high rates for ACT and vice-versa.

In fact, looking at the chart you can see that it is a split distribution.

A professor at the University of Rochester (my alma mater) has begun a project to better evaluate the value of a liberal arts education, and to try and quantify what impact it truly has on the career trajectories and future earnings of graduates[1].

[1] https://www.rochester.edu/pr/Review/V79N6/0304_lennie.html

The benefits of a higher education are more than just career path and earnings.

Perhaps they are, perhaps they aren't; I'm not a Doctor of Education, so I can't and won't claim that I know the answer to that question.

My question to you, is applicable regardless:

As an industry-destined BSCS student, in what ways does my future improve as a result of my education, except for career path and earnings? And if there are such (nontrivial) ways, then why should I care about them?

I can only speak from personal experience but I grew a great deal as a person while pursuing my B.S. I had experiences and met people I never would have otherwise. I very nearly stayed in my small rural town working retail and farming jobs but my education opened doors to new people and experiences. My worldview is much different now than it was before college and my life is much more fulfilling on a personal level.

While I was fortunate to select a degree that had positive economic benefits for me that is far from the only benefit. My B.S. was far from just a job training course. I studied a variety of topics and was exposed to ideas that I never considered before. I learned to think much more critically and to be more open to the world.

You should care about this because money and a job are not everything in life. I know that sounds cliche but I am finally reaching a point in my life where I understand that. My education helped me reach this point.

I really cannot say enough about how much my education means to me. Even if I did not leverage that education to start my career I would still consider it to be completely worth it.

> I very nearly stayed in my small rural town working retail and farming jobs but my education opened doors to new people and experiences.

So, that still falls under career path and earnings.

It doesn't, please read the rest of my post.

There is no evidence of this.

This is such an absurdly HN comment.

I would like to read a study that evaluate the value of education for the full society. I think we need an educated public because Democracy needs it.

A simple thought experiment.

Consider bussing kids from 'good' places to bad and from 'bad' to good. See after how long their outcomes become equivalent. Do so at different grade levels to measure the convergence time versus age of displacement.

And now for the actual experiment: how do you impactfully present such results? I assume no outcome. Just the existence of an outcome.

I posit that priceless data would be worthless in American society.

Such busing was court-mandated in the US for several decades. It is widely considered to have been a failure.

Amongst progressives, busing is considered one of the only effective mechanisms for reducing educational inequality.

That it only moderately alters test scores is also true, but that is far, far from being the same thing as a failure.

Busing is largely out of favor because it’s inconvenient for parents. Even I, who live in Berkeley precisely because it still buses-to-desegregate its students, was relieved when the school lottery meant my daughter would attend our nearest school.

That depends on what you classify as failure. It failed in the sense that the white and wealthy mostly fled to the suburbs and reversed any progress integration had made. It succeeded in the sense that in the places and times where schools were well integrated the racial achievement gap closed.

The racial achievement gap closed? That would be a huge story and I would be surprised if I missed it...unless you mean the gap closed within the school (i.e. not relative to society at large) because the remaining whites achievements were lower than the average before they fled, i.e. selection bias :)

Integrated schools help to reduce racial achievement gaps. In fact, the racial achievement gap in K–12 education closed more rapidly during the peak years of school desegregation in the 1970s and 1980s than it has overall in the decades that followed—when many desegregation policies were dismantled. More recently, black and Latino students had smaller achievement gaps with white students on the 2007 and 2009 NAEP when they were less likely to be stuck in high-poverty school environments. The gap in SAT scores between black and white students continues to be larger in segregated districts, and one study showed that change from complete segregation to complete integration in a district could reduce as much as one quarter of the current SAT score disparity.


Not only were they more successful in school, they were more successful in life as well. A 2011 study by the Berkeley public policy professor Rucker C. Johnson concludes that black youths who spent five years in desegregated schools have earned 25 percent more than those who never had that opportunity. Now in their 30s and 40s, they’re also healthier — the equivalent of being seven years younger.


Desegregation and Black Dropout Rates J, Guryan. 2001. has some good data on the decline in black dropout rates in districts with integration.

Integrated schools help to reduce racial achievement gaps. In fact, the racial achievement gap in K–12 education closed more rapidly during the peak years of school desegregation in the 1970s and 1980s than it has overall in the decades that followed—when many desegregation policies were dismantled.

Why I don't find that evidence convincing: https://news.ycombinator.com/item?id=13904403

Care to summarize? It's a wall of quotes.

Summary from the post: "My take is that if desegregation has any impact on achievement, it is small and drowned out by other factors."

Sorry for the late response. I haven't gone to the links yet but the quotes in italics give me pause. They seem to be at pains to state things in a way which looks like it means one thing but may mean another. The second sentence is a nice start to an investigation but is not evidence really, just a temporal association. The third sentence is about poverty, not integration.

The fact that the gap is larger in segregated districts is difficult to interpret without knowing if the comparison is to the population as a whole (i.e. blacks in the district to whites as a whole) or within the district (i.e. blacks in the district to whites in the district). I'd also want to know if how those districts compared to other districts. Were these high achieving schools? Low achieving? Somewhere in the middle?

I guess if I were to boil it down it'd come to this. If integration doesn't help minority students then you have to believe in seperate but equal. If you don't believe that our schools are seperate but equal then you should be in favor of integration so that everybody has ownership in the problems our schools and children face.

But I mean let's be honest here, are you really going to accept that integration worked? I'd like to believe it but experience tells me probably not. Because of the backfire effect I don't think you're likely to be convinced by some guy on the internet quoting social science research (especially since no research is perfect.) So here, let me give you a more human element. Here's three This American Life episodes, the first two are explicitly about integration and the third is about kids from a poor, public school visiting a rich, private school which is slightly less topical but helps to get the point across I think. Just listen and consider please.




To save people from litigating busing, there was a thread on this earlier this year: https://news.ycombinator.com/item?id=13902960 (I cited evidence saying that busing did not close the gap).

Do you have any pointers to data showing this close? Most of what I've seen shows the gap being shockingly persistent in the face of intervention across a wide variety of systems, but I'm always looking for more.

I reccomend Malcolm Gladwell's podcast on the subject. He makes a strong argument that the failure to integrate teachers to any degree caused a large part of the failure. Some of his sources are on the page, but I haven't reviewed them well personally.


Aren't you basically describing the desegregation bussing that the US has had since the 50s?


Somewhat. I am hypthesizing more than ethnicity being used.

Something like a magnet school?


>> Consider bussing kids from 'good' places to bad and from 'bad' to good.

there's different types of 'good' and 'bad'. for example, the whole IQ thing came into the light in the UK when poor Jewish kids from the ghetto (bad) were clearly outperforming their non-Jewish counterparts from more affluent (good) neighborhoods.

How many parents, who probably paid a lot of money and work two full-time jobs to live in the "good" place so their kids could go to the good school, are going to be OK with this experiment? Do you plan to force them, if there aren't enough volunteers to get statistically significant results?

San Francisco, some of America's most expensive real estate, doesn't tie school assignments to geography. The system hasn't been voted out yet.

30 percent of children in San Francisco attend private schools — the highest rate of private-school attendance in California, and the third-highest in the nation

All school is private. Sometimes you pay tuition, which seems better for accountability, and sometimes you pay more for housing. That people send their kids to school is unfortunate. It seems like if you took the daycare out of it, school would not be structured the way that it is at all. It is a source of daycare for children first, and an attempt at productively using that time second.

The thought experiment isn't doing the bussing (effectively impossible as others have noted). The thought experiment is how would you present the resulting data.

I'm not a fan of bussing. It is unfair to the good kids. "Bad" kids know how to deal with other "bad" kids. They grow up in that environment and know the rules of the game.

Good kids grow up in a different environment. The consequences of not knowing to handle oneself around "bad" kids are very high. There is no doubt that on average "bad" kids are much bigger bullies with lower empathy.

The consequences of not knowing how to conduct oneself around good kids as a "bad" kid are much lower in severity. For the most part, the parents of good kids set limits to the aggression of their children. For example, the biggest social rejects at my upper middle class schools weren't ever really bullied. People pitied the complete rejects and most people thought anyone who would bully the rejects is a complete douchebag.

I can't imagine how the social rejects or even the normal nerdy kids at my high school would have fared at a "bad" school.

Then there is the minority immigrant issue. Segregation is good for certain minority groups that do not have the language skills and cultural knowledge to integrate into America. They segregate themselves into a separate community that can provide support for its members. Desegregating these communities and their children will have negative effects.

The children of these immigrants will have the worst time dealing with the "bad" kids because they may have stunted language skills and lack cultural knowledge and context to be able to effectively navigate bad situations.

And to get a better picture of the winners and losers:

- > 90% of African Americans are born in America (and presumably have parents, family and connections who know English and understand American culture).

- Spanish is ubiquitous in the USA and the Hispanic population is huge. Hispanic people have a much lower need to congregate into a community compared to other minorities.

The article's hypothetical gubernatorial scenario is a big turn off. It states that Mississippi has higher SAT scores than Connecticut so Connecticut's Teacher Unions are bad.

As the article doesn't at all remark on the suspicious causality of that, it makes me more skeptical of the rest of the article.

That’s not at all what the article claims. It presents a hypothetical scenario, based on real SAT average scores, where somebody can reach a very wrong conclusion (teacher unions are bad) based on this data. Then the article proceeds to explain how the difference in the average scores can be attributed to self-selection among students who take the SAT rather than unions. So, the article not only remarks on the suspicious causality but the suspicious causality is the article’s main point.

I understand what you are saying, but really don't think that suspicious causality is what the article is about at all. It's about selection bias. (not the same as causality)

Indeed after reading the article I feel it is generally well written and compelling. However the contrived teacher union scenario is a big distraction for me as I was reading the article hoping that issue would be addressed and it wasn't. It would have been much stronger without this gaping logic hole introduced in the beginning.

Nothing of substance to contribute but I'm immensely pleased to see Freddie DeBoer writing again.

I think a corollary of the author's point is that by the time someone is college age, selection bias is the most powerful force in education.

I'd be curious to see a study of younger people to see if perhaps there are some assimilation effects.

Someday, people will wise up to the fact that success has more to do with psychological programming than resources. Actually they won't because they won't ever get enough of the programming themselves to realize it.

I like the way the best public schools in Brazil select: randomly. In the first years is just a draw, in the advanced ones they have just a proficiency test, and then draw among who not failed.


You are rambling. I do not mean that as an insult, just as an observation that might help your writing. Maybe you could do it like (1) Get a book on writing. (2) Read the book. (3) Do the exercises. Seems to work for you.

However, the way you learn best doesn't have to be the way that is best for everyone else. See, when I took calculus in high school, I (1) Got the book. (2) Flipped through it in an afternoon. (3) Decided the exercises were too simple to bother. (4) Went to school where the teacher said the book was worthless and he wasn't going to use it. (5) Graduated high school with credit for freshman calculus due to an agreement with the local university. My point: if you are smart and have a good source of knowledge, it doesn't matter whether it's a book or a teacher, you're going to learn something.

Now, what happens to people who aren't smart, but try to do (1)-(2)-(3) anyway?

First they have to do (1) correctly. You mentioned there are plenty of old, good, cheap books available. I mentioned that the book I got was not good. (It wasn't old or cheap either.) Do you know of a place that aggregates recommendations for the right books to get?

Second, they have to do (2) correctly. Reading a book is simple, right? Except if it was written in by someone who knows their stuff, but doesn't really know which parts are hard, because they all seem equally easy. So the average high school student is going to get stuck on a sentence like "Obviously, a convex polytope in R^n is just a finite intersection of closed half spaces." because they have trouble remembering what all those words mean and where they were defined. Add to that that they might not yet be comfortable with intuitively manipulating the underlying concepts, despite doing the exercises.

(3), the exercises. Consistently high-quality exercises are hard to come by. (Again, if you have any recommendations ...) I have seen exercises that introduced a new concept just to be able to define the problem. I have seen exercises that included a hint for an intermediary step to the solution, except the hint could only be arrived at by a mistake in the correct derivation.

If a student gets stuck somewhere along the way of doing (1)-(3), what they need is someone who can find out what the student doesn't yet understand and then explain it to them. Essentially, a teacher. If you have the necessary perseverance, you can probably replace the teacher by trying other books until one has the right explanation to help you across the hurdle. Most students would probably still prefer a good teacher.

Ok, most teachers aren't actually good. But most books aren't actually good either. It all comes down to being able to select the right ones.

My writing was fine! It was effective! My first post on that topic was too short. My "rambling" -- actually it was well enough organized -- worked because you understood it just fine.

As I mentioned, to solve some of the exercises I gave, will need the college material. That one about a convex polytope is an example. It turns out, that is not an easy exercise. In particular, surprisingly, it does not yield to the usual ideas in, say, W. Rudin, Principles of Mathematical Analysis.

My first post was partly an exaggeration: Anyone who can work all those exercises might just be admitted to, say, second year in a Ph.D. program. The exaggeration was so extreme that, sure, anyone who can work those definitely has done a lot of independent reading and is in nearly all respects quite far above college level material.

The questions you asked about the books do need to be answered, but to save length mostly I omitted answers. Actually, the omitting is okay: Finding good books is part of the work.

So, how to do that? Sure, for the high school books: (1) do an Internet search and see what the most popular/recommended books are/have been. (2) Go to the local Board of Education, talk to their subject matter specialists, see what books they recommend. (3) Go to high schools recommended as being really good and see that books they are using. (4) Anything else can think of.

For your concerns that a student could get stuck on the exercise about convex polytopes, they likely won't encounter such a problem because, as I warned, it needs some college material. But maybe the challenge of the problem would motivate a determined student to do what they could to get a solution, and that might be good.

For that question about the high school material, that material is so simple and the best books should be so well written that there should be no problems -- I never had such a problem in first year algebra, ..., solid geometry.

For a college calculus text, again, the best of those books have been so highly polished for so long that, again, no teacher should be needed. In particular, I did make clear to avoid the AP Calculus materials I regard as junk.

Then in what I wrote, the next book would be abstract algebra. I didn't get stuck there, either, but I didn't try to work all the hardest exercises. I might have included, "If in the whole book, omit, say, 10 exercises, fine. Why? Because some of the exercises might actually have errors in their statements, are placed in the book before the material needed for a solution, or for a solution actually need material outside the book and much more advanced."

With that cautionary, wise, flexible, non-rigid, and non-absolute statement, your concerns about students getting stuck and needing a good teacher shrink.

Sure, as I stated, my suggestion to solve the problem of the OP, that is, for disadvantaged students, was only for math and only for talented students. That for such students it appears, from my background, yours, and much more, that my partial solution to the terrible floundering around struggles of the OP are quite good. It's no joke: A lot of talented, disadvantaged students should be able to do this.

Gee, guys, a lot of talented, disadvantaged students do really well at basketball which in some ways is more difficult, e.g., as I mentioned, no teacher, text, or exercises. And no answers in the back of the book.

Heck, I wasn't disadvantaged, but, still I got a Ph.D. in applied math from one of the world's best research universities, and 85+% of everything I learned for that degree and 90+% of all the pure/applied math I have learned was from independent study much like I described, e.g., how I did really well with high school plane geometry, college calculus, and more. I omitted some of the details of what I did, a LOT, with linear algebra, ordinary differential equations, statistics, numerical analysis, and more.

Again, my point is: For the struggles in the OP, there is a partial solution: For a disadvantaged but talented student, go for math with a lot of independent study. Why? For one, the approach of just get a book, read the book, work the exercises can work really well there, all the way through qualifying exams in the math department at Princeton (on their Web site at least at one time they stated that courses are introductions to research by experts in the fields, no courses are given for preparation for the qualifying exams, and students are expected to prepare themselves for the exams on their own).

For the grim stuff in the OP, my suggestion is a very good to know lesson. In some good ways, it's much better than anything I knew until well into my Ph.D. Had I and/or my parents understood this lesson, then starting in about the sixth grade I could have just raced ahead and been ready for math graduate school by high school graduation.

You seem to understand some of what I wrote. If you want, then add to what I wrote, e.g., with some books, say, MathOverflow, and maybe more. And maybe there are some good massive open on-line courses (MOOCs). But I'd say to be careful: The time I looked at Khan Academy on calculus, I concluded that they didn't understand calculus well and had bad material. And I've commented on the AP Calculus material.

Uh, sure, James Simons has been running Math for America or some such. Well, there maybe some efforts there to help talented students who want to rush ahead in math and use that as a way to get free college and grad school educations. Sure, Simons was math chair at Stony Brook so knows a lot about math education. And he may be able to recommend some really good books!

Generally, a student who has done really well on their own in math deserves and maybe often can get a lot of praise and scholarship offers. Or, IIRC, good graduate departments in math have a lot more tuition scholarships than they have applications from good students.

> My writing was fine!

Your posts are too long. This is one of the reasons you're getting downvotes.

You are really super slow on the uptake, reading comprehension. Again, once again, over again, yet again, one more time, this time just for you, my writing is fine. Just fine. Nothing wrong with it.

Instead of anything wrong, I presented a very well informed, well documented, well reasoned, well explained, serious, practical solution to a really big problem in the OP -- how to get good educational results for disadvantaged kids. Maybe if I repeated that statement 5000 more times, 1000 different ways finally your bone headed reading comprehension and arrogance would finally begin to get it.

My original post at


actually WAS short. No one but no one here understood it. So, I gave a longer post, repeating over and over the simple, central point. Finally one reader began to understand but not very well. Then you come along, also don't understand, and piss on my leg and want me to think it's raining.

For what I posted, you don't get it. I just gave an innovative, powerful solution to part of the nasty problem in the OP, and you just flatly didn't get it. So, with your slow comprehension, I didn't write too much; I wrote too little. You needed much more.

In the future I suggest you avoid anything I post. My posts are way over your head, and I'm not often going to stoop and bend to repeat, explain over and over, different ways so that people like you can understand.

You can't understand what I wrote or most of what I write.

You CAN understand? Nope, I don't believe you. To see, let me see your solutions to any three of the exercises I listed in my first post at


I'll return in 24 hours and grade your results. Bet: From just your own work, you can't work even one of the exercises.

Uh, I'll make it easier: For the exercise in infinite differentiability, you are welcome to get all the help you can find in 24 hours. Use the Internet. Ask math profs. Go for it! Let's see your really good abilities!

You are not up to my level.

You should not to comment on my posts: You don't have the background or ability.

Just don't read my posts. Please don't. Read something else.

You're being incredibly rude to other community members. Since this is obviously not OK, would you please stop?

The problem with your posts is not that they are hard to understand, it's that they are not nice to read. For one, you come off as arrogant, as if you were the only one who has any idea what they are talking about. You present a solution to help disadvantaged students learn better, and I agree that your solution is fine. But it will only work for students that have the will, time and ability to help themselves, and I think that is only a tiny minority of all students, and even tinier when you only consider economically disadvantaged ones.

That your first post didn't mention any of the factors that could make your solution unworkable probably contributes to the downvotes, because you come across as somewhat naive, and not in a cute way. I don't think it deserved to be flagged, but it wasn't a good post either.

Your second post is much too repetitive. Repetition is very memorable (you got "Get the book. Read the book. Do the exercises." stuck in my head quite well), but if you don't enrich it with convincing arguments, it just becomes tiresome to read.

Your third post is actually quite reasonable (but still really long, not sure if you could have made it shorter). You clarify that your solution is only for talented students and you mention a bunch of things that would be good for such students to know.

Your fourth post is really bad. You directly attack someone who was trying to help you with stylistic advice and tell them that they didn't understand your post, deny that there is any problem and proceed to repeat yourself. That you got called out by a moderator should give you a hint how far off the mark your writing is.

That said, consider me nerd-sniped! I quite enjoyed doing your exercises, although I had to look up some definitions here and there. Unfortunately, they don't fit in a single comment, so I have to see how many self-replies I am allowed.

Example: In a separable metric space, each closed set is the union of a perfect and a set that is at most countable.

Definition of separability [1]: a topological space is called separable if it contains a countable, dense subset; that is, there exists a sequence of elements of the space such that every nonempty open subset of the space contains at least one element of the sequence.

Defintion of perfect set [2]: a subset of a topological space is perfect if it is closed and has no isolated points.

Let C be a closed set in a separable metric space. Choose the perfect P as C minus the set of C's isolated points I. Since they are isolated, we can assign to each point p in I an open neighborhood N_p without any of the neighborhoods overlapping. Because each neighborhood contains at least one distinct element of the separating set S (is that what you call it?), the cardinality of I is at most that of S, i.e. countable. Thus C is the union of a perfect (P) and set that is at most countable (I).

[1] https://en.wikipedia.org/wiki/Separable_space [2] https://en.wikipedia.org/wiki/Perfect_set

Examle: For positive integer n and the set of real numbers R, show that convex f: R^n --> R is continuous.

Choose an arbitrary point x in R^n and consider a hypercube centered in x. Each face of the hypercube together with the centerpoint x determines a hyperpyramid, and each point p within the hypercube is contained in at least one such hyperpyramid.

Since the hyperpyramid is convex, the point p can be expressed as a convex combination of its vertices, which is linear in the coordinates of p. Due to the convexity of f, there is an upper bound of f(p) that is a convex combination of the values of f on the vertices of the hyperpyramid, using the same coefficients. This means that the upper bound is linear in p as well.

Importantly, the bound is identical on the boundary of two hyperpyramids, since the coefficients will only involve those vertices that are shared between them. This means that f is bounded above by a piecewise linear function, which is tight in x (with the trivial convex combination).

There is a second hyperpyramid to consider, which is formed by p together with the opposite face of the hypercube, and which contains x. By the same convexity argument, there is an upper bound on f(x) that is a convex combination with coefficients linear in p.

The bound can be written as sum_v w_v f(v) + w_p f(p) >= f(x) and rewritten as f(p) >= 1/w_p (f(x) - sum_v w_v f(v)). This is possible because w_p = 0 would imply that x is a convex combination of the v alone and therefore in the hyperface. This bound is no longer linear in p, but still continuous.

Like the upper bound, this bound is identical on the boundary of two hyperpyramids containing p, because only the vertices shared by the opposite faces will be relevant. Additionally, the bound is tight in x as well, where the convex combination is w_p = 1, w_v = 0.

In summary, f is bounded above and below by continuous functions in a neighborhood of x, with the bounds coinciding at x itself. This implies that f must be continuous in x as well.

Example: For positive integer n and the set R of real numbers, for any closed set C in R^n, there exists a function f: R^n --> R so that f(x) = 0 for all x in C, f(x) > 0 otherwise, and f is infinitely differentiable.

I have a hunch that this can be solved similarly to the construction of bump function on a given compact set [4], except the support is an open set and I'm not sure what the "appropriate scaling" should look like.

[4] https://en.wikipedia.org/wiki/Bump_function#Existence_of_bum...

Example: Prove that there are no countably infinite sigma algebras.

Definition of sigma algebra [3]: a σ-algebra (also σ-field) on a set X is a collection Σ of subsets of X that includes the empty subset, is closed under complement, and is closed under countable unions and countable intersections.

Assume a countably infinite sigma algebra does exist. Consider the sets in the algebra that do not have non-empty proper subsets in the algebra. In particular, their intersections with other sets are always empty. If there is a countable infinity of them, they can be mapped to the one-element sets of natural numbers, and their closure under the operations of the sigma algebra is isomorphic to its powerset, which is uncountable.

Therefore there can be only finitely many such sets. Now consider the collection S of sets that remains after removing their (finite) closure under the operations of the sigma algebra. Obviously S is still countably infinite. Since each set A in S has a proper non-empty subset B in the sigma algebra, it also has another: the intersection of the complement of B with A. If those subsets are also in S, they in turn can be split into two non-empty proper subsets.

If this process of binary splitting stops at a finite depth, it creates a binary tree whose leaves are not in S (since they can't be split) and whose inner nodes are the unions of their two children. By induction, this represents each node in the tree as the union of finitely many sets not in S, including the root node A. However, this contradicts the choice of A, which means that the process never stops and there is an infinite path of the tree.

Now consider those nodes that are split off this infinite path, i.e. those that are children of a node on the path, but are not on the path themselves. Since each of them is by construction disjoint from their siblings on the path and their descendants, they form a countably infinite collection of disjoint sets. By the same construction as above, they can be mapped to the one-element sets of natural numbers, which means that their closure is uncountably infinite.

Therefore, the assumption that countably infinite sigma algebras exist is false.

[3] https://en.wikipedia.org/wiki/Sigma-algebra

Example: For triangle ABC, by Euclidean construction, find D on AB and E on BC so that the lengths AD = DE = EC.

Running out of time here. I guess you can do it algebraically: two variables for the placement of D and E, two quadratic equations, get the solutions in terms of square roots, multiplication, division etc., then implement the calculation using compass and straightedge.

Example: For positive integer n and the set of real number R , show that, on any finite intersection of closed half spaces C in R^n, any linear function that is bounded above on C achieves its least upper bound on C.

If C is empty (i.e. the closed half spaces do not overlap), the claim is false, since every real number is an upper bound of every function on C, therefore the least upper bound does not exist.

However, for non-empty C and a linear function f bounded above on C, any x in C induces a lower bound l for the upper bounds, which guarantees the existence of a least upper bound u.

The point where u is achieved can be determined by an iterative procedure involving a set of half-spaces H (initially H_0 is empty), a point x in C (initially x_0 is chosen arbitrarily) and a search direction d (initially d_0 is the gradient of f).

The following invariant is maintained: for any boundary of a half-space in H holds that x is on the boundary and d is parallel to it, and for any x' in C, f(x') > f(x) implies that the dot product d (x' - x) is positive.

This holds for the inital conditions, since H_0 is empty and d_0 is the gradient of f, which means that d_0 x' = f(x') > f(x) = d_0 x is equivalent to d_0 (x' - x) > 0.

If d ever becomes zero, the invariant implies that there are no points x' in C with f(x') > f(x), which means that f(x) = u.

Otherwise, the value of x_{n+1} is chosen from the linear subspace x_n + t d_n, where t in [0, infinity). As this is a closed, convex set, its intersection with C (which is an intersection of closed, convex sets) is itself closed and convex. Since f keeps increasing linearly with t, the upper bound u on f immediately induces an upper bound on t. Summarily, this constrains t to an interval [0, s]. Then x_{n+1} = x_n + s d_n.

This step maintains the invariants, since x moves parallel to all boundaries of sub-spaces in H_n and x_n was contained therein, thus x_{n+1} is also contained in all these boundaries. Because the search direction doesn't change, its invariant is maintained as well.

At this point, there must be a half-space h that "cut off" the line at x_{n+1}, which means that x was not moving in parallel to it. Setting adding h to H_n keeps x_{n+1} contained within all the boundaries, but makes d_n no longer parallel to them. By also setting d_{n+1} to the component of d_n that is parallel to the boundary of h, this can be reinstated.

However, now it must be shown that f(x') > f(x) implies d_{n+1} (x' - x) > 0. From the previous step, it holds that f(x') > f(x) implies 0 < d_n (x' - x) = (d_n - d_{n+1} + d_{n+1}) (x' - x) = (d_n - d_{n+1}) (x' - x) + d_{n+1} (x' - x) where d_n - d_{n+1} is perpendicular to the boundary of h and away from it. For d_{n+1} (x' - x) to be non-positive with the overall result staying positive, (d_n - d_{n+1}) (x' - x) would have to be positive. But this means that x' must be a non-zero distance outside h, and is therefore not in C. The only remaining possibility is that the invariant is maintained.

The above algorithm must terminate in a finite number of steps, since there is only a finite number of half-spaces that can be added to H. Because the algorithm can only terminate by finding an x in C where f(x) = u, this proves that the least upper bound is achieved.

Registration is open for Startup School 2019. Classes start July 22nd.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact