
The Secret Algorithm That's Keeping Students Out of College - _bxg1
https://www.wired.com/story/algorithm-set-students-grades-altered-futures/
======
ppod
>grades from past grads at their school to predict what they would have scored
had the pandemic not prevented in-person tests

Wait, this is like reverse affirmative action, no? Students at schools that
previously performed badly will have their grades downweighted?

~~~
chongli
_Students at schools that previously performed badly will have their grades
downweighted?_

The article does not make it easy to understand how the actual model works.
According to the IBO [1]:

 _We will use a calculation that is based on the relationship between
coursework marks, predicted grades and subject grades to estimate the subject
grades candidates would have received if the exams had gone ahead. If the
relationship between these elements shows that in previous sessions candidates
globally tended to achieve higher outcomes on their exams than their
coursework, the calculation used this session will reflect that._

You can call this "reverse affirmative action", but really it's anti-grade-
inflation. If you attend a school where students historically perform much
better on coursework than on the exam, your predicted score on the exam will
be lower. On the other hand, if your school tends to perform better on the
exam than in coursework, then your predicted score on the exam will be higher.

So why would one school underperform on coursework and another underperform on
exams? Grade inflation. Lots of schools make their coursework easier to try
and boost the chances their students get into university. The whole point of
standardized testing is to counteract that.

So in this case, it may be very unfair to some students who might have been
outliers on the exam, but there's no way of knowing that without actually
running the exam.

[1] [https://ibo.org/news/news-about-the-
ib/covid-19-coronavirus-...](https://ibo.org/news/news-about-the-
ib/covid-19-coronavirus-updates/)

~~~
s1artibartfast
> If you attend a school where students historically perform much better on
> coursework than on the exam, your predicted score on the exam will be lower.
> On the other hand, if your school tends to perform better on the exam than
> in coursework, then your predicted score on the exam will be higher.

This is terrifying. It works for the average student, but distorts the
performance for any student who isn't typical or better than their school.

It seems that it would be better to just ignore the testing component of the
IB score this year when judging students. Instead they modeled what the
students would have gotten on the test, and then judged them based on their
modeled performance.

~~~
chongli
_distorts the performance for any student who isn 't typical or better than
their school._

It distorts the performance of a student who underperforms in coursework
relative to their peers but overperforms on exams, at a school where students
overperform on coursework and underperform on exams. In other words, it
punishes the student who does unusually poor on coursework that is already
much easier, on average, than typical coursework from other schools.

How likely are such students to succeed at a competitive university? Not very
likely at all, I would think.

~~~
s1artibartfast
I don’t agree. It also punishes students with good coursework and who would
have performed well on the test, but happen to be in schools which
historically perform poorly on IB tests.

~~~
chongli
If the student outperforms their peers on coursework, then they won't be
punished, they will have an estimated score higher than their peers.

~~~
s1artibartfast
yes, they would be higher than their peers _at their school_ , but then
penalized because the cohort before them performed poorly on the IB test.

For the students who would have gone on to test well on the IB test, this is
clearly a disadvantage.

E.g. : 1) You are in the 10th percentile of your class. 2) Students from your
school in the 10th percentile scored in the 20th percentile on the IB test
last year 3) You are assigned a 20th percentile on the IB test you can not
take.

This sucks if you would have done better than the 20th percentile on the IB
test.

Why model an IB test score at all? just apply a weighing to the coursework
grades if you want to standardize outcomes between schools. it would be more
straight-forwarded.

~~~
chongli
They modeled the scores because they couldn’t hold the exams this year due to
the virus. As for your description, that’s a gross oversimplification. What
they’ve actually done is predicted people’s test scores based on their
performance in coursework relative to their peers.

A person in the 10th percentile in coursework is already worse than 90% of
their peers and is unlikely to go to university no matter what school they
attend. On the other hand, a student in the 90th percentile of a low
performing school is going to be fine. Their predicted score is going to be
well above their peers.

~~~
s1artibartfast
> They modeled the scores because they couldn’t hold the exams this year due
> to the virus.

I said why make up a test score instead of just using weighted coursework.

> As for your description, that’s a gross oversimplification. What they’ve
> actually done is predicted people’s test scores based on their performance
> in coursework relative to their peers.

This is exactly what I described. taking someone's performance percentile in
their school and predicting what they would score on the IB test. This appears
to be exactly what they are doing

>A person in the 10th percentile in coursework is already worse than 90% of
their peers and is unlikely to go to university no matter what school they
attend

Sorry I was not more clear, I meant top 10th percentile.

>On the other hand, a student in the 90th percentile of a low performing
school is going to be fine. Their predicted score is going to be well above
their peers.

No, a student in the 90th (top 10th) percentile of coursework at a low
performing school is the ones who are harmed. They will be assigned a higher
IB test score than most of their school, but lower than the 90th (top 10th)
percentile student in a school with a better track record on the IB test.

~~~
chongli
_I said why make up a test score instead of just using weighted coursework._

That's what they're doing. They're weighting someone's coursework by an
estimate of how they'll do on the test, based on how well students at their
school have done on the test in previous years relative to coursework in those
years.

 _No, a student in the 90th (top 10th) percentile of coursework at a low
performing school is the ones who are harmed_

Of course, they are harmed by the fact that their school makes coursework too
easy. The alternative, of course, is using unweighted coursework grades which
harms the students at schools that make the coursework extra hard.

The problem is that the IBO doesn't actually know what a student's abilities
are until they write the exam. They also don't know how hard the coursework is
at their school, in an absolute sense. They only know how hard the coursework
is relative to the exam, based on a comparison of previous years' course
grades with exam grades. Schools that consistently make coursework too easy
will inflate the grades of their students. If those students underperform on
the exam, the IBO will know that those schools are practicing grade inflation
and adjust accordingly. That's all there is to it.

------
prepend
In the future I just have to do really well in kindergarten and all the rest
of my school and career will be predicted by a ML model with no correction
whatsoever.

I’m an IB grad and 5/6 exams could have been done electronically. Should at
least give students the option to take exams.

------
quantified
The article doesn’t include any commentary related to students that were mis-
predicted in the other direction, i.e higher-than-expected scores. If the
results stand, they’ll be an interesting case study in the predictive value of
IB scores on college performance.

~~~
MattGaiser
I suspect nobody is willing to go on record about that.

------
naveen99
My 13 year old son just started doing classical mechanics on ocw.mit.edu No
one can keep students out of open courseware. signaling is a zero sum game,
but there is always a shortage of students who have the interest and aptitude
to learn.

