
What the world can learn from the UK's A-level grading fiasco - ransith
https://blogs.lse.ac.uk/impactofsocialsciences/2020/08/26/fk-the-algorithm-what-the-world-can-learn-from-the-uks-a-level-grading-fiasco/
======
makomk
In reality, I think this fiasco has more to do with politics than algorithms.
One of the big problems is that the alternatives had downsides which were
largely glossed over for political reasons - for example the university growth
cap isn't just an arbitrary creation, it exists to stop more prestigious
universities siphoning off a disproportionate share of students and leaving
less prestigious ones in deep trouble and this seems likely to happen now.

Another big problem is that private schools seem to have made more realistic
predictions of their students' results than state schools, as demonstrated by
the fact that just taking many of their predictions as-is only caused slightly
more grade inflation than in the state school results where 40% were
downgraded - and this was portrayed in the press as proof that it was an
attack on state schools because their results were disproportionately affected
by the algorithmic downgrading.

A third problem is that what it was OK to be concerned about varied depending
on which political side it benefitted. For example, there was a completely
false and made up claim on social media and in publications that should know
better that the Education Secretary had said taking the results would cause
students to be promoted into jobs that they weren't competent to do:
[https://fullfact.org/education/gavin-williamson-fake-
quote/](https://fullfact.org/education/gavin-williamson-fake-quote/) I can't
imagine that it would've gone down any better if he claimed students would get
onto courses they weren't good enough for, yet a week later after GCSEs went
the other way and used predicted results that exact claim was uncritically
regurgitated by the media.

~~~
remus
It seems like a classic political no-win situation: if you go with an
algorithm then anyone who gets a lower than predicted grade is going to
complain (and of course has every right to, as any algorithm is likely to
unfairly disadvantage some section of the population). If you give everyone
their predicted grades then you're accepting massive grade inflation which
makes the grades far less useful as a predictor of a student's ability which
then means lots of problems further down the line.

~~~
asplake
However, the unfairnesses of the two options are very different. On the one
hand, you leave it to educators and employers to select candidates based on
unreliable grades, versus excluding swathes of the population from
opportunity. Leave it to decision-makers that seem to fetishise exam grades
and you come to what I'm sure the vast majority of people regard as the wrong
conclusion. Yes, there would have been fallout, but so much less toxic had
they chosen otherwise.

~~~
rbecker
> On the one hand, you leave it to educators and employers to select
> candidates based on unreliable grades, versus excluding swathes of the
> population from opportunity.

That "versus" is unwarranted - unless everyone is given top marks, exclusion
happens in either case. In fact it would happen in that case as well, as top
colleges (practically by definition of 'top') cannot accept everyone. All
you've done is changed who gets excluded, by not adjusting for differing
school grading.

> decision-makers that seem to fetishise exam grades

Calling it "fetishizing" is a fine way to suggest there's something wrong with
it, without stating what, or how to improve it. Would it be better if, instead
of on the basis of grades, students were judged based on who they know, or how
much they can donate to the college?

~~~
asplake
Ofqual Chairman Says It Was A "Fundamental Mistake" To Believe Algorithm
Grades "Would Ever Be Acceptable"
[https://www.politicshome.com/news/article/ofqual-
algorithm-m...](https://www.politicshome.com/news/article/ofqual-algorithm-
mistake-williamson)

~~~
rbecker
"Students will now receive grades based on their teacher’s estimate of what
their grade would have been"

But if those estimates are improved using statistics, there's a political
fallout. There's _some_ valid criticism of the algorithm used (far less than
that BBC article tries to imply), but there's no question the algorithm's
estimates were more accurate.

So much ink was spilled calling the algorithm biased for its 4% increase in
A-grades for independent schools, yet teacher's 40% increase of grades above
the expected average is... what? Unbiased?

On any other topic, such a position would be called "anti-science".

~~~
asplake
There’s a fundamental asymmetry here. Fail a student unfairly, and the harm to
them is potentially irreparable. Pass more than usual, and you increase
competition for places and while there’s certainly some unfairness there, the
system will ultimately compensate through interviewing, delayed starts, etc.

~~~
rbecker
> the system will ultimately compensate through interviewing, delayed starts,
> etc.

A roundabout way of saying that some students that would have been accepted to
their chosen college, won't be, because their grades weren't as inflated as
their competitions. Isn't that also potentially irreparable harm, not just
"some unfairness"?

~~~
asplake
There’s a massive difference between not getting your preferred place and
being denied the opportunity even to apply

~~~
rbecker
So you would consider using the algorithmicaly adjusted grades an improvement,
_if_ in case of a failing grade, the teacher's estimate was used instead?

And a follow-up question: How many such cases were there? Where the adjusted
estimate failed a student, but the teacher's estimate didn't?

------
supernova87a
This virus situation has (or will) produce a lost year of productivity. But
people don't want to admit that and are still trying to operate under old
constraints or processes. It will break somewhere. They want to keep schools
running, and years / classes of students progressing through. But they also
want to have some semblance of standards, without being able to teach properly
or assess progress properly. All these factors cannot be simultaneously
satisfied.

As others have said, you might just have to drop standards and have a lottery,
or open enrollment. Putting in these ridiculously bad algorithms is worse than
falling back on random chance.

High school/university is where it becomes easiest to hide behind bad logic
and less obvious consequences. I hope we're not going to do the same for
doctors, pilots, etc. and just let them pass the test because classes were
canceled.

~~~
simonh
It's not about passing the test though, it's about sorting out who gets to go
to university and who doesn't.

If the training of pilots one year was compromised so none of them could
certified, we'd go a year without certifying any new pilots. We'd cope
somehow. Universities can't go a year without any new students though, that's
just not a reasonable outcome.

On the face of it basing grades on predicted outcomes and adjusting for the
historical record of specific schools to overestimate those predictions seems
to be a pragmatic approach.

~~~
heavenlyblue
You can’t learn to fly without really flying (although that is also probably
arguable - there should some skills which are transferable from simulators).
However a lot of western higher education is based on giving enough info on
how to study yourself.

------
scast
Serious question: Why weren't the exams taken?

Everything I have read about this glosses over this critical bit which I am
very curious about. I understand there is/was a worldwide pandemic and all of
that, but it feels to me that there wasn't necessarily a reason exams couldn't
be taken in socially-distanced class rooms (with students wearing masks, as
well as the teachers, of course) or if needed be in repurposed venues, like
football stadiums (leaving plenty of space between students).

Was there a concern for fairness should the exams proceed? Or was it safety?
Or did the government put their foot down on no exams? Or was it the
teachers/unions? What happened?!

~~~
DaiPlusPlus
> it feels to me that there wasn't necessarily a reason exams couldn't be
> taken in socially-distanced class rooms (with students wearing masks, as
> well as the teachers, of course) or if needed be in repurposed venues, like
> football stadiums (leaving plenty of space between students).

Exactly what you’re proposing just isn’t feasible, because of the sheer
numbers. I see American posters on reddit complain about having to do a
handful of finals at the end of high-school, and the SAT/etc as well, which I
understand is optional.

“In my day...” (wow I’m old... this was only about 12 years ago) my sixth-form
had about 1,300 students (650 in yr12/L6, 650 in yr13/U6). Most L6 students
did 5 AS-level subjects and 4 A2 subjects. Most subjects are then comprised of
modules (e.g. In mathematics: Pure P1/P2/etc, Stats S1/S2/S3/etc, Mechanics
M1/M2/etc, Decision/Discrete D1/D2/etc), and each module has its exam at the
end of the term/trimester.

In my case, I took 5 subjects at AS-level (L6) and had no less than 8 exams in
the May-June of my first year. Multiply that by 1,300 - with probably over 100
different exams. That’s almost 11,000 Covid-safe exam-sittings that need to be
arranged.

Social-distancing regs mean that you need _four times the floor-space_ for an
exam room than previously (doubling distance from 3 feet to 6 feet in both
directions). Doing exams outside on a field wouldn’t work: inclement weather
and even a gentle breeze and writing on paper difficult and distracting. Doing
it indoors means you’ll not only fill your sports-halls and cafeterias (which
we did every year anyway) and need to spill-over into smaller classrooms -
which means you need many more exam invigilators.

...and invigilators, in my experience, tended to be older people (60s-70s) -
often recent retiree teachers. The exact same people who are quite
legitimately fearing for their life over Covid so we can’t blame them for
choosing to stay home.

...so we have a 2-month long period where schools and colleges need 4x the
space and with far fewer authorised staff to oversee it.

Some schools will be able to handle it, others won’t. If it’s a combination
secondary-school + sixth-form then they’ll also need to handle GCSE exams for
the yr11 kids and SATs (unrelated to the US SAT exam) for the kids in yr9.
Additionally schools also have “mocks” for yr10 (mock-GCSEs, but they still
count towards your score in yr11). There may also be additional testing done
for other years at the county-level. So that’s another few thousand exam
sittings to add to that. My secondary-school sent the Yr7 and yr8 kids home
for a week if they were overloaded with handling exams for so many. So if the
LEA/exam-boards were to press-on with the exams then that’s unfair to the kids
at schools that don’t have the capacity.

——-

Grading “by algorithm” - especially when that algorithm isn’t public - nor
probably even we’ll-understood by the MPs in-charge - is a bad idea, yes. But
I can’t think of a workable alternative: people’s lives should not depend on
the outcome of exams - but we can’t trust teachers own subjective grading of
their own students to be necessarily and sufficiently objective enough.
There’s no economically-viable solution to this problem - even without a
pandemic going on.

——

Now that I think about it - I suppose _one_ option would be still do in-person
exams, but only do the core/essential exams for the most important course
modules and use that as the basis for university admissions - so if this
pandemic happened 12 years ago I’d only sit the P1/P2/P3/etc exams for
mathematics and disregard Mechanics/Statistics/Discrete - ditto for physics,
and so on. So the exam load would drop from my estimate of 11,000 to maybe
7,000-ish - I don’t think that would be small enough to manage still)

~~~
Twixes
This was feasible here in Poland and some other European countries managed to
proceed with exams seemingly safely as well.

------
lordnacho
It's hard to see how the algo could do anything but disappoint. If you don't
have the exam information, what information have you got left? Just results
from previous years, and a guess as to where you might be in the class, from a
teacher who doesn't want to disappoint anyone.

But people have since forever seen exams as a way to stand out from the crowd,
individually. If you went to a bad school, exams were a chance to show some
extra effort. If you were a slacker, you'd be found out.

Without the specific exam results, all you've got is the reputation of the
school, and chances are variation within a school is greater than between.

------
jonplackett
The problem was not using an algorithm, it was that they used a crap algorithm
that could award students grades that were higher than those even available on
the paper that they sat, and made it impossible to get a high grade if no-one
fromm your college got one before.

I'd love to see the details of the algorithm itself. Surely it can't have been
too difficult to put some guard rails in there to prevent it moving any grade
more than say, two places (maybe even one) from the predicted grade.

Instead it seemed to have the freedom to do whatever it liked. Moving Bs to
fails. Students predicted to fail getting As.

That's just incompetence on the part of whoever made that algorithm.

~~~
tonyedgecombe
_and made it impossible to get a high grade if no-one fromm your college got
one before_

Well that is unlikely. If you are going to handle it algorithmically then what
else would you expect. Clearly the teachers can't be relied on because now
everybody is getting an A*, the grades are effectively useless as an
indicator.

I was disappointed although not surprised that the government caved on this.

~~~
Traster
It's not unlikely, it's practically certain. If you have hundreds of schools
there absolutely will be a handful of students who do better than previous
years and there will be a handful of students who do worse than ever before.
Maybe it's not a big deal to hand a C to someone deserving of an E, but it's
certainly a big deal that the moron at the school that gets good grades (read:
private school) gets a pass, whilst the genius at the school that historically
did badly (read: state school) gets screwed. Now, given that's how this system
works, take a look: do you think the politicans repsonsible for this went to
the state school?

~~~
tonyedgecombe
There are always outliers.

Interestingly the private schools seem to be to be grading their students more
accurately, the problems were more common in state schools. The moron in a
private school was unlikely to get their grades inflated. I'm not a fan of
private schooling but in this case they appeared to be doing the right thing.

There will always be outliers, no system is perfect but what we have just done
is worse in my mind.

------
LatteLazy
We know that university admissions is a very poor, deeply flawed process. That
people object this year but didn't object last year is very telling. It shows
that people will happily accept a bad system with poor results AS LONG AS they
are given some small illusion of personal control over that system. As long as
people can say "people like me are very unlikely to do well BUT _I_ will be
fine because I will work harder than people in my group do" they're happy,
even though they are by definition wrong most of the time.

~~~
NovemberWhiskey
The students just wanted to be measured against the objective standard that
they were promised.

This doesn't require assumptions of naïveté on their part about equality of
opportunity w.r.t teaching quality, schools resources and so on.

------
rob74
Well, what can we learn from this? That it's not a good idea to simply assign
a grade to someone for an exam they didn't take, and it's an even worse idea
to base this grade only partly on their own performance? And, if you are going
to do it, you'd better make sure that the grades exceed the students'
expectations, otherwise of course there will be protests...

------
076ae80a-3c97-4
The only true fair solution would be to put everyone back at least a year and
wait until the exams can actually be done. In the mean time they could've
invested the energy in getting an online exam system working and
improve/innovate online learning mechanisms so that if we go back into
lockdown again they are ready for it.

~~~
simonh
I don't think doubling the size of the classes in the year below, and forcing
schools to retain an extra year's worth of students is at all practical or
would actually solve any problems. Let alone all the new problems it would
create. What do Universities do if they have a whole year's worth of students
just not show up, then suddenly gets two years worth come in at the same time?

I suspect you've not actually thought this through.

------
tokai
That anyone ever thought it was a good idea boggles the mind. Random lot would
have been better.

~~~
anitil
Oh think of the natural experiment if you randomly assigned Universities to
students.

Awful, unethical, but fascinating

------
dang
Recent and related:
[https://news.ycombinator.com/item?id=24185621](https://news.ycombinator.com/item?id=24185621).
I feel like there may have been others?

------
agent008t
How did they deal with this problem in other countries, say Ireland or Japan?

~~~
makomk
It looks to me like Ireland was planning on using literally the exact same
process, but after it caused such a political problem for the UK government
they announced that data on how schools had previously performed wouldn't be
used in the standardisation process:
[https://www.citizensinformation.ie/en/education/state_examin...](https://www.citizensinformation.ie/en/education/state_examinations/leaving_certificate_2020_calculated_grades.html)
(Notice how the linked explaination document from May 2020 says this would be
the basis for the calculated results.) How that's meant to work I'm not sure.
The results aren't out yet.

~~~
smcl
Additionally Scotland had an similar issue - algorithmically adjusted grade
estimates were discovered to have punished kids from poorer schools. This
happened a few weeks earlier than the English A-Level results, which is
interesting because the ruling party in the rest of the UK:

1\. took a GREAT deal of pleasure in loudly trashing the Scottish government
in the press ...

2\. knew they faced the EXACT same issue themselves ...

3\. did absolutely nothing about it ...

4\. then did a frantic cleanup of the attacks (deleting tweets and such) after
the English debacle unfolded

At this point I shouldn’t be surprised. It’s to be expected - they’re
politicians playing their little game. But this debacle was avoidable - they
could have faced the music earlier and started coming up with an alternative,
but they chose instead to gloat, score a cheap win and kick the metaphorical
can down the road.

------
renewiltord
Why not just hold the tests, except outside and far apart. They have stadiums
and stuff. Honestly, this whole thing seems like a manufactured problem.

You could hold them outside, and stagger them so there are ten papers, and
then just normal-curve-adjust to the first paper.

Other countries already do multi-day tests easily. And if you don't want to do
curving, just treat it the way you treat GREs/IELTS/TOEFL etc. Don't curve and
just use overall percentile.

~~~
oh_sigh
Maybe cheating? It would be easier to sneak a peek at your smartphone if one
proctor has to cover an acre of students.

~~~
swyx
if you can cheat a test off a smartphone its probably not a good test

~~~
oh_sigh
Cheating via smartphone can encompass anything from directly looking up an
answer, to sending someone the essay prompt and having them send you a
completed essay which you just copy down.

Can you name a good test by your standards?

------
_0o6v
Having worked with engineers and software managers in the UK Government, it
doesn't surprise me that this was such a fuckup.

The solution would have been not to completely close schools in the first
place - there was no evidence that children were particularly susceptible to
COVID or at risk then, and there's lots of evidence now that they aren't.
Exams could well have taken place as normal - exam halls are, by definition,
socially distanced.

~~~
makomk
Not closing schools was the government's preferred option for exactly the
reason you point out. Unfortunately, the teachers' unions had different views
on the matter and it's rather difficult to keep schools open without teachers,
especially when they can easily just carry out a mass sick-out. Exam halls are
probably a bit of a transmission risk though since they involve a bunch of
people who wouldn't normally mix sitting in the same room sharing the same air
for a few hours.

~~~
nicky0
Also once the schools have been closed it becomes a lot harder to properly
prepare the pupils for the exams.

~~~
smcl
You’re right and this is often ignored - the few weeks or so prior to exams
where the curriculum is largely over can significantly change your grades

