
Why did the A-level algorithm say no? - dynamite-ready
https://www.bbc.co.uk/news/education-53787203
======
corin_
Two things worth noting that the article doesn't go into...

1) _" The year-on-year rise in proportion of students achieving A or A+ grades
was much higher at independent schools than state comprehensives"_

[https://www.theguardian.com/education/2020/aug/13/england-a-...](https://www.theguardian.com/education/2020/aug/13/england-
a-level-downgrades-hit-pupils-from-disadvantaged-areas-hardest)

Essentially, students lucky enough to be in private education were more likely
to see their grades go up / stay the same, compared to students receiving free
education seeing grades going down.

Has been explained away as a byproduct of a sensible algorithm, but
considering the government's track record and private school backgrounds (85%
of the UK's Prime Ministers have been educated at Eton, arguably the country's
most exclusive private school) it's hard not to be sceptical that this wasn't
intentional and class-motivated.

2) _" Appears that Ofqual's algorithm caused today's A-level chaos. Ofqual
chair Roger Taylor, also chairs the Centre for Data Ethics & Innovation
(CDEI). Cummings' fave AI consultants - Faculty, have some juicy contracts
with CDEI. And Faculty's COO Richard Sargeant is on CDEI board"_

[https://twitter.com/milesking10/status/1293886007771893762?s...](https://twitter.com/milesking10/status/1293886007771893762?s=21)

(Ofqual is the department that oversees exams and responsible for this
situation, they report to Gavin Williams the government's Education Secretary.
Cummings is a controversial special adverse to the Prime Minister, and Faculty
is a firm he used when he was leading one of the two primary pro-Brexit
campaigns before the referendum. Subjective statement: Cummings is a vile,
evil man.)

~~~
mjburgess
> it's hard not to be sceptical that this wasn't intentional and class-
> motivated.

It's incredibly easy "not to be sceptical".

The teacher's predicted grades are, on average, _unprecedented_. And represent
many times greater than any previous yoy increase. This is far less true at
top-performing schools where teachers are predicting far more consistent
grades (by the very nature of being a top performing school).

Teachers shouldn't have been asked to predict grades. That's the heart of the
problem. Not some harebrained in-public conspiracy.

This whole thing is dumb. And the media are deliberately failing to explain
basic facts about the education system to provoke this kind of nonesense
outrage.

~~~
corin_
I completely agree that the system put in place is poor and that it causing an
accidental bias is very feasible.

But if you're going to put this system in place, with months of planning, you
should make damn sure that it isn't massively favouring the children who are
already advantaged in life. Which it statistically is, whether intentional or
not.

They screwed it up so spectacularly that the education secretary had to make
concessions to it two days before the results, then a day after the results
Ofqual made a statement about who could appeal that contradicted the education
secretary, before withdrawing that statement a few hours later saying they
were still deciding.

So sure it could be entirely down to incompetence not malice. But given either
is possible, and this government's track record of cronyism I don't see why
it's unreasonable to be sceptical.

~~~
mjburgess
It doesnt "favour" any children.

Those children were already going to get the highest results.

There is no incompetence here. The predicted grades are a good estimate of
what the results would have been.

The issue is that all teachers significantly over-predicted their students
grades, but less so at high-performing schools. (Why? because the grades are
more consistent at those schools, so there's less room for error).

~~~
corin_
Independant schools (private schools) received 4.7% more A/A* grades than they
did in 2019. The most common type of non-private school, Secondary
Comprehensive, received only 2% more A/A* grades than in 2019.

The algorithm didn't keep the gap between paid and unpaid education, it
expanded it.

(Source for these numbers and more in my first comment.)

I was lucky enough to not only go to two private schools, but the second one
was ranked #1 in England for exam ranks 2 of the 3 years I was there (and
helped me get good grades despite being extremely unmotivated and lazy). But
at least my privilege was to be given a good education that made learning /
passing exams (the latter more) easy, not that the government decided my
school deserved an artificial grades boost compared to state schools.

~~~
mjburgess
Right, and what was the previous yoy disparity?

If the previous YoY was also 5%/2%, then it is just repeating the trend.

It isnt the job of statisticians to "fix" social inequality by BSing some
grades. Its their job to estimate likely exam results based on previous
performance.

There's absolutely no grounds on which to assume that growth in A*/A across
these demographics would be the same; and of course, very many reasons to
presume they'd be different.

~~~
ppod
You're making very strong statements over a huge assumption that you have no
data to back up. This is not like previous years. The students sat no exam,
they were ranked within their school, and then assigned grades based on the
previous performance of their school. It is reverse affirmative action.

~~~
peteretep
What are the strong statements you think he’s making?

------
Diggsey
There's no good answer here, the unfairness is an inevitable side effect of
lockdown, along with people dying due to stopped cancer treatments and a host
of other issues.

Using teacher's predicted grades is stupid: teachers use predictions in
whatever way they think will best motivate their students. If the student is
lazy, they'll under-predict to shock them into revising more. If the student
is not very confident, they'll over-predict to encourage them.

Unless the students are actually allowed to take an exam, all methods are
unfair. Using mock results is at least tangentially related to the students'
abilities.

~~~
nelaboras
You are under the mistaken assumption that somehow exams are more fair or get
a better view of what students can do. It's well established that exams
measure a few limited congnitive constructs and its not the smartest or most
hard working students that perform best, but rather those who are prepared by
teachers for those exact constructs and the precise way of answering them. A
brilliant student at a public school might study hard and still not get as
good results as a mediocre and lazy student at a private school which tells
them and drills them on the exam topics (often having their teachers in the
exam boards and anyway paying more so that those teachers that care and know
how yo play the game ate likely to be recruited there).

It's absurd to think a central exam gives a more accurate picture of... What?
Aptitude? Effort? ...? Than the teachers can have from working with students
often for several years, seeing their work ethics, involvement in class,
personal situations, etc.

In short: the system was broken and biased to begin with (at least in England,
Scotland is doing much better). Now this bias and favour of 'better' schools
is explicitly enforced through this weighting algorithm.

~~~
harry8
Exams, being far, far from perfect /ARE/ more fair. Why? because it is the
same exam for everyone. The person marking the exam has no information about
the person who wrote the paper, not their name, not what they look like, not
how rich they are, not how pretty they are, not what school they attend.

Whether or not you can achieve better outcomes or get a better indication of
capability with other means, that total anonymity and an absolutely common
hurdle is a kind of fair that is tough to match in assessment. It shouldn't be
under-vauled. We should be sure that any alternative really is better.

Let's not pretend that teachers' aren't subject to unconscious bias. I've also
met a few who were just horrible human beings who had no business being in
teaching.

~~~
kbut
Not what school they attend: Except they dont mark their own schools and i
think they keep the schools papers together in bundles. Plus markers have been
to open days and continuous development days with the other teachers. plus
they are having meetings daily to discuss how to mark classes of answers
(where teaching methods used come up). i think the net of this is that they
often do know what school they are marking (especially for the outliers - best
and worst schools). For some subjects e.g. computer science then the pupils
will answer in the langiages they were taught, so if its a python answer you
know its one of the schools that teaches python)

There is kind of just a problem overall with comparing private schools to
state schools. At the very least this process may have worked better if they
had two data sets, one for private one for atate schools.

Beyond that though a stastical solution was never going to work, you cannot
generate accurate discrete/quantized data, only representative random data.

private schools imho teach you to pass the exam, they are more cut throat
about focussing on this than state schools. and the teachers seem to fight
harder for their kids - the difference in personal statements from private
school teachers compared to state schools on uni applications would be an
interesting study.

I am not saying i have answers, just intereted in thw discussion. Overall i am
always left with a few basic ideas to keep things fair (though i sometimes
think some of these may be dumb and i am not suggesting do all of them):

-Zero charitable status or governemnt support for private education.

-cap the number of students a uni can take from one school (e.g. dont take the top 100 students, take the top 1 from 100 schools)

-ban people who were not state educated from governemnt positions. This is contravertial but imho you effectively live a different life when you go to privare schools (class sizes, support and oppourtunity are in comparable)

------
gwern
"In Scotland the accusations of unfairness prompted a switch to using
teachers' predicted grades. These predictions were collected in England too -
but were discounted as being the deciding factor, because they were so
generous that it would have meant a huge increase in top grades, up to 38%.
There were also doubts about the consistency and fairness of predictions and
whether the cautious and realistic could have lost out to the ambitiously
optimistic."

As always, the question when using an algorithm or test is, 'compared to
what?'

------
krisoft
This was so foreseeable when they anounced the plans.

One possible solution could have been if they do the same statistical
estimation game and then offer a choice to each student: you can take your
estimated grades or go for an exam. If you take the exam you get what you het
there. It might go lower or higher than the estimate. This would have
incentivised that only those who are certain of themselves and their estimated
grades are insuficient towards their goals will go for the tests. And since
these are outliers already there will be a lot less of them than the general
population, thus it is easier to organise socially distanced exams for them.

Of course nothing is easy at scale and many details need to be ironed out but
thats what we keep the Miniszry of Education for.

~~~
thom
The whole point is that it hasn’t been possible to sit exams.

~~~
Mirioron
Why not? Is it truly so impossible to spread students out a little more? Maybe
don't do oral exams, but anything written shouldn't be an issue.

------
lordnacho
I've been listening to talk radio here in the UK, and it's a constant stream
of disappointed kids/teachers/parents phoning in. Some of the stories are
horrendous, people expecting AAA getting CCD and similar. It seems unlikely
that you'd ever have three teachers predicting you to get top marks, but you
end up with that on results day.

But about the algorithm, it seems to be an impossible task. You cannot satisfy
the requirement of the gross statistics being roughly the same, ie little
grade inflation on average, while also identifying talented individuals.

The outline of the algorithm seems to be that you basically take the gross
stats for earlier years from a school, and then you take the teacher's guesses
about who will do well, and make up a distribution of grades around that. This
is probably what most people would do if forced into such an exercise. And you
can add wrinkles to it like a bit of grade inflation, adjustment for
overconfident predictions, and so on.

But you can't get that piece of information that you really need, which is a
mix of signal and noise about the true distribution of the kid's abilities. If
you have a talented year in a low performing school, you'll never know. And
vice versa.

The year before I finished the IB, two kids got the top 45/45\. That's
normally achieved by something like 1/400 kids worldwide, and it was a class
of maybe 30 kids. That would never have been awarded without examinations.

In addition the other presumed goal, of having the grades as close to the
predictions as possible, give the algo designer an incentive well known from
machine learning: guess safely. Squish the distribution on the edges, that way
you're not off much. This is bad for people who shoud be getting As, because
many will get Bs. And it's good on the other end, though I don't know where
the lump ends (C or D?).

------
naruciakk
Couldn't the UK just have a regular exam, just with the social distancing
measures implemented? The exams were held in many European countries with
these measures and nothing bad happened and the children has received a fair
examination for the University

~~~
remus
Presumably it'd be very difficult to set such an exam because pupils will have
missed so much of the normal curriculum.

~~~
sukilot
Why? High schoolers can read books and video chat.

~~~
onion2k
Children from poor backgrounds can't video chat because they don't have access
to computers. So that's nonsense.

~~~
toyg
So instead their grades got nuked for a lifetime. Hurrah!

------
nelaboras
An obvious point not discussed is that the exams system was already biased to
begin with. Previous years were already unfair as great kids at bad schools
have much fewer chances than less great kids for which the parents can pay a
great school. England especially is awful at achieving a fair system (that
gives everyone a fair chance at good grades) because schools are so stratified
- by design of funding rules, etc.

~~~
throwaway_pdp09
You're stating the obvious, a very well-known obvious. Perhaps you can suggest
any system that is not intrinsically unfair (I'd be interested as I went both
to schools in sink areas, and an expensive private school). (this was in the
UK BTW)

> ...is awful at achieving a fair system [...] because schools are so
> stratified - by design of funding rules etc.

I'd say it was more down to the culture of parents in the poor areas mostly
not giving a shit about their kid's education (and the roots of that aren't
that the parents are intrinsically bad, it's a mindset of apathy inherited
from their parents and the daily pressures that come with permanently worrying
about money).

~~~
CaptainZapp
I'm not surprised that you're downvoted and may be into oblivion. But you
actually make a very valid point.

That said I also believe that the current UK government is far beyond any
benefit of a doubt in terms of competence and maliciousnes.

~~~
throwaway_pdp09
I'd prefer constructive comebacks from people rather than downvotes. I can't
learn from downvotes.

But OMG the competence of the UK govt. It's like they literally can't do...
anything without screwing it up. It is bizarre. Then they try to paper over it
with words.

~~~
CaptainZapp
I think it's mostly due to this statement:

"I'd say it was more down to the culture of parents in the poor areas mostly
not giving a shit about their kid's education"

Which sounds nasty, but rings true for a big part of England. It's also an
uncomfortable truth, which basically always leads to downvotes.

Once a year (safe for this year, obviously) I'm in Blackpool at the Rebellion
Punk Festival[1]. What you see in Blackpool and how people live there is very,
very different from the impression you get when just visiting London.

People wouldn't believe how poor and destitute some areas in the UK are and
the priority of a lot of people in such areas is not really education. Young
women want as many kids as possible, since government benefits beat the shitty
work prospects they have in such areas and the future for young people in
general in those places are so bleak that it often leads to an understandable
"why bother?" attitude.

And yes, this attitude is inherited, or actually worse. When there is a rare
young person trying to be an achiever you can almost be sure he's mocked and
bullied by his peers.

[1] [http://www.rebellionfestivals.com/](http://www.rebellionfestivals.com/)

~~~
throwaway_pdp09
That is a most excellent response, and I agree with everything you say,
including how and why that bit comes across nastily. Ta for that, I really
must learn to be less of a tosser.

> rare young person trying to be an achiever you can almost be sure he's
> mocked and bullied by his peers

Yes. Someone on the web (maybe here, maybe slashdot) said that when someone
tries, they're seen as trying to get above their station. Consequences follow.

I spent a couple of years on a sink estate in the UK as a kid. I know what it
does to people.

------
tqi
Are we to believe that teacher's predictions are less fraught with bias and
inequity?

([https://www.bbc.com/news/education-53776938](https://www.bbc.com/news/education-53776938))

------
Jabbles
Do teachers recommend a grade, or a score? If they only recommend grades, how
did the algorithm pick which students to change the grades for?

~~~
chongli
The algorithm took the teachers' recommended grades and used them only to rank
the students at that school. The actual grades were calculated based on past
years' performance for each school. So if you were the top performer at a low-
ranking school, your grade would be in line with top performers at that school
in previous years.

~~~
jarvist
No, teachers submitted both predicted grades, and in large cohort classes
(such as English and Maths), a rank ordering of the entire cohort.

~~~
glomph
They had to submit ranking even if the class size was very small.

------
HeavenFox
Can someone explain to me the significance of A-level grades? If I understand
Wikipedia correctly, students apply to universities using predicted grades
anyways, and receive offers that are conditioned on the final grades. How
about just make all offers unconditional? And if you are not going to
universities, do anyone actually care about your A-level grades?

~~~
corin_
At least a couple of places, including one of Oxford Uni's colleges, have
agreed to take anyone who was offered a conditional place, and there has been
a hashtag trending on Twitter of #HonourTheOffer

I expect many more to follow, so it's possible that the universities will
clear up an awful lot of that mess - but while some academics are saying yes
great bring them all in, others are strongly against the idea of expanding to
let more people in this year.

As to whether they're significant apart from for UCAS applications
(university) - I'm not too sure. Personally I wouldn't care much about them on
a CV of someone I was thinking of hiring in the best of years yet alone if I
knew their results came from 2020.

But with unemployment rising fast, and much worse soon once government
gradually removes support offered over the summer to keep people furloughed
(companies paid to have their staff not working but kept officially on the
payroll until they could get back to work), we've already heard stories in
some places or a shop or a pub advertising a job offer and getting 1000+
applications... and in a terrible job market, school exam grades could likely
be a decider for who gets called to interview for many working class jobs.
(They already are even outside a terrible job market, that just amplifies it.)

edit: Oh, and while I don't know what proportion of people doing A Levels
don't want to go to university (or want to go, but fail) - we're also a few
days away from this same situation happening with the GCSE exams, which pretty
much everyone legally has to do around age 16.

------
Stierlitz
“There have been two key pieces of information used to produce estimated
grades: how students have been ranked in ability and how well their school or
college has performed in exams in recent years.”

“there was no direct connection between an individual's prior achievement and
their predicted grade.”

Who wrote this algorithm and can we see it?

------
Magnets
[https://www.ucas.com/file/292726/download?token=wswAnzge](https://www.ucas.com/file/292726/download?token=wswAnzge)

"In 2019, 21% (31,220) of accepted 18 year old applicants met or exceeded
their predicted grades, a decrease of 3 percentage points. In addition, 43.2%
of accepted applicants had a difference of three or more A level grades – an
increase of 5 percentage points (7,190 applicants more) since 2018."

Teacher predicted grades are almost useless.

Now Scotland has caved to political pressure and wants to inflate grades

[https://www.bbc.co.uk/news/uk-scotland-scotland-
politics-537...](https://www.bbc.co.uk/news/uk-scotland-scotland-
politics-53723734)

"However, these grades, taken overall, would represent a significant
improvement on previous years - including a jump of 20 percentage points in
the pass rate for pupils from the most deprived areas."

"If the results had purely been based on the estimates from teachers, pass
rates at grades A-C would have increased by 10.4 percentage points for
National 5, by 14 percentage points for Higher and by 13.4 percentage points
for Advanced Higher.

These estimated results would have led to a higher annual change than had ever
been seen before in Scottish exam results if they had not been moderated by
the SQA."

[https://www.bbc.co.uk/news/uk-
scotland-53636296](https://www.bbc.co.uk/news/uk-scotland-53636296)

A really difficult situation when what you have to work from is already
inaccurate.

------
wiredfool
You can curve the scores on a meaningless exam and give the impression that
some have learned and some have not, or I guess, you can skip the exam
entirely and go to the end product.

Pirsig would have something to say here.

------
matthewheath
While these events are terrible, I am pleased that the decisions made by
Ofqual are open to judicial review—and that the Good Law Project are
considering issuing a claim to force Ofqual to fix this mess.

------
bananaface
[deleted]

~~~
pjc50
The A level is a two year course with assessment at the end. There are no
previously examined grades, only estimated and mock ones.
[https://www.aqa.org.uk/subjects/science/as-and-a-
level/chemi...](https://www.aqa.org.uk/subjects/science/as-and-a-
level/chemistry-7404-7405/scheme-of-assessment)

(Modular and coursework based A levels were popular under labour but reverted
by the Tories)

~~~
dkarp
Wow, I didn’t realise it changed.

What a horrible situation where a whole year group are reliant on their
teachers’ opinion of them and the success of other kids at the same school the
previous year. Neither of which these children have any direct control over.

~~~
mcguire
Eh, well, if they're the right sort of people, they'll catch it on the bounce.

------
pfortuny
Well, there were no exams proper. The algorithm gives a mark for the A-levels.

------
jansan
So is "controversial" the same as unfair or bad? Journalists often use
"controversial" to discredit something that is not in accordance to their
ideology. Controversial politicians (which politician isn't controversial?),
controversial university professors, controversial speeches, controversial
thoughts...

~~~
chongli
It's controversial in the sense that a bunch of people are protesting about
it. Whether it's actually unfair or bad is unknown.

Fairness is a complicated thing. A lot of it is based on feeling and relative
status, rather than some objective measure of well-being. One person might
find it deeply unfair that another person has a lot of money, relatively
speaking. But take that money away and redistribute it equally between the two
and now the latter person may find the situation unfair.

~~~
weego
It's very clearly unfair, I'm not sure what your example has to do with
anything.

No students performance should be a sum based on how previous students have
done at their school. That's literally reinforcing the postcode divide in
education quality that were supposed to turn a blind eye to.

My home town school has historically refused to even put children in for exams
if they didn't get at least a C grade in their mocks because they valued
league table position so highly (helps with investment and alumni donations
etc). That cynical and should-be-illegal approach clearly benefits their
students in this situation more than schools in the area that do right by
their children.

And more than any of that, this is the real world effects of the bullshit
algorithm peddlers have been pushing for the last few years. Algorithms can
solve any real world problem given the data. Well they can't. They can give
the proposition of an answer to the problem, but at costs to some that are too
high to bear.

~~~
oh_sigh
My high school had one person 3 years prior to my senior year get into an Ivy,
and then before that it was no one in recent memory. In my senior class, 6 of
us got into Ivies. I can only imagine what my life would be like if we only
got the "fair" allotment of zero Ivy spots in a system like this.

~~~
stordoff
Similar happened to me (UK) - my school went from performing under the
national average for the preceding three years to being notably above for my
year group, and no one could remember anyone going to Oxbridge, but there were
two of us in that year (the school barely even knew about the extra exams we
had to take - required for one, optional for the other - so the prep. was more
or less entirely self-taught).

------
macspoofing
A big pile of 'meh'. They are trying to make a statistical guess based on
priors because the pandemic prevented the exams from being written. They are
going to get things wrong. If they have an appeals process for things that are
completely out line (e.g. a passing student getting a failing grade), then
that's the best you can hope for given the circumstances.

In the big picture, the good students are going to get good markers, average
students are going to get average markets, and bad students will get bad
markets. Again, it is what it is.

~~~
alexanderskates
The problem is that the prior appears to be placed over the school, rather
than the individual -- ie if your school had a low proportion of high
achievers in previous years, students are finding their grades marked down,
almost regardless of their performance. This results in particularly high
grade disparity between independent and state schools. So it is not so much
the case that good students get good marks, bad students get bad marks, but
rather good schools get good marks, "bad" schools get bad marks.

~~~
macspoofing
Because mark inflation is a thing, and schools are different. There are
schools that will give out A+ and others a B, for the same academic
performance. School reputations don't change from year to year. So a school
that inflates marks one year, will more than likely inflate them the next
year. Are you suggesting that this should not be taken into consideration?
That you should simply trust the relative weight of grade (relative to all the
other schools) at face-value?

>"bad" schools get bad marks.

Why the quotes? There are schools that are at the bottom of academic rankings.
That's a fact of reality.

>students are finding their grades marked down, almost regardless of their
performance.

What numbers are we talking about here? No matter what algorithm or heuristic
they chose, some unfairness was going to happen and my argument is that these
cases are overstated. I'm sure they exist because we're talking about hundreds
of thousands of students all in different circumstances. Having said that, in
cases of egregious outcome, an appeals process would make sense. Also I'm sure
many universities will take the pandemic into consideration and the fact that
this was a best attempt at replicating standardizing test result without a
test actually being taken.

~~~
pjc50
Per other thread, A levels are marked outside the school by the qualifications
body.

~~~
macspoofing
Right. A levels were not written so a statistical model was created to
approximate A level results based on prior performance of applicants
correlated with their school and course marks. Yeah, I get all that. It
changes nothing about my argument.

