
A-Levels: The Model is not the Student - tosh
http://thaines.com/post/alevels2020
======
7952
I don't understand why they even _need_ to gives these kids a single grade.
The biggest consumer of grading are schools, colleges and university anyway.
Just make more of the underlying data available to them and let the decide for
themselves. And as an employer I think it would be useful to have a more
complete picture of the student than just a single grade. All this fiasco does
is demonstrate how unnecessarily reductive the whole concept is.

~~~
michaelt
_> And as an employer I think it would be useful to have a more complete
picture of the student than just a single grade._

The current system isn't coping well with COVID-19 but let's not throw the
baby out with the bathwater here.

The current system is simple, and that means it can be seen to be fair. If
studying X at university Y needs grades AAB that's easy to understand and you
know it's not biased.

I much prefer that to the American system where every university has an opaque
system that purports to look at a complete picture of the student then
_mysteriously_ picks white students over Asian students with better grades and
can't explain why.

~~~
maxehmookau
> The current system is simple, and that means it can be seen to be fair. If
> studying X at university Y needs grades AAB that's easy to understand and
> you know it's not biased.

If only it were that simple. Universities often also take in to account the
outcome of in-person interviews which are inherantly biased.

~~~
StevenWaterman
Very few universities still do in-person interviews because of that bias. Only
Oxford and Cambridge require them. UCL and Imperial offer them optionally.

Some courses like Medicine do have interviews at institutions that usually
don't interview.

~~~
toyg
“Only” the two most famous (and arguably best, in a number of fields)
institutions. They set the tone.

~~~
smcl
They're certainly a good indicator that the class system is alive and well in
the UK, but since the rest of the universities rely on the Highers/A-Level
results without any interviews I'm not sure what you are implying by "set the
tone"

------
lordnacho
I've been hearing loads of reports of kids getting marks way off what was
predicted, and clearly the algorithm is... wrong. This essay is a good
explanation of why. One thing that annoys me a bit is that we as a society
have discovered all sorts of interesting thing about statistics and decision
theory, causality, etc, but then we don't apply them. Why didn't Ofqual find
some people who understood how stats work, and and ask them for advice?

It seems clear that if you just take a distribution from previous years and
lay them on top of a ranking, you are creating a lot of problems. And you are
hiding your incentives on the penalty function, which is something he
mentioned. People have correctly figured out that a good kid at a bad school
will not be found, and that the marginally second best kid in a year is going
to get marked down.

Whatever you think of exams, and I'm no fan, they do give people that mix of
noise and signal that society needs to put to rest the issue of who deserves
what. They're not entirely a lottery, not entirely skill or hard work, but if
we need a way to distribute X number of future doctors, that's the long
established way to do it.

~~~
teamonkey
> Why didn't Ofqual find some people who understood how stats work, and and
> ask them for advice?

Because, fundamentally, the current UK government is not interested in
scientific evidence or taking advice from experts in an area (c.f. Michael
Gove's famous line "people of this country have had enough of experts")

~~~
throwaway45349
At least include the full quote, which is far more reasonable in-context:

"I think the people of this country have had enough of experts with
organisations with acronyms saying that they know what is best and getting it
consistently wrong."

~~~
halostatue
Wouldn’t that apply to the Tories, though? They _consistently_ get it wrong
after saying they know best.

~~~
Smaug123
But they don't profess to be experts.

~~~
mhh__
Hence Dominic Cummings?

~~~
Smaug123
Cummings explicitly claims _not_ to be an expert, and decries the fact that he
can't seem to find anyone competent at the higher levels of the civil service
anywhere.

------
DanBC
It's useful to read some examples to see how devastating this is to children,
and how unfair it is that high achieving young people who have done well all
throughout their education have suddenly been downgraded.

[https://twitter.com/lewis_goodall/status/1295069110829752323...](https://twitter.com/lewis_goodall/status/1295069110829752323?s=20)

And this describes the advantages that private schools got:
[https://twitter.com/hugorifkind/status/1295292925463724032?s...](https://twitter.com/hugorifkind/status/1295292925463724032?s=20)

------
duaoebg
I’m no fan of ham fisted normalization.... but I think the bigger issue is the
fairness aspect of thresholding based on class sizes.

I think it should be possible to create ‘virtual classes’ above the threshold
by combining smaller like classes together.

Such a model would be fairly ham fisted.

~~~
truculent
This is not merely "ham-fisted normalisation". In many cases, the student's
attainment is being completely ignored!

> but I think the bigger issue is

The tricky thing about this scandal is that it is actually about 8 different
catastrophes for students all enmeshed together. This makes it hard to know
where to start critisising the government.

------
DonaldFisk
"Gavin Williamson and Ofqual have apologised to students and their parents, as
they announced that all A-level and GCSE results in England will [now] be
based on teacher-assessed grades."
([https://www.theguardian.com/education/2020/aug/17/a-levels-g...](https://www.theguardian.com/education/2020/aug/17/a-levels-
gcse-results-england-based-teacher-assessments-government-u-turn))

------
LatteLazy
People should note that even in non-covid times, UK domestic university
admissions is almost entirely based on predicted grades. Predicted grades are
much more important than actual grades. The only difference this year is an
extra layer of noise in the form of an algorithm. It's a very strange, unfair,
socially regressive system (I may be a little bitter).

~~~
Nursie
Unless things have changed in the last 24 years * , offers are based on
predicted grades, and requirements are set on the actual grades achieved in
order for that offer to transform into a place. Those students who did not
achieve the requirements and those universities who still have places to fill
then enter the "clearing" process to match up.

Were you predicted worse grades than you achieved, and as a result not offered
places where you would have liked?

( * they may have, significantly, it's been a long time)

~~~
mytailorisrich
There are a lot of unconditional offers these days, which means that the
university will not look at actual A levels' results at all.

Edit:

" _38% of applicants (97,045 applicants) received at least one offer with an
unconditional component in 2019, increasing from 34% (87,540 applicants) in
2018, and continuing the year-on-year growth in offers with an unconditional
component since 2013._ " [1]

This year at least one Oxford College has already decided to honour all offers
without looking at actual A-Level results , whereby effectively turning all
offers into unconditional offers [1]. Granted, this is not a normal year.

Let's be honest, though, if you got an offer from Oxbridge you're very good so
they're not exactly taking a risk and it gets them brownie points.

[1] (PDF)
[https://www.ucas.com/file/250931/download?token=R8Nn7uoI#:~:...](https://www.ucas.com/file/250931/download?token=R8Nn7uoI#:~:text=with%20an%20unconditional-,component.,an%20unconditional%20component%20since%202013).

[2]
[https://www.theguardian.com/education/2020/aug/15/a-levels-r...](https://www.theguardian.com/education/2020/aug/15/a-levels-
row-oxford-college-to-honour-all-offers-despite-results)

~~~
edh649
A common trick the universities use is to tell candidates that they'll upgrade
them from a conditional (e.g. AAB offer) to an unconditional offer if they
make that university their firm choice within X days.

I believe it's a way for them to better predict how many students they'll have
next year, otherwise it can fluctuate pretty wildly based on exam day results.

------
neilwilson
Notice how Ofqual can write a 319 page "it's not our fault" report, but can't
seem to publish their code on Github.

That explains more than anything what the problem is with the British Civil
Service. The epitome of "nobody every got fired for buying IBM".

~~~
dynamite-ready
The civil service? Pretty sure this is Dominic Cummings overriding the civil
service. But there's plenty enough wrong in this story to avoid invoking his
name.

------
mytailorisrich
There are plenty of data available on each students. The issue is how to
normalise them in order to create a national ranking.

Mock exams may be useful because, arguably, the results should already be
pretty normalised, but I'm not sure if these took place this year.

So what is left is to work with is the grades the students have had throughout
the year and previous years and, importantly, also what school they are in.

The latter is heavily charged politically so they are wary of communicating on
how that impacts their model.

It seems to me, though, that pupils from top private schools are not
complaining too much about having their grades downgraded, so it seems that it
did not happen too much for them.

~~~
NLips
On the whole, students from private schools had significant grade inflation
over previous years, often because the predictions were used as-is. This left
fewer high grades "available" for state schools (in order to maintain the
whole-country numbers).

------
dynamite-ready
So apparently, the government are due to announce an official u-turn -
[https://www.thetimes.co.uk/article/results-to-follow-
teacher...](https://www.thetimes.co.uk/article/results-to-follow-teachers-
grades-in-exams-u-turn-05zp3jvl3) \- and grades will be based on teachers
predictions, which, according to the article in the OP's link, should have a
.7* correlation with official results from the past.

It's still going to prove controversial, but sounds much better than the
apparently woolly 40 - 75% accuracy for the OFQUAL algorithm mentioned in the
article.

------
maxehmookau
Everyone keeps referring to "The Algorithm" like a mythical beast.

"The Algorithm" must exist. Can we see it?

~~~
JonAtkinson
Yes. It's section 8.2 onwards of this document:

[https://assets.publishing.service.gov.uk/government/uploads/...](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/909368/6656-1_Awarding_GCSE__AS__A_level__advanced_extension_awards_and_extended_project_qualifications_in_summer_2020_-_interim_report.pdf)

~~~
maxehmookau
Amazing! I (wrongly) assumed it wasn't public.

------
mhh__
Update: they've decided the A level grades will be the predicted grades

~~~
unholiness
Source?

~~~
Roonerelli
[https://www.theguardian.com/education/2020/aug/17/a-levels-g...](https://www.theguardian.com/education/2020/aug/17/a-levels-
gcse-results-england-based-teacher-assessments-government-u-turn)

------
mhh__
The big thing that stands out is that there have been some glaring problems
like native speakers getting C's in their language at (say) German or French
A-level i.e. so they slacked in the mock tests.

It suggests that their dataset was either way to narrow or their regressions
far too simple

------
lisper
Can someone please explain this to me?

> Obviously it has gone catastrophically wrong

I have no idea what this is referring to. The author is obviously (!) assuming
some knowledge on the part of the reader that I do not possess. Can someone
please fill me in?

~~~
onei
Not sure what the equivalent is outside the UK, but A Levels are the exams at
the end of school and used to determine of you have the necessary grade for
university. If you don't get the required grades, you don't go to university.

The problem is that due to lockdown, no one sat the exams. So they kind of
guessed what the grades should be based on predicted grades from teachers or
the grades assigned by mock exams. However, those awarding the grades felt
that predicted grades were too optimistic and there is no standardisation for
mock exams between subjects at the same school, let alone multiple schools
across the country.

What the exam boards seem to have done (and I don't think they've admitted to
this) is moderate the grades based on past results from the school. However,
schools in poorer areas traditionally do worse, so students in those areas
lost anywhere from 1 to 3 grades, e.g. an A becomes a D. This includes
students that never had a D at any point in the past. So then the government
said students could appeal. Then they said exam grades would be adjusted back
to no lower than their predicted grades. Or they could resit them once
lockdown ended.

Meanwhile university places are filling up based on the first set of grades.
Some universities are doing better, notably one of the Oxford colleges who
decided to honour all offers made on the basis that they have no new
information to change their offer on. But those that didn't apply to these
universities have to go through clearing but still don't know what their
grades are. Universities need to confirm places for accomodation which is also
filling up fast.

To call it a catastrophe isn't far off the mark.

~~~
lisper
Thanks.

------
dang
I assume this recent thread is related:
[https://news.ycombinator.com/item?id=24179635](https://news.ycombinator.com/item?id=24179635)

------
Nursie
> The objections appear obvious, with the dropping of teacher predictions
> implicitly being...

Twice in the opening text it talks about teacher predictions being dropped,
but they aren't, the teachers ranked the students and this is used as part of
the input data. Just not the entire input data.

If it was the entire input data then we know that, miraculously, this would
have been the smartest set of kids to ever come through the system. So clearly
just going by teacher predictions is also wrong.

I'm sure there are other valid objections to what has happened, but this isn't
really one.

~~~
Normal_gaussian
A ranking is different to a prediction. The text notes that. The prediction is
dropped.

Arguably, a prediction depends on a ranking but contains less and more data.
Less data in that only part of the intrastudent rank is maintained, more in
that the ranks are tied to grades.

It would seem that you agree that teachers cannot be expected to produce fair
absolute grades, but will produce fairer rankings - which is supported by the
articles references.

In terms of the objection being obvious - it is obvious in a political sense
that people would quarrel with the prediction being dropped as it both favours
them and is more personal. Mathematically its presented as obvious that this
is preferable [1].

[1] personaly I object in principal to us estimating grades full stop. I'd
have preferred spending the last five months rewriting exams based on covered
material and then a mammoth effort at getting kids to sit them safely. However
that requires foresight and funding.

~~~
Nursie
> personaly I object in principal to us estimating grades full stop

It's clearly a massive kludge - those rankings are only estimates, lots of
kids would have performed better or worse than estimated, and effectively they
are being granted grades based on a whole load of factors other than their own
actual performance. I feel very sorry for them.

------
dmurray
> The very idea of fairly guessing a student's grade is problematic, and only
> acceptable if you forget we are discussing real people, with individuality,
> dreams and purpose. They would have certainly been happier sitting their
> exams, whatever it took to do so safely.

They almost certainly...wouldn't? The exams are stressful and hard work: both
high- and low-achieving students were celebrating when the standardised exams
got cancelled here in Ireland and I'm sure it was little different in the UK.

At best, those who think they'd get better grades in the exam would prefer it.
The ultimate use of these grades is zero-sum (there are only so many
university places) so every decision that hurts one student is helping someone
else.

It doesn't really matter how good the model is, it's an impossible task
politically. There will always be some people aggrieved by the new process and
they will always have plausible ways to spin it as racist or classist or
whatever else is necessary to fuel the outrage machine. As if the previous
setup was some kind of examplar of egalitarianism.

~~~
Joeri
If some students would prefer to take exams and some not, it seems obvious to
let only those students who want take the exams. Propose the model’s grades to
the student. If they accept it, you’re done, if they refuse then let them do
an exam. Likely 80% or more of students would accept the proposed grades and
it would make it doable to organize socially distanced exams for the rest.

~~~
avvt4avaw
There are exams taking place in the Autumn, so this is kind of what is
happening.

~~~
a_humean
They will have missed this cycle of admissions, and not all students have the
luxury of taking a gap year. The students have also missed a term of tuition,
and so have probably not covered the course material necessary to pass or do
well in autumn.

Children/adults in care (people without families that can support them) might
not be eligible for support for another year, and will often be expected to go
out and work to support themselves. Poor families often rely on the extra
financial support that ends with "finishing" education.

There are a lot of stories out there of children/young adults in exactly this
situtation that should be off to skilled work training or university that
might end up having their entire education wrecked over this despite being
able academically. And this is all with the context that private school
students often have not been subject to any "standardisation" process due to
their small class sizes, or have had their awards upgraded above predicted
grades.

We just need to give this year a free pass in much the same way as workers
have been given a free pass with the furlough scheme. We can live with one
year of inflated results.

~~~
jon-wood
Its a particularly bad time for students not being able to take a gap year as
well - traveling is pretty much of the cards for those who would have
otherwise have done that, and for those who would have spent the year working
they're being thrown into that amongst the highest unemployment rates in
decades, and up against people who have extensive experience doing the sort of
jobs they'd be applying for.

------
david_draco
Guessing one of two outcomes correctly 50% of the time is pretty poor.

Guessing one of five outcomes correctly 50% of the time is pretty good.

------
ancomsi
If remote learning is going to continue for the foreseeable future why don't
assessments adapt to that environment? Stop testing and maybe rely more on
projects and prepared speeches? Maybe continuous in-class real-time
questioning to see if the student is keeping up? Sounds obvious but I know for
a fact that it's not happening.

I've always thought those things were better than exams before remote learning
regardless and many school boards weigh them similarly in aggregate. Testing
was always a bit of a flawed model pandemic or not.

~~~
Nursie
An exam tests that you have absorbed and can apply the knowledge. In subjects
like mathematics I'm not sure what prepared speeches might give you, and
grading based on real-time, in-class questioning is going to be subject to
bias from the teachers.

The whole reason we have a controversy over this is that the teachers were
asked to give predicted grade to their pupils, and in aggregate their
predictions were improbably high compared to other recent year groups.

~~~
ancomsi
The system I'm suggesting which admittedly wasn't fleshed out in a single
paragraph is really a complete overhaul of assessments to fit remote teaching.
You'd have to actually craft real-time continuous assessments for each subject
that fit each subject, just like you have to do for exams anyway, but the
benefit is you do it in mass amounts making it easier to get a mean
performance compared to just 1 or 2 exams. How it currently works is that
these types of assessments are treated as second class citizens which i guess
was really my frustration.

Also these days I'd heavily debate your premise- i think exams are more of a
test of memory and being able to game the system especially in the UK where
you are given a test bank of past papers which are almost identical to exam
papers. I was a terrible student until 2 weeks prior to the final exam where I
grinded maybe 6 or 7 past papers and at least 3 out of 10 questions were
carbon copied from previous papers. I would not be able to approach a problem
that fit outside of the structure of the problems in the exam papers in the
real world.

~~~
Nursie
> You'd have to actually craft real-time continuous assessments for each
> subject that fit each subject

Which sounds like continuous, standardised, externally applied examination.
Leaves rather little room for individuality in teaching techniques or pace...

> How it currently works is that these types of assessments are treated as
> second class citizens which i guess was really my frustration.

In as much as these things exist at all, they are part of the teaching method
and aren't in any way standardised.

I'm not saying your proposal is wrong, but it would lead to a much more rigid
system.

~~~
TheOtherHobbes
It's essentially the same as the hiring problem.

You're trying to work out who is a good fit from very limited data. You can
run some practical tests - whiteboard, paper exam - but that doesn't take into
account daily variations in performance, and it disadvantages outliers who may
have very useful skills or character traits that don't tick the usual boxes.

The problem may be more that education is seen an industrial production line
with authoritarian control of outcomes than as a personal development tool.
Early apprenticeships and interning opportunities with real tasks to solve and
real challenges to face might be better at developing a broader range of
talents _in context_.

