One of the most effective exam techniques I have seen, which is uncommon in my country but maybe common elsewhere, is oral exams. I TA’d for a German professor who did them for his stat/ML course, and so got to sit in on all of them (I also took the course myself in an earlier year so had the experience as a student as well). The process was:
1. Give students a list of about 100 questions in advance, more than you could memorise. Some were simple like “write down the formula for X”, some were more complex like “derive the backprop update algorithm”.
2. Pick the first question difficulty based on the student’s assignment grades
3. If they go to it right, pick a harder question, otherwise an easier one. If you aren’t sure whether they really understand, interrogate them about the answer, ask follow up questions etc.
4. Choose a grade based on which questions they got right.
Firstly this was highly effective at making students actually learn the material because most were worried enough about embarrassing themselves in front of the professor that they prepared well. But also, it was extremely fair because it’s essentially impossible to cheat and fake what you know.
I suspect many professors would avoid this because it’s harder to justify the grades at the end to a third party. But if you record the exams, and the student is clearly failing to answer simple questions, it’s quite hard for them to argue they were treated unfairly.
Of all the written exams I’ve seen and taken, I’ve never seen a process as fair or effective as these oral exams.
> the student is clearly failing to answer simple questions, it’s quite hard for them to argue they were treated unfairly.
I know you didn't quite say that, but to be clear: Fairness isn't defined by someone being able to argue that they were treated unfairly. You still have room for bias in how you treat weaknesses in student answers, and few students are going to give perfect answers for everything. And even without any specific-to-the-student bias I'm going to be more consistent in grading the same question if I go through a stack of written answers en-bloc than if I grade replies with hours or days inbetween.
Randomized questions also has potential for bias that feels like it could be stronger than question selection for a written exam.
That's not to say there isn't value in doing oral exams, but they are not the solution for everything either.
Oral exams are awesome. The problem is they don't scale up to a high number of students. A teacher with hundreds of students literally cannot afford the time and effort to properly examine the knowledge of the students. So they make these multiple choice tests they can just apply to every student in parallel.
All of these problems with academia are rooted in the fact it's a mass education system designed to teach hundreds if not thousands of people all at once.
You’re right, it was a lot of effort on the part of the lecturer and it didn’t scale well as the course got more popular. I think it’s best suited to smaller, later year niche courses.
To back this up - I went through the oral exam experience on both sides (student and TA) with two different German professors and it worked incredibly well. We offered oral exams as a remedy for a few situations:
- suspected/known cheaters
- students who missed exams due to illness
- times in the course where we needed to get a good grip on whether our students understood the coursework the way we'd taught it
It was scary for a lot of students, especially the ones without great English, but it was always, in my view, incredibly fair.
edit: based on your comment history you're also in Australia - I have a feeling we might be talking about the same place :)
My experience is completely different. Due to the lack of record and a lot of both personal experience and trusted accounts I deem such exams completely untrustworthy.
The bias is both ways. Some teachers/examiners decide on the spot which questions to ask. Not all questions are the same. Some are easier and depending on the sympathy one can get one.
Sometimes due to lack of accountability I even experienced complete dummy exams - a reason for which I decided to drop off from one. In college there was known strategy around one exam to not take part in written one as it was hellishly hard but instead go for oral one, where exam consisted of simply showing up.
This goes other way around, too. E.g. powerful member of exam committee of exam I took long time ago commission was infamous for failing people if not bribed in a very specific manner (not cash to hand but require buying specific set of services from family member). I dismissed that as malicious rumors but it proved to be true. I wasn’t failed but was scored inadequately which slowed my career progress for next few years.
I was told by academic staff foolproof strategies for failing students on oral exams. Like provide quick series of questions from various areas of topic, when they stumble due to stress finish them off with few extra ones. Other - ask questions so deep and profound they’ll either omit the details or start argue on ambiguity which are both enough reasons to fail.
There was small scandal about PhD exam recordings (which are obligatory) being purged 10 days after the exam, even though one of failed student decided to fight the result. Due to lack of evidence the exam result was held valid.
It really depends what kind of person is doing the exam. I’ve been always told that if one wants to skip exam they should go for oral instead of written one. Currently many are recorded but as they are widely used there’s minuscule chance they’ll ever be reviewed, and I recommend reviewing any piece of video if you think it’s easy. Find 3 seconds on 90 minutes video where student hands envelopes to the committee.
People on generally aren’t nice and while there is a sentiment for the teachers they aren’t any more special than and other group. You get lazy, power hungry or simply malicious people there too. And thus written record is a safe two way.
1. Give students a list of about 100 questions in advance, more than you could memorise. Some were simple like “write down the formula for X”, some were more complex like “derive the backprop update algorithm”.
2. Pick the first question difficulty based on the student’s assignment grades
3. If they go to it right, pick a harder question, otherwise an easier one. If you aren’t sure whether they really understand, interrogate them about the answer, ask follow up questions etc.
4. Choose a grade based on which questions they got right.
Firstly this was highly effective at making students actually learn the material because most were worried enough about embarrassing themselves in front of the professor that they prepared well. But also, it was extremely fair because it’s essentially impossible to cheat and fake what you know.
I suspect many professors would avoid this because it’s harder to justify the grades at the end to a third party. But if you record the exams, and the student is clearly failing to answer simple questions, it’s quite hard for them to argue they were treated unfairly.
Of all the written exams I’ve seen and taken, I’ve never seen a process as fair or effective as these oral exams.