It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject.
AI is here to stay; new methods should be used to assess student performance.
I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with having 'calculators' in our pockets 24/7 now.
We need to accept that the world has changed; I only hope that we get to decide how society responds to that change together .. rather than have it forced upon us.
Written assay evaluation is not and has never been an effective evaluation. It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it. Think about that the next time you look at your student debt, it couldn’t even buy you 30min time per class individually with the teacher to evaluate your performance. Instead you had to waste more time on a written assignment so they could offload grading to a minimum wage assistent.
When I studied physics at Exeter University they still used the tutorial system and finals. Tutorials were held fortnightly; the tutorial groups were typically three or four students. There was no obligation to turn up to lectures or even tutorials. You just had to pass the end of year exams to be allowed to continue to the final. The class of degree that was awarded depended on the open note final exam and the report of the final year project. That report had to be defended orally. Previous years exam papers were available for study as well but the variety of questions that could be asked was so vast that it was rare that any questions were repeated in the finals.
It seems to me that this is pretty much immune to plagiarism as well as being much better for the student.
Fellow UK person - the style of exam that you describe is pretty hard to cheat unless you can find another person to go in your place. I think various institutions have tried digital invigilation but have had little success (and I think this is just a bad idea anyway).
However, you also mentioned a final project. You’d be shocked how much commissioning exists where people have their projects produced for them. I’m not talking an overly helpful study group, I mean straight up essay mills. Tools like ChatGPT make the bar for commissioning lower and cheaper. I don’t know how you can combat this and still have long-term projects like dissertations.
My final year project was a 120 page report of measurements of electron spin resonance together with the design of the experimental apparatus. I had to defend the design, conclusions (which I have long forgotten, it was in 1977), and justify the methods and calculation all orally to two academics.
I doubt that anyone could have produced a plausible report without actually doing the work. And to defend it one would have to understand the underlying physics and the work that was done. Plus I think my supervisor and the other two students who worked with me on the project would have remarked on my absence from the laboratory if I had simply bought the paper!
You can still have long term projects and dissertations so long as the degree is awarded for the defence of the dissertation rather than the dissertation itself; that is the student must demonstrate in a viva that they understand everything in the dissertation rather than merely regurgitate it.
I think that in your case you've correctly observed that it would be nearly impossible to commission or otherwise fake your particular dissertation/project because of its experimental nature, and that you were called to a viva.
There are certainly similar projects being completed by students every year, and doubtless those students are not cheaters, but for each dissertation like yours, there are probably 10 or more projects that are not collaborative and have no artefacts or supporting evidence other than a written report. Such projects are fairly easy to commission. For a reasonable price (potentially thousands of dollars) you can pay a poor research student in the same field as you to churn out a mid-tier dissertation. This can be detected with a viva, but the academics need to be very confident before accusing someone of cheating. More often than not, you can get away with it and just get a not great grade.
I think that in general the natural/formal sciences don't suffer nearly as much as social science and humanities do, simply because exams and labs tend to highlight irregularities, and cheaters are less likely to be drawn into "hard" fields. However, it still exists in every field.
That won't work in a tutorial system, the student will be quickly discovered to know nothing about the subject. And in open note finals, as in the Exeter Uni. Physics department of the 1970s, regurgitation of course material was of very limited utility because you were never asked for that kind of response. The quantum mechanics final didn't ask a single question that had been directly answered during lectures, it asked us to extend what we had learnt. That exam was what I think Americans might call a 'white knuckle ride'. Open note finals really sort those who understood the subject from those who thought they could just look up the answers, the invigilators spent a lot of time shushing people searching through rucksacks full of notes.
Many years later I took a course in C# at a university in Norway and that was not merely open note but also open book (you could take the set book in). Again that gives the exam author the possibility to really discover who knows what.
I doubt that your rich middle-western student would have passed either of these
What about those of us who can explain our ideas and thinking clearly and in great detail in writing but would struggle to even prove we've heard of the topic orally?
These systems exist in no small part to train that ability, which is crucial to making it in the upper reaches of business and politics. The approach is probably also good for teaching the material, but training in speaking and arguing is more than just a side-effect of it—it’s part of the point.
Lots of elite prep schools in the US use a similar system, for similar reasons.
I'm not going to sugar coat it and it may sound harsh, but I doubt this is ever truly an issue outside of the minute edge cases.
Yes, there are people who have trouble with public speaking to a debilitating degree, but it would be excessively rare for someone to not at the very least in a one on one with their professor/teacher be able to be so badly affected as to not seem they've even heard of a topic or at least be able to prove they've worked on it to a certain degree.
I would be immediately skeptical of any student who claims they are completely unable to explain their knowledge unless they are allowed to work in complete isolation with nobody to monitor they aren't cheating in some way.
This is the kind of opinion that should be common sense but is highly controversial in the modern educational climate, for whatever reason. Probably the whole, "You can't judge a fish by its ability to climb a tree" quote being misapplied constantly.
(not the CP, but went to a university with a tutorial-style system)
I think the hard answer is that to some extent you just have to learn to. I mean, you could sit silently in supervisions if you really insisted, but to participate properly you just needed to build the confidence.
Is it fun? No, but it's a pretty accurate reflection of life after school: nobody in the real world gives you points for "couldn't say the right thing at the right time, but was thinking it"
In real life you need to be able to communicate written, in formal talks, and in informal discussions.
Those of you who severely lack any of the three will be penalized. Just like someone who can discuss a topic orally but could not write it up would be penalized.
you could ask for reasonable accommodations - e.g. if you have a recognised medical condition, or even just going through a rough time - e.g. ask to be allowed to write down your answer while they wait.
Yeah, the General Problems exam was a nightmare, I think the professors competed each year to come up with the toughest questions. Getting 50% was an excellent score.
It did force you to learn all the material though, especially as at the end of 3-4 years you may have forgotten some of it, like Optics or whatever. It was pretty hardcore though, especially compared to my friends studying other subjects.
I agree. There are small question about bias (gender, race) etc in these oral systems, but I think they are resolvable and much better than written essays (which are now written by AI).
In a written exam they can cover the names - give you a random number as you enter the room and you write that on the paper, and but your name and number on a different paper. You also need to type everything out on a computer with spell check. (and even then if you write bucket or pail will identify you but it is unlikely any professor knows you well enough to tell those)
When you audition for a symphony you perform behind a curtain and are required to wear soft slippers (so they can't tell if you are a wearing high heals - female).
We can probably use voice changers so the examiner cannot tell who you are by your voice, but those tend to be fatiguing.
If you can't trust a professor to professionally and impartially grade someone's work, the system would likely collapse. This is not to say that there hasn't been cases where professors have been shown to be biased, there has. But the premise of universities is to give professors some autonomy in the way they teach and evaluate students.
No system is 100% the question is are we good enough. As a white male I haven't seen many problems - but also because I'm in the group least likely to see one.
True but I think there's still an element of falsifiability to a teacher's evaluation of an essay that doesn't exist in an oral exam or interactive discussion. An essay is an artifact and if a teacher is giving student A worse grades than student B, a third party can look at that artifact to see whether it's remotely reasonable. A 1:1 discussion or an oral defense is much more subjective.
Not saying this is a fatal flaw, but there is a bit of a tradeoff there.
You can still do written essay evaluations. You could just require proctored exams whether or not you use software like Examsoft. If it's a topic that benefits from writing from a store of material, you can permit students to bring either unlimited supplemental printed material or a limited body of printed material into the exam room.
For longer essays, you can just build in an oral examination component. This face time requirement is just not that hard to include given that even in lecture hall style settings you can rely on graduate student TAs who do not really cost anything. The thing is that the universities don't want to change how they run things. Adjuncts in most subjects don't cost anything and graduate students don't cost anything. They earn less than e.g. backroom stocking workers. This is also why they, by and large, all perform so poorly. 30 minutes of examiner time costs maybe $11 or less. Even for a lecture class with 130 students, that's under $1,500. Big woop.
There are some small changes to grading practices that would make life very hard for AI cheaters, such as even cite checking a portion of citations in an essay. The real problem is that US universities are Soviet-style institutions in which gargantuan amounts of cash are dumped upon them and they pretend to work for it while paying the actual instructors nothing.
So get more TAs. They cost less than Class B CDL drivers and will drag themselves over broken glass to take these jobs. And 65 hours of work per semester for oral examinations seems entirely reasonable. A week and a half for an FTE spent observing, with another half week to full week for grading seems completely reasonable for capstone semester work.
There is truth to this perspective but it's also missing one of the fundamental purposes of writing essays in an educational setting. Writing essays isn't just about evaluation, it's also about teaching you how to think.
The process of reading textual material, thinking about it, and then producing more textual material about the stuff you just read (and maybe connecting it to other stuff that you've read in the past) is a critical way of developing thinking skills and refining your ability to communicate to an audience.
The value of that shouldn't be overlooked just like the value of basic numeracy shouldn't be overlooked because we all carry calculators.
You're right that it would be better if post secondary institutions would test people's ability to think in more ways than just what they can regurgitate onto a piece of paper, if only because that can be easily cheated but that doesn't mean that there isn't personal benefit in the experience of writing an essay.
I may not be the best writer but I am a better writer because I wrote essays in university, and I may not be great at math but I can reason and estimate about a variety of things because I have taken many math courses. These things have ultimately made me a better thinker and I am grateful to have had that imparted to me.
You're completely correct. Learning how to write taught me how to think, and researching and writing essays taught me what I believe about nearly everything on which I have strong opinions.
However, +90% of students will not now do any of that work. I got out of teaching (coincidentally) before LLMs appeared, and even then +80% of students did not experience that benefit of the essay process even with a grade (and plagiarism consequences) to motivate them. Now that decent-ish prose is a few keystrokes or Siri-led "chats" away, that's what they're going to do. That's what they're going to do.
I know of - I think it's up to four, now - former colleagues taking early retirement, or changing careers, rather than continue teaching Humanities in a world of LLMs.
All excellent point, but I'd like to add that it also forces you to do your own research the correct way, by surveying the current state of academic research and then finding and incorporating scholarly sources into your own arguments. Every academic essay I ever wrote after high school started with a trip to the library and JSTOR. I had to guide my own education instead of learning from the teacher and then repeating what had been taught.
> Written assay evaluation is not and has never been an effective evaluation.
I kind of disagree.
I've kept a blog for almost 20 years now and one thing is for sure: well-structured writing is very different from an oral exam the writing allows for restructuring your thoughts and ideas as you go and allows for far more depth.
I don't think, for most folks, that they could have as much depth in an F2F as they could in their writing with the exception of true experts in their fields.
The written essay has a cohesiveness and a structure to it that provides a better framework for evaluation and conveyance of information.
Well, that's not necessarily true. I was perhaps the most importunate student ever, and lingered around my professor's offices whenever they were open. I had endless questions, off topic and on. I was curious sure, but I was also annoying and pushy and wouldn't take no for an answer.
In fact, the only reason I use the word 'importunate' to describe myself, is because that's what my undergrad advisor called me.
So I at least was able to get well over 30m with each professor to discuss whatever I wanted. But likely that's b/c there wasn't a lot of competition.
The fact that one person can easily take a cup of water from a lake does not mean the lake supports for every person to take a cup. In fact, if everyone had tried to take that cup, then there wouldn’t even be a lake for the one person to take a cup from.
> It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it. Think about that the next time you look at your student debt, it couldn’t even buy you 30min time per class individually with the teacher to evaluate your performance.
Average student debt after a 4 year degree is ~$35,000 after ~45 courses. Before even running the math it should be obvious the gigantic cost of higher ed over 4 years is entirely unrelated to what an instructor would be making for ~23 hours of work (barring a secret society of multi millionaires). I.e. the problem you're identifying is the vast majority of $ spent in higher ed is not going to time with your professors, not that doing so is itself expensive.
> Written assay evaluation is not and has never been an effective evaluation.
Could not disagree more. Researching, formulating arguments, can give a student a complete view of the subject that studying for tests misses. But, similarly to tests, it probably depends on the skill of the teacher in creating the right kind of written assignments.
I'm not so sure that writing takes more time than studying. For starters, you don't have to memorize anything, and you can limit yourself to the assigned topic.
Of course, it can be that students don't take studying for an oral exam seriously, and trust the teacher to only ask superficial questions.
> It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it.
So the obvious solution is to make students to talk with an AI, which would grade their performance. Or, maybe the grading itself could be done by a minimum wage assistant, while AI would lead the discussion with a student.
It is, because I'm becoming tired with the current AI hype. It lasts too long to be funny.
OTOH, professor talking with a student is a good way to assess the academic performance of the student, but there are some caveats beyond costs. For example, professor will struggle to be an objective judge. Moreover even if they succeed, they would face accusations of discrimination in any case.
AI could solve this problem, but I'm not sure if AIs will be up to a task of leading the discussion. Though maybe if you try to assess students on their ability to catch AI on a hallucinated bullshit...
Actually deliberately introducing confidently delivered and reasonable sounding bullshit sounds like a fantastic way to suss out who knows their topic.
> It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject.
It never was. It's just even more ineffective now that AI exists, than before.
The central example of this is college admissions statements. Some kids have the advantage both of parents who can afford to give them the experiences that look good on such an essay (educational trips to Africa, lessons in two musical instruments, one-on-one golf coaching, that kind of thing), and who can hire tutors to "support" them in writing the essay. AI just makes the tutor part accessible/affordable for a wider segment of the population.
It would be naive to assume that, pre-AI, there was not a "gray" essay-coaching market as well as the "dark" essay-writing as a service market. That market still works better than AI in many cases.
It is not so black and white though: there is a difference between having your whole essay written by a tutor, or having some things corrected by the tutor, or the tutor giving you general tips that you yourself apply.
Just like there is a difference between having your whole essay written by a LLM, or having some things corrected by the LLM, or the LLM giving you general tips that you yourself apply.
Oh, I completely agree. In some cases, discussing a draft with your _university-appointed_ tutor before submitting your final essay is even part of the assignment (I believe Oxford/Cambridge humanities work this way), and a great learning experience, and a way for people who can't afford private tutors to get the same kind of coaching (how you get into this calibre of university in the first place notwithstanding).
> The central example of this is college admissions statements. Some kids have the advantage both of parents who can afford to give them the experiences that look good on such an essay (educational trips to Africa, lessons in two musical instruments, one-on-one golf coaching, that kind of thing), and who can hire tutors to "support" them in writing the essay.
This is an absolute disgrace. And then these are the people who lecture you on "inclusion".
You can decide whether it's evidence or not, but almost everything Freddie deBoer has ever written about college admissions points in that direction and I personally trust him on this topic, among other things because he presents scientific evidence that the SAT is much more reliable than some people make out to be, and is much more resistant to private tutoring. (One thing the SAT doesn't do is keep Asians out, however.) A search for "Freddie deBoer college admissions" with your favorite non-AI-shittified search engine should find lots of articles.
> I only hope that we get to decide how society responds to that change together .. rather than have it forced upon us.
That basically never happens and the outcome is the result of some sort of struggle. Usually just a peaceful one in the courts and legislatures and markets, but a struggle nonetheless.
> new methods should be used to assess student performance.
Such as? We need an answer now because students are being assessed now.
Return to the old "viva voce" exam? Still used for PhDs. But that doesn't scale at all. Perhaps we're going to have to accept that and aggressively ration higher education by the limited amount of time available for human-to-human evaluations.
Personally I think all this is unpredictable and destabilizing. If the AI advocates are right, which I don't think they are, they're going to eradicate most of the white collar jobs and academic specialties for which those people are being trained and evaluated.
> Such as? We need an answer now because students are being assessed now. Return to the old "viva voce" exam? Still used for PhDs. But that doesn't scale at all.
For a solution "now" to the cheating problem, regular exam conditions (on-site or remote proctoring) should still work more or less the same as they always have. I'd claim that the methods affected by LLMs are those that could already be circumvented by those with money or a smart relative to do the work for them.
Longer-term, I think higher-level courses/exams may benefit from focusing on what humans can do when permitted to use AI tools.
Yeah, LLM is kind of just making expensive cheats cheaper. You can do it without LLM, and indeed students did similar things prior to the release of ChatGPT, just less common.
> Such as? We need an answer now because students are being assessed now.
Two decades ago, when I was in engineering school, grades were 90% based on in-person, proctored, handwritten exams. So assignments had enough weight to be worth completing, but little enough that if someone cheated, it didn't really matter as the exam was the deciding factor.
> Return to the old "viva voce" exam? Still used for PhDs. But that doesn't scale at all.
What? Sure it does. Every extra full-time student at Central Methodist University (from the article) means an extra $27,480 per year in tuition.
It's absolutely, entirely scalable to provide a student taking ten courses with a 15-minute conversation with a professor per class when that student is paying twenty-seven thousand dollars.
I have 53 students in my class right now. A 15-minute oral exam works out to 13.25 hours of exam time, assuming perfect efficiency. As a comparison, our in-class time (3 hours over 16 weeks) works out to only about 48 hours. So a single oral exam works out to 1/4th of all class time.
But in principle this is not a problem for me, I already spend at least this much time grading papers, and an oral exam would be much more pleasant. The real problems will come up when (1) students are forced to schedule these 15-minute slots, and (2) they complain about the lack of time and non-objective grading rubric.
What PhD program doesn't require a public defense?
I'm currently a PhD candidate, and our program includes separate written and oral qualifying exams during the first year or two, and a public defense of the dissertation at the end. I thought some minor variation of this was nearly universal.
It's also my observation, by the way, that the public dissertation defense (and even the written dissertation itself) is less of a big deal than outsiders tend to think. What matters is doing the research that the advisor / committee wants, and working on some number of papers that get accepted into workshops / conferences / journals (depending on the field). Everything else seems to be kind of a check-the-box formality. By the time the committee agrees that someone has done enough to defend, it's pretty much a done deal.
Imagine Alan Turing's defense being a summary of 3 papers. The actual issue is that advanced education is increasingly not about doing fundamental scholarship but a pipeline for (re)producing a clerisy-intellectual class. There are a lot of leftist academics who point out this sea change in academia over the last century, see for example Norm Finkelstein's remarks on this but there are others who talk about this.
Oh yeah, there's a whole different discussion to be had (and HN does have it often), about the problems with peer reviewed publications and citations being the end-all for graduate students and professors.
My particular school and department is interesting because it doesn't have any hard requirement for publications, and it aims to have students finish a PhD in about three years of full-time work (assuming one enters the program with a relevant master's degree already in-hand). There has been some tension between the younger assistant professors (who are still fighting for tenure) and the older full professors (who got tenure in, say, the 1990s). In practice, the assistant professors expect to see their students publish (with the professors as co-authors, of course) and would strongly prefer to see a dissertation comprised of three papers stapled together, regardless of the what the school and department officially says. The full professors, on the other hand, seem to prefer something more like a monograph that is of "publishable" quality, maybe to be submitted somewhere after graduation. They argue that the assistant professors should be able to judge quality work for themselves instead of outsourcing it to anonymous reviewers. Clearly, there are different incentives at play.
Interestingly, in the UK strong student preferences against proctored exams and nervousness about how mental health issues interact with exams means universities are resisting dropping coursework, despite everyone knowing that most coursework is ai generated.
I think this varies dramatically from subject to subject. CS students at my university probably had overall 70% weighting on invigilated exams, but classics or business students probably had only 20% weighting and far fewer exams.
Oh yes. When I'm teaching a class of 200 students it's totally plausible that we're going to do 10 15 minute one on one conversations with every student. Because that's only 20 days non stop with no sleep.
We would need to increase the amount of teaching staff by well over 10x to do this. The costs would be astronomical.
I can't express how out of touch with reality this reply is.
The students paid me nothing. The university provides some TAs, that's it. But even if they gave me all of that money in cash to spend, this would be totally impossible.
I'm supposed to grade a student based on 1 conversation? Do you know how grading and teaching work? Can you imagine the complains that would come out of this process? How unfair it is to say that you have one 15 minute shot at a grade?
But fine, even if we say that I can grade someone based on 1 conversation. What am I supposed to ask during this 15 minute conversation? Because if I ask the every student the same thing, they'll just share the questions and we're back to being useless.
So now I need to prep unique questions for 200 people? Reading their background materials, projects, test results, and then thinking of questions? I need to do that and review it all before every session.
Even with a team of TAs this would be impossible.
But even if I do all of this. I spend hours per student to figure out what they did and know. I ask unique questions for 15 minutes so that we can talk without information leakage mattering. You know what the outcome will be? Everyone will complain that my questions to them were harder than those that I asked others. And we'll be in office hours with 200 people for weeks on end sorting this out and dealing with all the paperwork for the complaints.
This is just the beginning of the disaster that this idea would be.
It's easy to sit in the peanut gallery and say "Oh, wow, why didn't my arm surgery take 10 minutes, they just screwed two bones together right?" until you actually need to do the thing and you notice that it's far more complex than you thought.
Soviet professors were poor, so it was easy to bribe them to get passing grade. To weed out bribers, some trickery was used by state, so bribers can pay for few years or cheat on tests and then fail an exam anyway. In my class, 36 enrolled, 11 graduated.
Later, people learned that and started to buy diploma: faster, cheaper, no risk of failing the final exam.
Engineer from a regional institute received about 100 rubles, worker on a factory about 300 rubles (hehemon class), profesor up to 200 rubles, but profesors from top Moscow univs received 800-2000 rubles of hiden salary.
I heard something about 160 ruble a month student stipend. Although, maybe the parents were supplementing, but he said he had to pay them back for rent.
When they're paying 27k maybe they deserve a lower student to instructor ratio. And for that matter, a lower administration to student ratio. The whole system is very inefficient, there's a lot of room for improvement.
If I'm paying 30k$/yr the professor is damn well reading my essay. If they don't want to teach & grade, they can get a pure research position. Fun fact: pure research positions don't pay as well.
Roughly 50% of higher education occurs at community colleges. We don’t do research. What you pay for the class does not correspond to what I make. I’m not paid enough to do all the stuff that is suggested in the comments.
The top earning professors in the nation in mathematics are all very good research mathematicians
Fun fact: pure research positions don't pay as well.
Where do you get this from? The people I know with pure research positions get paid basically the same (after correcting for 'rank' and seniority) as those who split their time between research and teaching.
At least in the sciences, and in the US, there is also the issue that research professors tend to be on "soft money" -- that is they get a minimal salary from their institution but can increase it (up to a point) by getting grants that they can charge their time to. And they also tend not to be in the tenure track system. That being said, if they get large enough grants, they can make as much if not more than traditional tenure-track professors with defined salaries. But in years where they don't get much grant funding they don't make much at all (I used to be an non-tenure track research professor myself).
If their situation is that bad they can walk into a local staffing agency and get a factory job that pays 3x the federal minimum wage. Poor pay as a adjunct is a situation they choose for themselves for some reason.
I was an adjunct for a semester at a Big Ten university, many years ago. Like you say, there's usually a reason, such as collecting benefits while running some kind of side hustle. A teaching gig lends itself to this because the hours are flexible (outside of your scheduled class time), there is utterly no supervision, and no questions asked about what your other income sources are.
My office mate in engineering was trying to get funding for a start-up. I was trying to get a consulting business off the ground. Neither of us achieved those things, but whatever. He got a teaching gig at the community college, which is unionized and actually a pretty good situation. I found a regular day job through his network.
A friend of mine had an adjunct gig in the humanities, and used his off-time to learn how to code.
A lot of academic spouses get adjunct gigs, especially if they want to balance part time work with child care.
This is spot on! And that reason is peer pressure.
A lot of adjuncts sit around in precarious financial situations, developing serious mental health issues, and drinking problems because the system taught them that this is a form of success.
Going to industry and making money? That's failure. That's an "alternate career". Not scraping by in a system that couldn't care less about you. That's success.
It's pretty vile. I've never had a student become an adjunct. It would be a personal failure that I haven't given them the tools to thrive.
Pick 4 students per slot for oral examination and bring an assistant. That’s how my last exam worked. Assistant went through standard questionary and the main lector asked complex questions. The group of 50 was processed in a day with official grades and paperwork.
<< The grade was ultimately changed, but not before she received a strict warning: If her work was flagged again, the teacher would treat it the same way they would with plagiarism.
<< But that doesn't scale at all.
I realize that the level of effort for oral exam is greater for both parties involved. However, the fact it does not scale is largely irrelevant in my view. Either it evaluates something well or it does not.
And, since use of AI makes written exams almost impossible, this genuinely seems to be the only real test left.
> And, since use of AI makes written exams almost impossible
Isn't it easy to prevent students from using an AI if they are doing the exams in a big room? I mean when I was a student, most of my exams were written with just access to notes but no computers. Not that much resources needed to control that...
Good point. I agree, but it goes back to some level of unwillingness to do this the 'old way'.
That is not say there won't be cheaters ( they always are ), but that is what proctor is for. And no, I absolutely hated the online proctor version. I swore I will never touch that thing again. And this may be the answer, people need to exercise their free will a little more forcefully.
> Perhaps we're going to have to accept that and aggressively ration higher education by the limited amount of time available for human-to-human evaluations.
This will be it. [edit: for all education I mean, not just college] Computers are going to become a bigger part of education for the masses, for cost reasons, and elite education will continue to be performed pretty much entirely by humans.
We better hope computer learning systems get a lot better than they’ve been so far, because that’s the future for the masses in the expensive-labor developed world. Certainly in the US, anyway. Otherwise the gap in education quality between the haves and have nots is about to get even worse.
Public schools are already well on the way down that path, over the last few years, spurred by Covid and an increasingly-bad teacher shortage.
> Personally I think all this is unpredictable and destabilizing.
I completely agree, but then again it seems to me that society also functions according to many norms that were established due to historical context; and could / should be challenged and replaced.
Our education system was based on needs of the industrial revolution. Ditto, the structure of our working week.
My bet: We will see our working / waking lives shift before our eyes, in a manner that's comparable to watching an avalanche in the far distance. And (similarly to the avalanche metaphor) we'll likely have little ability to effect any change.
Fundamental questions like 'why do we work', 'what do we need' and 'what do we want' will be necessarily brought to the fore.
I think that we'll see fundamental changes, but it will be based on cheaper consumer goods because all of the back end white collar labor that adds costs to them will be (for all intents and purposes) free.
But we will see the absolute destruction of the middle class. This will be the death blow. The work week will change, but only because even more people will work multiple part time jobs. We'll think about what we need, but only because we'll have cheap consumer goods, but no ability to prepare for the future.
I think it's bleak. Source: most of human history. We're not, as a species, naturally altruistic. We're competitive and selfish.
>Fundamental questions like 'why do we work', 'what do we need' and 'what do we want' will be necessarily brought to the fore.
All the low paid, physically laborious work is not affected by AI, so there will be plenty of work, especially with aging populations around the world.
The question is will it be worth doing (can the recipients of the work pay enough) without being able to provide the dream of being able to obtain a desk job for one’s self or their children.
Historically that's more a question about community. Its a very recent phenomenon to have cultures where parents and grandparents are expected to take care of themselves or live in a home/facility.
Living in an elderly home may be impossible, too[1], meaning at best you can stay at the hospital until you die (which doctors are eager to achieve), at least in Hungary.
There is a sad, depressing world out there. One of my parents work at an elderly home, and the shit that happens there is just wild. Zero responsibility and accountability. Deliberate killing of people out of pure inconvenience, etc.
I am in favor of a "social support network".
[1] Requires money, e.g. pension, which is increasingly less, and they keep increasing the age.
Oh yes, I've heard my fair share of horror stories from elderly homes. I would like to say I'm glad they exist for those who have no other option, but even in the most expensive places I've personally seen its just no way to live in my opinion.
I agree, and it is terrifying that I may be paralyzed due to my chronic illness and end up rotting away on a hospital bed with nurses and doctors perceiving me as an inconvenience.
> All the low paid, physically laborious work is not affected by AI, so there will be plenty of work, especially with aging populations around the world.
Low paid physically labourorious work has been increasingly automated since the wheel started helping potters.
The history of computers often ends up referencing Jacquard looms because it used punched cards.
LLMs have been demonstrated controlling various robots.
AI's coming for all jobs, it's just a question of which jobs cease first.
> Such as? We need an answer now because students are being assessed now.
When I was in university (Humanities degree), we had to do lots of mandatory essays throughout the year but they counted little towards your overall mark, maybe 10% iirc.
The majority of marks came from mid-year & end-of-year exams.
A simple change to negate AI is to not award any points for work outside exams — make it an optional chance to get feedback from lecturers. If students want to turn in work by AI, it's up to them
> make it an optional chance to get feedback from lecturers.
My sense is that if writing was entirely optional, it would be too easy for students to let it slide; having 10% count toward your grade is a good "nudge" to get honest students to actually do the work. I'd just give a little admonition at the beginning saying that I'm not going to bother checking if you use LLMs, but if you do you're an idiot, because you'll be completely unprepared for the hand-written essay you have to compose yourself in the exams.
Yeah I think that's their rationale for doing it now alright but I mean rather getting into a big rigmarole of AI checkers & plagiarism checkers just trust students initiative to do the work, or not, and let the chips fall where they may come exam time.
University students should be capable enough for that, different at secondary level
Simple: you still write an essay and you may use ai to do so. Then you throw the essay out and go and talk with the teacher about it. If you can answer intelligently it’s because you know the stuff and if not then you don’t.
Such an increase can actually be quite feasible; quadrupling the labor spent on final examination would be perhaps a 10% increase for the total labor spent on preparing and teaching a university course, and at university level (unlike earlier schooling) we don't really have a shortage of educators, quite the opposite.
> Such as? We need an answer now because students are being assessed now.
My current best guess, is to hand the student stuff that was written by an LLM, and challenge them to find and correct its mistakes.
That's going to be what they do in their careers, unless the LLMs get so good they don't need to, in which case https://xkcd.com/810/ applies.
> Personally I think all this is unpredictable and destabilizing. If the AI advocates are right, which I don't think they are, they're going to eradicate most of the white collar jobs and academic specialties for which those people are being trained and evaluated.
Yup.
I hope the e/acc types are wrong, we're not ready.
> My current best guess, is to hand the student stuff that was written by an LLM, and challenge them to find and correct its mistakes.
Finding errors in a text is a useful exercise, but clearly a huge step down in terms of cognitive challenge from producing a high quality text from scratch. This isn't so much an alternative as it is just giving up on giving students intellectually challenging work.
> That's going to be what they do in their careers
I think this objection is not relevant. Calculators made pen-and-paper arithmetic on large numbers obsolete, but it turns out that the skills you build as a child doing pen-and-paper arithmetic are useful once you move on to more complex mathematics (that is, you learn the skill of executing a procedure on abstract symbols). Pen-and-paper arithmetic may be obsolete as a tool, but learning it is still useful. It's not easy to identify which "useless" skills are still useful as to learn as cognitive training, but I feel pretty confident that writing is one of them.
> Finding errors in a text is a useful exercise, but clearly a huge step down in terms of cognitive challenge from producing a high quality text from scratch.
I disagree.
I've been writing a novel now for… far too long, now. Trouble is, whenever I read it back, I don't like what I've done.
I could totally just ask an LLM to write one for me, but the hard part is figuring out what parts of those 109,000 words of mine sucked, much more so than writing them.
(I can also ask an LLM to copyedit for me, but that only goes so far before it gets confused and starts trying to tell me about something wildly different).
> It's not easy to identify which "useless" skills are still useful as to learn as cognitive training
Indeed. And you may also be correct that writing is one such skill even if only just to get the most out of an LLM.
What I'm describing here is very much a best guess from minimal evidence and the current situation; I would easily drop it for another idea if I saw even very minimal evidence for a better solution.
Effective Acceleration, the promotion of rapid AI development and roll out, appealing to all the deaths and suffering that can be prevented if we have the Singularity a year early.
Extremely optimistic about the benefits of new tech, downplay all the risks, my experience of self-identifying e/acc people has generally been that they assume AI alignment will happen by default or be solved in the marketplace… and specifically where I hope they're wrong, is that many seem to think this is all imminent, as in 3-5 years.
If they're right about everything else then we're all going to have a great time regardless of when it comes, but I don't see human nature being compatible with even just an LLM that can do a genuinely novel PhD's worth of research rather than "merely" explain it or assist with it (impressive though even those much easier targets are).
TYVM. Hopefully the inability to see ways this could go wrong or really look at the problem is sufficiently correlated with the lack of the tools required for progress.
> I remember being told at school, that we weren't allowed to use calculators in exams
I remember being told the same thing, but I happen to believe that it was a fantastic policy, with a lackluster explanation. The idea that you wouldn't have a calculator was obviously silly, even at the time, but underlying observation that relying on the calculator would rob you of the mental exercise the whole ordeal was supposed to be was accurate. The problem is that you can't explain to a room full of 12 year olds that math is actually beautiful and that the systems principles it imparts fundamentally shape how you view the world.
The same goes for essays. I hated writing essays, and I told myself all sort of weird copes about how I would never need to write an essay. The truth, that I've observed much later, is that structured thinking is exactly what the essay forced me to do. The essay was not a tool to asses my ability in a subject. It was a tool for me to learn. Writing the essay was part of the learning.
I think that's what a lot of this "kids don't need to calculate in their heads" misses. Being able to do the calculation was only ever part of the idea. Learning that you could learn how to do the calculation was at least as important.
It's actually not about beauty of the math, it's about something which is nowadays called a number sense. It takes a lot of practice to develop an understanding what these things called numbers are, how these relate to each other, what happens if you combine these with operational signs, how numbers grow and shrink etc. And you are damn right that there is no any use to explain it to the 12 year olds. Or even to 16 year olds.
Common Core was an attempt at teaching this directly. It gets so much ridicule because so few people have good enough number sense to recognize what they're seeing when shown a demonstration. Of course, since they didn't understand it, it then led to bad examples being created and shared, which just made it worse...
It didn't explain the goals well enough to parents, and many teachers didn't have the number sense themselves leading to many of the examples are passed around showing how the whole process is broken. There is also a question of if even works well, as it is somewhat akin to teaching someone the shortcut on how to do something before they have mastered the long way of doing it. Many experts in their fields have shortcuts, but they don't teach them directly to juniors in the field as there is value in learning how to do it the long hard way, as often times shortcuts are limited and only an understanding of the full process provides the knowledge of when best to apply different shortcuts.
> There is also a question of if even works well ...
No, it doesn't. And that's one of the main tragedies in modern discussion about education. A lot of people think that we have to teach the way how experts think. But it doesn't work.
Take a look at programming for example. Everything you ever really need to know about programming, can be described in single A4 probably. But it has no any use if you learn. It even makes things worse, because it's a thing you have to pay attention to, but you don't really understand. You have to learn via small babysteps, automate a lot of small things in your brain, practice a lot etc. There is no shortcuts in learning.
Very well put. I would actually suggest to not use calculators in high school anymore. They add very little value and if it is still the same as when I was in high school, it was a lot of remembering weird key combinations on a TI calculator. Simply make the arithmetic simple enough that a calculator isn't needed.
I don't remember exactly, but I think we were only allowed the simplest calculators in middle school (none before), and scientific calculators in high schools (mostly for the trigonometric and power functions). I got to use a TI in university, but never used it that much as I've got the basic function graphs memorized.
How old are you? And, for that matter, how old is the person you're responding to? In 1998, at least up to a TI-81 was allowed on the AP Calculus Exam (possibly higher than that, but you couldn't use anything that was programmable). I have to think it's been a very long time since no calculators at all were allowed for math exams unless you're talking arithmetic exams in elementary school where the entire point is to test how well you've memorized times tables or can perform manual long division.
It even came with handwriting built in as authentication mechanism! AI detectors hate this secret!
On a more serious note - US removed cursive from their curriculum almost two decades ago - something i cant wrap my head around as cursive is something the rest of the world(?) uses starting in middle school and onwards through the whole adult life.
21 states still mandate cursive in their curriculum.
There are lots of things I spent a lot of time learning school that I rarely use, but see the value in having learnt. Cursive, beyond a very basic level, is not one of those things.
Though I’m no education expert, perhaps there is a subliminal value to spending all that time.
I learned cursive in school and never used it. I have written countless essays by hand in grade school and college and never felt the need to do it in cursive because to me my cursive was just way more unreadable than my printing and not particularly faster.
I don't know what the rest of the world calls "cursive", but here in the US the cursive we get taught is strictly inferior: slower to write, less compact on the page, and harder to read (while also being strictly uglier than true calligraphy). It's a script designed for allowing you to avoid lifting a quill from the page and thereby avoiding ink blots; it's entirely obsolete.
I learned cursive, then reverted to print, but when I entered a phase of my life where I needed to write several pages a day I quickly went back to (a custom variant of) cursive because it was faster to write in a legible way than print.
When I rush print it quickly becomes illegible. When I rush my cursive it doesn't look quite as nice as it does when I'm writing steadily, but I can still read what I wrote ten years later.
From what I can tell it works because cursive letters are defined in a shape that lends itself to a quick moving pen. Once you learn that shape (both to write and read), you can quickly get words down on a page and then understand them later. If you just try to slur your print in an unprincipled way your letters distort in ways that make them harder to tell apart.
Now, I imagine someone could develop a slurred print that doesn't have connections between letters, but I'd probably call that a cursive anyway.
Yeah, the hand they taught us (in the early '90s) was some common one that I gather most places have taught in the US for decades. It never made any sense to me. Ugly, hard to read, and not even notably faster to write, even if you got good at it.
Later I found out it was developed for use with a fountain pen, designed with the idea that a correctly-faced nib would make some strokes bolder and others very faint, and to keep the nib always moving in a kind of flow to avoid spots, and to make it natural to keep the nib faced the correct way(s), plus with even more attention to avoiding raising the pen than most cursives, for similar reasons of avoiding spotting. That made all the downsides make sense—it's far less ugly and easier to read when written with a fountain pen, and may well be faster than many other similarly-clean methods of writing with one.
Why the hell we were still learning that hand decades into the dominance of the ballpoint, remains a question.
Write in cursive or in print, or even cut letter from a newspaper if you want. If you do it in a classroom in front of a teacher cheating is dramatically reduced
Dane here. We don't write things in cursive. Sure we were thought cursive in school (my only remedial class! What a waste of time), but we write on computers. Very occasionally we might need to write up a sign or something.
I did nearly all my exams on a computer.
At one point the best writing tool was the fountain pen. It was a great invention and it had an appropriate script: cursive, which was the natural thing to do given how the ink flowed.
However kids are messy and you really want them to use pencils because they don't have flowing ink. The reason for cursive in the first place was the flowing ink, so when we switched away from flowing ink, there was no reason to write in cursive.
Except of course to waste the only resource everybody agrees is okay to waste: kids time.
Lol, multiple people downvoted you for some reason. A lot of Americans have this weird love of joined up handwriting. I wonder if it's because so much of their cultural history is based around a handwritten document and the mythos of the "John Hancock"
The time would be far better spent on say personal finance lessons, which people seem to really struggle with. What's a mortgage, what's a loan, what gets taxed, that sort of thing.
Classroom time is limited, so if you include cursive writing in your curriculum you have to omit something else. Whatever you dropped to make room for handwriting is almost certainly going to be more useful to the students once they reach adulthood.
At the end of the day cursive writing is a hobby, not a skill. We don't need it anymore. It wastes priceless learning time at a critical juncture in our intellectual development.
My kids still have cursive in the curriculum (charter school but I believe the public schools in my district teach it too). Once my oldest hit 4th grade, all assignments had to be completed in cursive.
There's no particular reason people need to use cursive rather than printing. I personally always struggled with cursive due to eye-hand coordination issues and teachers' demand for it just seemed like hazing (and I'm a boomer). Good riddance to cursive.
Sorry but nobody uses "cursive" writing. People barely write once they leave education - they type. When they do write it's legible separate characters or it's unreadable scrawl.
My brother (mid-20s) still writes in cursive. He does so beautifully, completely legibly, and rapidly, at a faster rate than he, or the average person, can write non-cursive.
He's not the average person. His handwriting ability is both an impressive and useful skill, considering that he takes notes in a notebook, and can do so rapidly. Besides, your claim was that "no-one" uses cursive, which I simply was chiming in to point out isn't true.
This is mostly true, but it is also important to recognize that “hey just invent a new evaluation methodology” is a rough thing to ask people to do immediately. People are trying to figure it out in a way that works.
Sadly, this is not what is happening. Based on the article ( and personal experience ), it is clear that we tend to happily accept computer output as a pronouncement from the oracle itself.
It is new tech, but people do not treat it as such. They are not figuring it out. Its results are already being imposed. It is sheer luck that the individual in question choose to fight back. And even then it was only a partial victory:
"The grade was ultimately changed, but not before she received a strict warning: If her work was flagged again, the teacher would treat it the same way they would with plagiarism."
My first semester undergrad English course, the professor graded all my papers D or worse. Had to repeat the course with a different professor. They shared assignments so I re-used the same essays with zero modifications... but this time I got an A or higher!
I have a similar memory. I wrote an essay about a poem.
The poem was assigned to us, but for some reason the subject matter really chimed with me personally. I thought about it a lot, and—as a result—ended up writing a great essay.
Because I did well, I was accused of cheating in front the class.
An essay written under examination conditions is fine. We don’t need new assessment techniques. We have known how to asses that a student and that student alone for centuries.
A few of my high school teachers in the early 1990's made our final paper into a big project.
It was not just "turn in the paper at the end" but turn in your topic with a paragraph describing it. Then make an outline, then bibliography of the sources we were using. During the process we had to use 3x5 index cards with various points, arguments, facts, and the specific pages in the books listed in our bibliography. We did this because this was later used to make footnotes in our paper.
By structuring the project this way and having each milestone count as 5-10% of the overall grade it made it much harder to cheat and also taught us how to organize a research paper.
I suppose you could ask ChatGPT to do the entire paper and then work backwards picking out facts and making the outline etc.
No, but we had to write essays in class during exams.
There's a good question about the future and utility of long at-home research paper projects in school, but it's not a cornerstone of education.
In 9th grade I procrastinated the semestral paper so much that I bought an essay online that explored unexpected gay themes in Ray Bradbury's corpus of work. I was so lazy I didn't even read it first, only skimmed it, and then back to Runescape. So it's not like this is a new problem due to LLMs, and I think take-home semester projects are all quite bad for these reasons that predate LLMs.
(It turned out to be such a phenomenally audacious essay that my teacher started fascinated email correspondence with me about it and I was forced to not only study the essay but also read the quoted parts of his work. Ugh, backfire.)
Some of my experience of exams comes from a history degree where around eighty to ninety percent of the overall grade came from final exams. I can only speak of my experience but I don’t think this is atypical depending on educational system.
One of the reasons I mentioned the viva was an example of how we can decouple production of some work from an assessment of quality and some reason to believe that some candidate is capable of the work without assistance.
It would be unreasonable to spend five or so years working under examination conditions. But that doesn’t mean we can’t subsequently examine a candidate to determine likely authorship amongst other things.
We had in class essays in my history class in highschool. “Write everything you know about the triple entente” or something like that was often the prompt. You were merely expected to pay attention in class to pass not bring in outside research.
Yeah we always did that in high school for essays that were actually graded, otherwise there's always the option of having someone else write it for you, human or now machine. The only thing that's changed is the convenience of it.
The problem is more with teachers lazily slapping an essay on a topic as a goto homework to eat even more of the already limited students' time with busywork.
The lazy essay assignment is 100% real. However, the driving force there is not the teacher, but parental complaints causing ass-covering administrative mandates. "Why wasn't there any homework on topic X before the exam?" "We apologize so much for that, Mrs Keen. First, we will change Precious's grade, but from now on..."
My ability to write an essay under exam conditions is...poor. Thankfully there were less than a handful of essays I had to write as part of my undergraduate CS degree and I only remember one under exam conditions.
I think it's probably more concerning that spitting out the most generic mathematically formulaic bullshit on a subject is likely to get a decent mark. In that case what are we actually testing for?
Amusingly, willingness and capacity to conform to a system you are paying $30k a year for is a pretty good proxy for general intelligence. So maybe it's not that bad?!
It depends on what we think education is for. If the goal is to teach students, it's not so great. If the goal is to signal future employers the intelligence of the student, Maybe that's ok. But maybe the future employers should be paying the tuition instead of the student.
I won’t claim this is by design, but at the very least a side effect of writing term papers is getting practice at organizing your thoughts and drawing conclusions from them.
While writing term papers is a skill that is only minimally useful in the real world (save for grant writers and post docs, pretty much), the patterns of thinking it encourages are valuable to everything that isn’t ditch digging.
Maybe we can outsource this part of our cognition to AI, but I’m skeptical of the wisdom of doing so. Are we all going to break to consult ChatGPT in strategy meetings?
>AI is here to stay; new methods should be used to assess student performance.
Here is the brand new method - asking verbal questions in person and evaluating answers. Also allow high tech aides in the form of chalk and blackboard
The downside of downgrading technology like this is that tests and skills become less relevant to the real world.
For all their problems, 5000 word take home assignments in Microsoft Office have a lot in common with the activities of a junior management consultant, NGO writer, lawyer or business analyst. And same with for scientists but with Latex.
I’d rather hire a lawyer who could only do their job with AI than one who couldn’t use a computer to create documents or use digital tools to search case law.
Learning takes time. And the fully trained/educated/skilled/expert human performance is higher than AI performance. But AI performance may be higher than intermediate human performance after 1 or 2 semesters. But you need to reach intermediate performance first in order to later reach expert performance. During that time you still need a learning "slope", you need to be tested on your knowledge at that level. If you're given the AI at the outset, you will not develop the skill to surpass the AI performance.
Calculators are just one analogy, there is no guarantee it will work out that way. It's just as likely that this over-technologization of the classroom will go the way of whole-language reading education.
Was this ever effective? There was a lot of essay copy/pasting when I was in school, and this was when essays had to be hand written (in cursive, of course, using a fountain pen!).
Same with homework. If everyone has to solve the same 10 problems, divide and conquer saves everyone a lot of time.
Of course, you're only screwing yourself because you'll negatively impact your learning, but that's not something you can easily convince kids of.
In person oral exams (once you get over the fear factor) work best, with or without (proctored!) prep time.
Maybe it doesn't scale as well, but education is important enough not to always require maximal efficiency.
Depends on the homework, of course. In my head I guess I was talking about maths problems. Maths understanding, in my experience, greatly benefits from practice, and homework exercises might be useful there. Memorising the names of rivers ... maybe not so much.
The old colloquium exam format reigns supreme again. And that is fantastic. We shouldn’t reserve it for only “most important” occasions because quality education is important enough by itself.
We blasted through the “you won’t always have AI in your pocket” phase in a blink of an eye. Local LLMs were running on smartphones before the world got to terms of LLMs being used everywhere. It’s one of many examples of exponential technological advancement.
IMHO. With calculators introduced, there is zero add in you learning long division. Worse than zero, you could have done something better with your time. ChatGPT is a calculator for all subjects. People have a hard time letting that sink in.
Long (or short—screw long division, with its transcription error opportunities and huge amounts of paper-space used) division is a good exercise to cement the notion of place value, that happens to also teach you how to divide by hand for when it's occasionally more convenient than finding a phone/computer/calculator.
I still bet someone like you could pass any university exam in any subject with access to the ChatGPT app. Without any prep time. At that point it’s good enough.
On some exams in our university 20y ago, we were allowed to use any literature or lecture notes to answer the questions. The thing is, it was a high level abstract algebra. If you don't understand the subject, no amount of literature would help you to answer the questions correctly (unless you find the exact or a very similar question).
I believe it's still true today, but with future AI systems even highly abstract math is under the danger.
Just because a method of assessment became easily spoofable doesn't mean we should give up on it. Imagine if in the era before HTTPS we just said that the internet won't be really viable because it's impossible to communicate securely on it.
I still feel like AI detectors would work well if we have access to the exact model, output probabilities of tokens, We can just take a bit of given text, and calculate the cumulative probability that the AI would complete it exactly like that.
Probability is not an acceptable way to determine a student's future. They may have learned from the AI and remember some of the exact phrasing, and learned writing/language cues from it as well.
Agreed. I'm not a good writer, tending to stick to a somewhat abrupt, point-making structure almost better suited for bullet pointing. I've taken tips from other HN users on how to improve, but I have no doubt that had I been going through university these days, I'd probably be flagged too.
> we could never rely on having a calculator when we need it most—obviously there's irony associated with having 'calculators' in our pockets 24/7 now
That was just a simple quip to shut down student bellyaching. Even before we had pocket calculators, it was never a strong answer. It just had to hold over long enough so when you realized it was bad answer you weren't that teacher's problem anymore.
The actual answer was that they're complaining about a minor inconvenience designed for reinforcement, and if they really did need a calculator for the arithmetic on a test designed deliberately designed to be taken without a calculator, then they don't belong in that class.
The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements.
If the level of effort made is high, but the outcome does not comply in some way, praise is due. If the outcome complies, but the level of effort is low, there is no reason for praise (what are you praising? mere compliance?) and you must have set a wrong bar.
Not doing this fosters people with mental issues such as rejection anxiety, perfectionism, narcissism, defeatism, etc. If you got good grades at school with little actual effort and the constant praise for that formed your identity, you may be in for a bad time in adulthood.
Teacher’s job is to determine the appropriate bar, estimate the level of effort, and to help shape the effort applied in a way that it improves the skill in question and the more general meta skill of learning.
The issue of judging by the outcome is prevalent in some (or all) school systems, so we can say LLMs are mostly orthogonal to that.
However, even if that issue was addressed, in a number of skills the mere availability of ML-based generative tools makes it impossible to estimate the level of actual effort and to set the appropriate bar, and I do not see how it can be worked around. It’s yet another negative consequence of making the sacred process of producing an amalgamation of other people’s work—something we all do all the time; passing it through the lens of our consciousness is perhaps one of the core activities that make us human—to become available as a service.
Little Johnny who tried really hard but still can barely write a for loop doesn't deserve a place in a comp sci course ahead of little Timmy who for some reason thinks in computer code. Timmy might be a lazy arse but he's good at what he does and for minimal effort the outcomes are amazing. Johnny unfortunately just doesn't get it. He's wanted to be a programmer ever since he saw the movie Hackers but his brain just doesn't work that way. How to evaluate this situation? Ability or effort?
1. Whoever determined that he does not “deserve” this is wrong. There may be other constraints, but no one gets to frame it as “deserves” when a child wants to learn something.
2. If a teacher is unable to teach Johnny to write a for loop, despite Johnny’s genuine utmost motivation, I would question teacher’s competence or at least fit.
3. Like any mentor, a professor in higher ed may want to choose whom to teach so that own expertise and teaching ability is realized to the fullest. Earlier in life, elementary school teacher’s luxury to do so may be limited (which is why their job is so difficult and hopefully well-compensated), and one bailing on a kid due to lack of patience or teaching competence is detestable.
4. If Johnny continues to pursue this with genuine utmost motivation, he will most likely succeed despite any incompetent teachers. If he does not succeed and yet continues to pursue this to the detriment to his life, that is something a psychologist should help him with.
As for Timmy, if he learns to produce the expected result with least effort, for which he receives constant praise from the teacher, and keeps coasting this way, that does him a major disservice as far as mental mental and self-actualisation in life.
It's funny. You have created yourself a paradox. Replace comp sci with being a teacher. You have made the claim now that teachers can be incompetent but Johnny cannot be. Let's say Johnny wants to become a teacher and puts in lots of effort but just cannot teach. Now he is an incompetent teacher but at what point did he go from being judged on effort to being judged on ability? When he wanted to be a teacher and got a free pass for being a bad teacher? When he went for his first job and got a free pass for failing his exams? When his entire class learned nothing because he was unable to teach even though he put in lots of effort?
Where is the transition? At some point ability is more important than effort.
The paradox is only in your head. Do not confuse the process of learning a skill and practicing it professionally. The line between the two is beyond clear.
The question you refuse to answer is at what point should incompetencey be judged over effort. Little timmy who was always going to be a good teacher has now lost out because you the gave the position for the university place to little Johhny who everyone, despite all his everyone knew he was going to be a terrible teacher.
There is no benefit in always being praised for your efforts if you cannot deliver the goods.
I answered that and reiterated it. The outcome can be judged (and it is) when you do it professionally. Everything I said about evaluation on the effort was from the start about the learning process (the topic of this thread) and psychology in the critical formative period of young human’s upbringing.
> If a teacher is unable to teach Johnny to write a for loop, despite Johnny’s genuine utmost motivation, I would question teacher’s competence or at least fit.
This relies on everyone's ability to learn being determined solely by motivation rather than innate ability. As someone who has tutored both deprived children and the very bright I can say this unfortunately isn't true, even though the world would be a better place if it was.
> Professor Hawking's laidback approach to education continued during his years studying physics at the University of Oxford. ''I once calculated that I did about a thousand hours' work in the three years I was there, an average of an hour a day.''
The evaluation criteria don't need to be the same for your entire life. So if someone is taking an exam to decide whether they're fit to become a bridge engineer, ability should be the criterion. Little Johnny in school can still be evaluated based on effort. (In essence, over the course of the educational part of people's lives, slowly shift the criteria, and help them choose paths that will lead them to success.)
I believe that to learn well, you need to be challenged, but not too much. Ability-based evaluation only does that for students whose abilities happen to line up with the expected standard. It is bad both for gifted students and for struggling students.
> The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements.
I am really quite confused about what you think the point of education is.
In general, the world (either the physical world or the employment world) does not care about effort, it cares about results. Someone laboriously filling their kettle with a teaspoon might be putting in a ton of effort, but I'd much rather someone else make the tea who can use a tap.
Why do we care about grades? Because universities and employers use them to quickly assess how useful someone is likely to be. Few people love biochemistry enough that they'd spend huge sums of money and time at university if it didn't help get them a job.
> Someone laboriously filling their kettle with a teaspoon might be putting in a ton of effort, but I'd much rather someone else make the tea who can use a tap.
By your own logic, the student who fills the kettle with the spoon has produced the expected result.
Fast enough with the spoon and sky’s the limit, right?
A good teacher, while praising the effort, would help them find out about the tap. Not praising the effort would give the opposite signal! You have worked hard, and through no fault of your own (no one has built-in knowledge about the tap) you were essentially told that was for nothing?!
And if you have learned the tap, do you want to be done with it? Or be pushed to keep applying the same effort as with the spoon, but directed more wisely knowing that there’s a tap? Imagine what heights would you reach then!
The worst teachers are in whose class 30% of the students are filling their kettle with spoons all their time, 30% simply dip them into the puddle and never get used to do the work, 30% give up because what is even the point of filling the kettle when their home has a hot water dispenser.
You may be mistaking “the world” with “education” or “learning”. Producing a result is not evidence of learning progress. During learning, result is a somewhat useful metric if it roughly correlates with the level of effort, but relying only on result when determining whether to praise or reward a person during the learning stage is always a recipe for issues. A student may quickly learn to reproduce the desired result and stop progressing.
I've found that in adulthood, I've still been judged on results, not effort, and unless we're going to drastically reduce student:teacher ratios, I don't see how you even could judge on effort. Some kids are going to learn more quickly than others, and for them, no effort will be required. At best you might assign them busywork, but that doesn't take effort just as it wouldn't take effort for an adult to do the work.
I also don't think effort can be recognised in some spaces; as a programmer, I often produce results that in the end, result in very few lines of code written, looking at the end result alone doesn't indicate much.
It's like looking at a hand carved match-stick judging the result as low effort, not knowing that they started with a seed.
The end result is never the code itself. In fact, the end result exists over time, and often the shape of the result in the time dimension is better the shorter the code and the more thorough the intangible forethought.
But yes, I don’t know how clear must I be about it—this is learning (for very young humans still psychologically immature), that’s exactly why it has to be spelled out that evaluation must be on the effort, precisely because it is never on the effort in any other activity in adulthood.
As long as we don't have the resources to devote 10+% of the workforce to teaching, the bar will be wrong. The bar was wrong for me during school and university, and I found teachers who gave high weights to homework or even attendance quizzes to be extremely obnoxious.
On setting up expectations for adulhood, I think this is exactly backwards:
> If you got good grades at school with little actual effort and the constant praise for that formed your identity, you may be in for a bad time in adulthood.
Praising a child for effort without results seems way more likely to set them up for a surprisingly bad time as an adult. My personal experience has been that the "good grades/rewards without effort" thing has continued and seems pretty likely to continue through adulthood as long as you go into some kind of engineering.
Yes, this is a failure some or all school systems suffer from today, as I pointed out in my comment.
> My personal experience has been that the "good grades/rewards without effort" thing has continued and seems pretty likely to continue through adulthood as long as you go into some kind of engineering.
Based on my observation, people who are comfortable doing the work in engineering achieve completely different heights than people who got used to coasting. As applied ML spreads across industries, the difference between the competitiveness of those two categories will only become more pronounced. Furthermore, from my observation those who got used to coasting suffer from issues like perfectionism, narcissism, rejection aversion, and similar.
Sooner or later in adulthood not doing the work stops achieving results deserving praise.
> Praising a child for effort without results seems way more likely to set them up for a surprisingly bad time as an adult.
Not “without results”. Results are critical. However, if results do not comply with whatever requirements, that is not a factor in whether to reward, unless you reward compliance. Rewarding compliance has to happen to some degree, but should not be overdone unless your goal is to foster uncreative cogs.
It's fairly simple in most situations. If it doesn't involve a computer, it's handwritten in class. If it does involve a computer, it's a temporarily offline computer. We have figured out solutions to these problems already.
You forgot “no homework that counts, or a prison- or monastery-like environment where you have no access to any of these technologies for the length of academic term”. No, humans have not ever had a similar problem before, and also some of the solutions to various problems that we have figured out in our past are no longer considered reasonable today.
When institutions use simple rules to respond to change and rigidly follow them without due judgement, then some will fall through the cracks, and others will grift off them
Let's not assume a lot right now. OpenAI and other companies are torching through cash like drunken socialist sailors. Will AI be here as a Big Data 2.0 B2B technology? Most likely, but a viable model where students and laypeople have access to it? To be seen.
We all mooched off of dumb VC money at one point or another. I acquired a few expensive watches at Fab dot com at 80% off when they were giving money away, eh.
> [...] but a viable model where students and laypeople have access to it? To be seen.
You can run GPT-4-equivalent models locally. Even if all software and hardware advancements immediately halt, models at the current level will remain available.
A TI-84 Plus calculator (over 20 years old) is still useful today.
In isolation, I don't think a model necessarily becomes less useful over time. It'll still be as good at summarizing articles, translating text, correcting grammar, etc. for you as it is today.
If things do continue to advance and new models are released, which I think is likely, the old ones become less useful by comparison and in situations where there's competition. But then, through hardware/algorithmic improvements, better models also become feasible for universities/open-source groups/individuals - so you shouldn't be stuck with a 2024 era model.
How useful is it to argue about what would happen in the extraordinarily unlikely eventuality that all LLM development will cease in 2024, wherein everyone with the proclivity to use LLMs will be stuck with exactly these same models for decades to come?
My expectation has been that OpenAI is hoping to parlay dumb VC money into dumb government money before the tap runs dry.
If done right they would go from VC money with an expected exit to government money that overpays for incompetence because our only way out of deficit spending is through more debt and inflation.
Seems the implication is socialist sailors are spending someone else’s money on their drink, as opposed to hypothetical capitalist sailors who spend their own money.
This is similar to how the AI companies mostly spending VC’s money buying these accelerators from nVidia.
Sailors from certain former socialist states have a reputation for drunkenness that goes beyond the normally high levels of drunkenness of other sailors.
AI is here to stay; new methods should be used to assess student performance.
I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with having 'calculators' in our pockets 24/7 now.
We need to accept that the world has changed; I only hope that we get to decide how society responds to that change together .. rather than have it forced upon us.