

Mechanical turks, AI now grading student essays - yummyfajitas
https://chronicle.com/article/article-content/128528/

======
wisty
The real question - is it necessary, or even a good idea, to vertically
integrate teaching and assessment? Maybe for a specific area of string theory,
where both teachers and assessors are thin on the ground. But for calculus
101?

~~~
metachris
Exactly -- teaching and grading should be separated. I'd prefer a smart AI
over some random remote assessor any day.

I always disliked the usual subjective way of grading, in particular if it
takes several weeks to receive them. Having a smart program grade me
instantly, and fairly, I think I would have liked it!

~~~
troutwine
My wife is doing graduate work in Old English; most of her assignments to her
students have a strongly quantitative nature: did you or did you not translate
this correctly? Her peers, however, are in more murky territory. They study
things like Gender and Being and Ideas of Progress in Post-Shakespeare
England: strongly quantitative work. Hearing them talk at parties, the most
common conversation is one of how they approach grading, being what rough
heuristics they use. That some clever, overworked person is grading a essay
means no more than having a heuristic markov process applied to establish some
value for a given text but poorly owing to exhaustion and disinterest and
arbitrary bias.

If we absolutely must peg a number to a person's work then we might as well
have a computer do it reproducibly. I can't imagine there will be many
University humanities departments happy to adopt such "inhuman" methods: best
to wear actual humans down to nothing.

~~~
troutwine
Please forgive the spelling issues and the last "quantitative" where I meant
rather "qualitative". I've been struck by a miserable cold.

------
pavel_lishin
> Software has no emotional biases, either

So what sorts of biases _does_ it have? Remember the ol' adage about
programmers figuring out any sort of metric, and exploiting it? I wonder if a
group of students could find a way to game the grading system.

~~~
_delirium
The essay-grading software I've seen described in enough detail to figure out
how it works has a heavy bias towards enforcing specific structural rules,
while mostly missing "quality of argument" type questions. So a student
writing to exploit its bias would want to worry more about formal structure:
essay organization, correct grammar, use of certain "must-have" keywords,
etc.; but worry less about things like avoiding logical fallacies in their
argument, because the software won't notice those.

~~~
pavel_lishin
It would be amusing if true AI rose out of our attempts to catch cheating
students.

~~~
eru
That would be fun, though I believe that the problem of catching students
doesn't provide enough data.

------
manux
> a "consumer-based approach" to higher education

That has to be another problem.

~~~
_delirium
I actually think it'd be a good idea for education to become more learner-
driven, but I agree it isn't a good idea without more changes. When you
combine "consumer-based" education with an education system that is heavily
oriented towards just-get-the-credentials, that's what makes it unworkable,
because then what the customer really wants is just the credential ASAP. For
learner-driven education to work, the learner (or "consumer" if we must) has
to genuinely have things they want to learn, along the lines of what
constructionist education, Montessori, unschooling, hacker culture, etc. try
to aim for.

