

College professors grading essays purely by computer - is this legit? - AskHNForHelp

Hi HN, need some opinions here. My girlfriend is going to college at Oklahoma City Community College. She's enrolled in a intro history class there. Today she tells me that their professor told them to submit their essays through this url:<p>http://katengine.com/collector/he-hist/prompt/MyHistoryLab011/begin<p>He explained to them that this is an automated system that will actually grade their essays and only if they want an exception will he review their work. He also told them that their work would be used to further train the system - he didn't tell them anything about how to opt out of this if they don't want to. I'm not sure if this is something he is doing himself or something the college is adopting in their classes.<p>All i've been able to find about this katengine is this document:<p>http://www.reedpetersen.com/portfolio/pe/pa/writetolearn/download/kat-engine.pdf which makes it seem like a simple latent semantic indexer.<p>There's another marketing PDF here : http://www.reedpetersen.com/portfolio/pe/pa/writetolearn/download/wtl-faq.pdf<p>that has a section on their "Intelligent Essay Assessor" which claims it works by checking similarity with some model essays. This seems like a simple LSI model with some added features.<p>I'm pretty irate about this- i work on machine learning and distributed systems myself and given the state of the art, I just can't fathom how a teacher could responsibly decide that a system like this can actually assign a student a final grade for essay type answers. I'm planning on asking my girlfriend to raise this issue in her class.<p>So my question is this - does anyone here have any more experience with these kind of essay grading systems - are they actually able to do a reasonable job (and is my indignation overblown),  or is this a case of people using poorly understood technology in bad ways that should be protested or at least explained in greater detail to the students.
======
dalke
The details at <http://en.wikipedia.org/wiki/Turnitin> may be of thin help. At
least one Canadian student has protested, and won. The number one point I took
from it would be to advise her to not agree to the clickwrap license. Then she
could at least protest about the abuse of her copyright.

If you two want to really stir things up, you can start a company which does
text analysis of student essays and hire her with an exclusive contract to
provide corpus material.

Some background for the company. "katengine.com" appears to be owned by
Knowledge Analysis Technologies.
<http://www.forbes.com/forbes/1998/1228/6214047a_print.html> has more info
about the company, which was acquired in 2004 by Pearson Education
<http://kt.pearsonassessments.com/whyChooseUs.php> .

If the teacher will be using the 'Intelligent Essay Assessor' from
[http://www.mywritinglab.com/learn-about/writing-
practice.htm...](http://www.mywritinglab.com/learn-about/writing-
practice.html) then yes, it's a variation of LSA, as you found. There's a fact
sheet there with a picture of what the GUI might look like.

Based on that name,
[http://www.d.umn.edu/~tpederse/Courses/CS8761-FALL04/Project...](http://www.d.umn.edu/~tpederse/Courses/CS8761-FALL04/Project/Readme-
SanLorenzo.html) says "This was developed in the late nineties by Tom Landauer
and his team and is based on Latent Semantic Analysis. In this, using a matrix
transformation technique known as Singular Value Decomposition (SVD), deeper
hidden relationships between words and documents are discovered, and these
relations are used to understand relation between words and sentences."

I found [http://writing.ucdavis.edu/news-1/carl-whithaus-essay-
publis...](http://writing.ucdavis.edu/news-1/carl-whithaus-essay-published-in-
measuring-writing) which describes an essay critical about machine grading. I
couldn't find that essay though.

[http://www.huffingtonpost.com/2012/03/29/robo-readers-the-
ne...](http://www.huffingtonpost.com/2012/03/29/robo-readers-the-new-
tea_n_1388227.html) quotes: "A prankster could outwit many scoring programs by
jumbling key phrases in a nonsensical order. An essay about Christopher
Columbus might ramble on about Queen Isabella sailing with 1492 soldiers to
the Island of Ferdinand -- and still be rated as solidly on topic, Shermis
said." as well as "Computers also have a hard time dealing with experimental
prose." So she might gum up the works by using an unusual but legitimate
style.

Speaking of gumming up the works, I suppose that the essay must be submitted
to the teacher in a text or word processor format? That is, hand-written or
printed essays are prohibited? Or embedding a strange font which shows a "b"
for "a", etc., and where the text is transposed accordingly? Pull quotes to
throw off the formatting? Hidden text, colored white? That would be
subterfuge.

Edit: Pearson also has "second-tier graders", says
[http://news.slashdot.org/comments.pl?sid=2757605&cid=395...](http://news.slashdot.org/comments.pl?sid=2757605&cid=39528641)
, with a human involved.

~~~
aaron695
"A prankster could outwit many scoring programs by jumbling key phrases in a
nonsensical order. An essay about Christopher Columbus might ramble on about
Queen Isabella sailing with 1492 soldiers to the Island of Ferdinand -- and
still be rated as solidly on topic, Shermis said."

To be smart enough to prank on a subject and pass would mean a pretty in depth
knowledge hence the mark would be correct.

In fact a human would incorrectly say the student was bad at the subject so it
makes the computer superior in marking students actual knowledge of the
subject :)

~~~
notahacker
Assuming there are _existing lecture notes highlighting key information_ , you
could probably pump them into the sort of "article spinners" used by SEO
spammers and produce something that looked like accurate, original writing to
a computer algorithm, without having any actual knowledge of the subject at
all.

Studying history (even at junior high school level in the UK) is more about
interpreting sources and making reasoned arguments, than simply regurgitating
facts and references. You can parrot or parody a list of facts and references
you've been given whilst being a hopeless history student. Sounds like a
machine comparing with a "model essay", meanwhile, would penalise those
students foolish enough to construct original arguments and support them with
reference material beyond the compulsory reading list.

------
dgunn
I'm not sure what the issue is here. I bet checking and scoring based on a
combination of grammar, spelling, and a rough relationship to the topic at
hand would produce pretty good results. Further, if you feel you were graded
unfairly, the professor specifically said he will review it personally.

He's probably already overworked and underpaid. Why not help him and the
college implement technology that could prove to be very helpful.

------
jpwright
Only if students are also allowed to submit essays generated by a computer.

~~~
phaus
This is an interesting idea. If you had access to the grading algorithm's
source code, wouldn't it be easy to randomly generate essays that always score
100?

~~~
dalke
I don't see why. The code could be very complicated to understand, or the
grading mechanism could be hard or impossible to reverse. (Just like
cryptographic hashes.)

Better would be access to the training corpus. Then use that to train, say, a
Markov generator, and have it replace words from a thesaurus, in hopes of
bypassing a similarity detector.

------
JohnHaugeland
So I guess I think that you might be well advised to go to the dean and say
"listen, we pay you good money, and we expect our instructors to actually do
their jobs, rather than to farm them out to machines."

At the same time - and I mean no disrespect - this is community college we're
talking about. The purpose of community college is to pick up skills, rather
than a liberal arts education; you should, at that price, expect to get a
somewhat reduced response.

I am with you here; this seems lazy and inappropriate.

But you get what you pay for. OKCCC is $75 a credit hour. By contrast, my alma
mater currently charges $24,000 per year (out of state,) and nominal is 15
credits a year. Even trade schools tend to cost several hundred dollars a
credit hour.

At that kind of price difference, you have to realize that you're forcing a
lot of corner cutting.

~~~
dalke
You are inappropriately harsh. Students go to a CC for several reasons. Some
go to a CC because it's cheaper and closer to home. These people plan to
transfer to a 4 year college after getting an associate's degree. Others
didn't do so well in high school and want additional remedial courses before
going to a 4 year college. In many states, it's easy to transfer from a CC to
a 4 year state college, because the additional work needed for an AA degree is
more meaningful than the work to graduate from high school.

For example, "The regents of the University of California report that 30
percent of UC graduates attended a community college before transferring to
the UC."

Yes, as you say, CC is also for career education. It's also for non-
credit/continuing education. I took several CC classes in the latter category.

And the comparison of OKCCC to your alma mater is almost useless. Better would
be to compare to Oklahoma State where it's $147.50/credit hour. (There's
another roughly $110/credit hour in "fees". For an arts&sciences degree, the
overall cost is about $275/credit hour. For OKCCC it's $99/credit hour,
including fees.)

The "corner cutting" doesn't need to be in the education. The mandatory fees
for OSU include "student union renovation", "campus recreation", "Academic
Excellence", "Library Automation & Technology", "Life Safety & Security", and
others. Since many people go to a CC because it's local, they are less likely
to spend time in the student union or in on-campus recreation so the
corresponding fees are smaller. The library at a CC is likely smaller, and
less in need of automation.

Also, the lab equipment for a two year program doesn't need to be as advanced.
My 4 year college upper-level physics lab had microwave generators, an x-ray
crystallography machine, and a radiation source. Why should a CC also have
them?

These aren't cutting corners, these are inappropriate expenses for the goal of
the students and the school.

That said, a CC program isn't going to have tenured research professors in a
field teaching an introduction course, and won't have the fellowship/intern
programs in place, or foreign exchange programs. These can (indirectly) affect
the quality of education.

But to say that these are all cut corners is to idealize what an education
must be.

~~~
JohnHaugeland
> Students go to a CC for several reasons.

Yes. For some time I was one of them. I am uncertain why you interpret what I
said to be harsh.

Respectfully, it seems to me like saying "look, at the price you pay at
McDonald's, there's not enough money floating around to get a filet."

This is realism, not harshness. It costs a lot of money to run a school, and
the US doesn't really fund its community colleges.

.

> And the comparison of OKCCC to your alma mater is almost useless.

It would be nice if, while saying I'm being inappropriately harsh, you
differentiate between something that's useless and something you merely do not
understand the use of.

.

> These aren't cutting corners, these are inappropriate expenses

Sure. They're also not even slightly the dominant costs in running a school,
and not what I referred to.

But you're right, the topics you raised, that I did not, aren't particularly
cut corners.

.

> That said, a CC program isn't going to have tenured research professors in a
> field teaching an introduction course, and won't have the fellowship/intern
> programs in place, or foreign exchange programs. These can (indirectly)
> affect the quality of education.

My my. And here I thought you didn't agree with me.

Of course, these also aren't the dominant costs in running a school, and
_also_ aren't what I was talking about.

As a point of preference, I generally ask someone to explain themselves before
calling them wrong in public. But maybe that's just because I don't like
arguing with my own guesses about what someone meant. YMMV.

.

> But to say that these are all cut corners is to idealize what an education
> must be.

I would request that you re-read what I said more carefully. The things you
point out aren't actually related to what I was discussing.

