
Exam Credit for Knowing What You Do Not Know - mhb
http://daviddfriedman.blogspot.com/2017/03/an-application-of-economics-to-teaching.html
======
whack
There was a Decision Analysis class at Stanford that took this idea to an
extreme. For each multiple choice option, you had to indicate the percentage
confidence of that option being correct. The percentages for all 4 options had
to add up to 100%. The score you got for each question = 1 + ln(confidence-
assigned-for-correct-answer).

So if you were completely sure and gave the correct answer 100%, you would get
a score of 1. If you gave it 90%, you would get a slightly lower score. If you
"guessed wrong" and gave the correct answer 0% probability, ln0 is negative
infinity, so you would instantly fail the entire assignment. It was a pain,
but it really made you reassess how well you know stuff, rewarded people who
truly knew their stuff, and burnt anyone whose confidence exceeded their
actual abilities.

~~~
Cyph0n
Why would anyone ever put 0% then? I would just redefine "impossible" to be an
arbitrary percentage that guarantees I don't nuke the question.

Quick question: is confidence taken in percent form (ln(90)) or decimal form
(ln(0.9))?

 _Edit:_ I am aware of the implication of having percent values, guys/gals.
But if it's in decimal form, a 5% confidence level will end up losing you the
current question AND three others.

~~~
derefr
I assume that was part of the lesson: in Bayesian decision theory, an agent
that assigns 0.0 or 1.0 confidence to something can no longer update on new
evidence about that thing. Saying something is 0% likely means being _so sure_
about that thing, that nothing—no matter how amazing—could or would ever
change your mind.

Basically, no human is _actually_ ever 0.0 or 1.0 sure of _anything_. Assign
five or ten nines of confidence, but not 0.0 or 1.0. (It's a lot clearer when
you remove the log scale and notice that a 1.0 confidence-value translates to
you saying you're "infinitely confident", which seems ridiculous.)

~~~
hyperpape
I suppose this is a bad place to ask this, but given that you need to do math
to update your credences and also that you can derive any proposition from
Pr('2 + 2 = 4) != 1, how does this work?

~~~
derefr
Basically: probabilities and credences aren't the same thing. They're measured
on the same _scale_ , but probabilities can be used to reason about either
facts or evidence, while credences are only "about" evidence (though
potentially evidence _about_ facts.)

To put that another way: it is "100% true" that 2 + 2 = 4 (in ZF set theory);
just like it is "100% true" that the sun _will_ rise today, because it already
did (and otherwise the Monty Hall problem would have a different answer.)

But the credence in an agent's mind _can_ and _should_ only be informed by
_observations_ of these facts; and the "observing hardware" cannot be 1.0
trustworthy. (Your senses can be imprecise; the brain can be buggy or biased;
the environment being observed can be the construction of a Cartesian "evil
demon"; etc.)

Given an _artificial_ agent with perfect knowledge (e.g. one who is able to
directly entangle itself with the environment it is trying to predict, rather
than just one able to entangle itself with _evidence about_ the state of an
environment), some things _can_ theoretically be held to have 0.0 or 1.0
probabilities (e.g. events that already happened)—but given an _embodied_
agent, the "maybe my processing is flawed" argument should still make these
into non-infinite _credences_.

~~~
platz
If the Higgs field quantum tunnels to a lower energy level today in our
vincinity, the sun will not rise tomorrow. Never put 100%

~~~
derefr
I didn't say "the sun will rise tomorrow"; I said "the sun will rise _today_
"—as in, the (trivial) probability that the sunrise that _already_ happened,
will happen, looking forward from the more distant past to the more recent
past. Given that we are talking about the probability of that event occurring
_from_ a reality where it occurred, the updated probability of it occurring is
100%. (But we cannot have 100% credence, simply because we can't actually be
100% sure we're in said reality.)

------
epalm
I was a Computer Science student and teaching assistant at the University of
Toronto for a number of years (about a decade ago), and the 20% rule was in
full effect: "On a test, if you write 'I don't know', you get 20% of the marks
for that question"

As a student, this challenged you to know and understand what you know, and
what you don't, and promoted the idea that knowing what you don't know is, in
itself, valuable.

As a teaching assistant marking tests, this virtually eliminated long,
tedious, often vacuous answers where the student was clearly throwing darts
and hoping for a mark or two.

~~~
silverphoenix
I recall there was another professor at UofT during my time there that used a
scheme of "Everyone starts with 30% on each question, every relevant and
correct statement you write earns you marks, every irrelevant or incorrect
statement loses you marks."

~~~
derefr
It'd be interesting to use that approach _without_ having each question need
to be limited by asymptotically approaching a 100% per-question score. You
could do _so_ well on a question (proving things the teacher didn't expect the
students to have learned at that level) that it would help you out with your
scores on other questions.

Probably I'd just let the questions be scored with integers, start everyone
off at 3, +1 for good points, -1 for bad points, and then take the Euclidean
distance of all their question-scores to find out the total "length" of the
reach the student has into good-answer space.

------
gnicholas
When I was studying for the LSAT, I found it very helpful to mark a confidence
indicator of 1-4 on every answer. Then, when I went back and graded my
answers, I could see if I was missing questions that I knew I wasn't sure of,
or if I was overconfident/sloppy on questions that I thought I knew. Not quite
the same concept, but in the same vein of "know thyself".

------
OisinMoran
This is a great idea! I think there should definitely be more emphasis placed
on knowing what you don't know in a good education system. Even something like
knowing what areas of study exist can be quite valuable.

I had a friend acting as a research assistant to a biology masters student who
was tasked with removing duplicates from an Excel file. The masters student
said it would take about four hours because that's how long it usually took
her to do it. My friend made a simple script in about an hour that then did
the work in effectively no time. I am by no means a proponent of the "everyone
should learn to code" movement but I think there could have been a lot of time
saved here if the biologist just new the shallowest fact that computers are
good for doing tedious things.

Obviously specialization is still very important but I feel introducing more
shallow learning could really make a difference in both communication and
innovation.

If you like this general topic Terence Tao has an interesting piece in a
similar vein on assigning partial credit for true-false questions [1] that was
posted on HN a while back [2].

[1] [https://terrytao.wordpress.com/2016/06/01/how-to-assign-
part...](https://terrytao.wordpress.com/2016/06/01/how-to-assign-partial-
credit-on-an-exam-of-true-false-questions/)

[2]
[https://news.ycombinator.com/item?id=11821903](https://news.ycombinator.com/item?id=11821903)

~~~
icebraining
That sounds a little uncharitable to the student. I know that cars are good
for carrying stuff, yet I don't rent one to carry my groceries for three miles
from the store to my home. Finding a programmer willing to do it for free
might alone take her more than four hours.

~~~
OisinMoran
Sorry I probably should have emphasized that this wasn't a once off task but
was very frequent.

------
grecy
Somewhat related, during my studies I got into some monumental arguments over
the "using your answer from the previous question" rubbish.

Let's say I had no idea how to do question #10, but needed something to start
with for #11. I would answer #10, and come up with an answer that made #11 as
easy as possible. (I don't remember, something like x^2 instead of some
integral junk).

Then I would complete #11 using x^2 as input.

Genius, I still think. My professors, not so much.

------
gnicholas
In an economics class on Games and Strategies, the professor gave us a choice
for one question: you can accept an 80% score for this question, or you can go
to the front of the room and pick up the prompt.

Once you have decided to do so you will get whatever score you deserve —
there's no turning back and declining to answer the question after seeing it.

I believe we were given the topic of the question in the instructions. IIRC,
the class split about 50/50.

------
mikecsh
As an electronic engineering student, our exams were negatively marked. If you
got an answer wrong you incurred (-0.25 * $availablePoints). If you didn't
answer the question you scored 0. We despised this at the time..

Now a medical student, we don't have negative marking. I feel it's actually
very important in this discipline to know your limitations and not pass-off as
though you know more than you do. I have however encountered resistance to
this, having professors question my very reasons for doing medicine due to
leaving questions unanswered..!

~~~
datburg
Do not worry. We know the ocean of knowledge they throw/threw us in with full
expectation of us drinking in all that new material while being surrounded
with sharks and there is a heavy storm that keeps a person from taking a
breath.

Still Medical school kicks ass and life after it is colorless. Also, why are
you complaining? With 4 Subjects tested every week year 2, 40 genes implicated
in ovarian cancer and their frequencies are over the top. The only time I was
disappointed was when a concept slipped because of the lack of waling minutes
to master. Concepts are more important now as I see it in an age when any
symptom or fact that slips can be double checked. Best years of my life!

~~~
mikecsh
I wasn't complaining, but thanks for this!

------
Sniffnoy
Note that for multiple choice exams you can go further and have people assign
a probability distribution over the answers, rather than just picking an
answer. Then you can use a proper scoring rule:
[https://en.wikipedia.org/wiki/Scoring_rule#Proper_scoring_ru...](https://en.wikipedia.org/wiki/Scoring_rule#Proper_scoring_rules)

------
tedmiston
I had several CS courses where writing something to the effect of "I don't
quite know the answer this question (or don't have time to solve it), but the
formulas / algorithms / bits of info I think are relevant are: x, y, z." On a
difficult question this might get half credit.

------
dnautics
> One problem in teaching economics--the course is on the application of
> economics to law, for law students--is that it is possible to think you
> understand something when you don't, when you are substituting some vague
> things you think you know already for much more precise and quite different
> things you are supposed to learn.

Isn't this generally the problem with economics writ large? It's possible to
think you understand something when you don't, when you are substituting some
vague math you _can_ do / _can measure_ for the more detailed and difficult
(and possibly intractable) phenomenology that is reality?

------
pirocks
This is already done by certain tests like the AMC and the old SAT.

~~~
TillE
This used to be the case on AP exams too, prior to 2011 according to Google. 0
points for a blank answer, but -0.25 for a wrong answer.

------
mathattack
Side note - if the article sounds uber-rational econ, it may be in part
because the author is the son of Milton Friedman. Here is his first post on
the topic. [http://daviddfriedman.blogspot.com/2012/11/the-use-of-old-
ex...](http://daviddfriedman.blogspot.com/2012/11/the-use-of-old-exams.html)
There is a great quote at the end.

 _And, for a last comment ... . I like to say that being a professor is better
than working for a living, except when grading exams. One reason is that
grading exams is a pain. Another is that it is when you find out that you have
not done nearly as good a job of teaching as you thought you had._

~~~
_yosefk
While it's hard to argue that it runs in the family to some extent in this
case, David Friedman is an econ thinker in his own right and certainly goes
way further into what most would call extreme libertarian/"uber-rational econ"
territory than Milton Friedman ever did. In fact, I'd say that "The Machinery
of Freedom" ought to entitle David Friedman to the simpler introduction
"...because the author is David Friedman."

~~~
gpawl
His most famous book his named after his father's most famous book. His father
clearly sets the context for his work.

------
wrsh07
My favorite thing about this is something I heard from Dr Bender, Stony Brook
CS: if you get <20%, there's no way to argue you should have passed when you
failed, because a student who didn't take the class would do better

------
savanaly
If you're interested in the topic of making students divulge what their
confidence in their answer is, you might also like this classic post by
Terence Tao. [0] It explores how you might handle and grade a true/false test
where for each question the student has to fill in their best estimate of the
probability they got the question right.

[0] [https://terrytao.wordpress.com/2016/06/01/how-to-assign-
part...](https://terrytao.wordpress.com/2016/06/01/how-to-assign-partial-
credit-on-an-exam-of-true-false-questions/)

------
Asooka
I've seen it implemented in one regional mathematic competition. It was a
multiple-choice test, where leaving an answer blank netted you 3 points, while
answering wrong gave you 0 points.

------
gpawl
The author is apparently unaware that his "new" idea was standard practice for
statistically-validates tests (like the SAT) for decades?

------
redsn0w422
At UIUC, Professor Erickson has a policy that rewards students with 25% credit
for indicating they don't know the answer to a particular question, both on
homework problems and exams.

[1] [https://courses.engr.illinois.edu/cs473/sp2017/exam-
policy.h...](https://courses.engr.illinois.edu/cs473/sp2017/exam-
policy.html#other)

------
mdholloway
I was hoping this would look something more like, given a scenario and a
question about it, being able to clearly identify the pieces of information
without which the question is unanswerable.

That said, time saving and BS reduction are also valuable goals!

------
andrewclunn
I'm upvoting all the comments people didn't make.

