

Should CS students be graded on performance? - drfugly
http://sbcoded.com/2011/03/should-students-be-graded-on-performance/

======
moxiemk1
It depends on the context of the problem:

* If it's being framed as being applied to a small number of items on one machine, I'd mark down for premature optimization. Normally such assignments have a specific point, and going off the deep end with efficiency measuring often misses it.

* If it's being framed as being applied to huge data sets, asymptotic complexity is the metric to go off of.

* If it's being framed as a latency-dependent (UI, servers) then actual timing is the metric to grade off of.

I'd say that if you have an algo that is faster for small numbers, it usually
doesn't really matter that it's faster. Small numbers go fast no matter what.
Unless it's latency dependent, it's encouraging students to work on things
that don't matter at all.

EDIT: spacing EDIT2: spelling

~~~
drfugly
When you were in class, where you ever even introduced to the concept that
sometimes it might be possible that O() notation might not tell the whole
truth? I understand that for the most part you shouldn't have to worry about
it, but shouldn't we still warn students about this?

~~~
seancron
Could you explain what you mean a bit more by big O notation not telling the
whole truth? Are you referring to the fact that algorithms that look the same
in big O notation might have drastically different constants, which can affect
real-world performance?

~~~
Someone
It can be worse. A O(n log n) algorithm might be slower than a O(n^2)
algorithm for all practical values of n.

Also, one should be careful what to count. Sorting strings, for example, is
not quite O(n log n); average string length/expected offset of first
difference/whatever should also be in that O().

Along the same line, for many algorithms, cache-locality is more important
than number of CPU cycles. So, counting cache misses rather tha cycles can be
the better way to judge an algorithm.

------
patio11
Aside from teaching big O notation, I would be skeptical of grading for
performance. Teaching people that the performance code they write to solve a
toy problem _matters_ might lead them to go into industry and then perform
microoptimizations on code to solve performance problems when they are
_virtually universally_ caused by the interaction of many moving parts.

If you're doing web apps, application code is close to the last place I'd
reasonably expect there to be severe performance issues. You're far, far more
likely to either blow something architecturally ("We call a foreign API in the
request/response cycle.") or flub settings where simple best practices produce
repeatable results ("It is 2011 -- do you know where your gzip is?")

------
parfe
First, find a better color scheme for something you intend people to read.
That was painful.

Second, when I was a CS student I expected to be graded on my response to a
particular problem. If you wanted me to roll my own sort, I expect that to be
specified. If you want me to explain in comments why I chose certain
algorithms say something in class. I'm not going to waste time implementing an
efficient sorting algorithm for a 100 element list (or 1000 even) just to
provide a TA with a nerdgasm.

I feel this article is asking if students should be graded on an arbitrarily
deep rubric. I had enough to worry about as an undergrad without caring if a
bored TA was going to dock me 10 points for calling Arrays.sort.

~~~
moe
_First, find a better color scheme for something you intend people to read.
That was painful._

<http://rdd.me/3ltgecrd>

~~~
glenjamin
On my iPhone safari, readability killed all the big Os, makes the article very
hard to follow.

------
mvanveen
YES. My Data Structures and Algorithms class was graded by benchmark (ECS 60,
Sean Davis, UC Davis). Being graded this way was one of the most valuable
collegiate experiences I've had (I'm a fourth year on the way out).

Sean gives his students two "challenge programs" a quarter. The first program
is usually ultimately a matter of picking/optimizing the right data structure.
We do graph algorithms by the 2nd half and so the later challenge problem is
inevitably some graph algorithm.

The challenges are close to real world problems. For ex., our 2nd challenge
program was to determine the maximum # of donations Obama could receive at any
one time given a capacitance graph of his website's network topology (I took
the class around the '08 election). We had to design, implement, and test
everything by ourselves.

I don't agree with the naysayers who suggest that grading this way doesn't
teach the really important software engineering tasks. You really had to feel
out the problem domain in order to get the best optimization, and it the book
answer wasn't always the best answer. Optimization is about a lot more than
simply picking the right data structure or algorithm.

Partnering with the crazy russian genius with an 8086 instruction set in his
head that quarter made me the C programmer I am today.

TL;DR Grading by benchmark makes students better programmers.

------
DarkShikari
The ACM contest has it right: specify the size of the input and the running
time available. The program must be able to run an input up to that size in
the given runtime. That's it: no more, no less.

At Harvey Mudd College, we have a class based on the ACM which works
similarly, though with relaxed runtime constraints: 42 minutes per problem. In
practice, you rarely need more than a few seconds, but the 42 minute cutoff
(combined with large problem sizes) is a great way to prevent people from
using brute-force approaches when there's a smarter option available.

~~~
JoachimSchipper
42 minutes? I recall more than one ACM problem that could be bruteforced
_just_ within the running time (in C). Actually, I did really badly in one
contest because I applied serious mathematics to one such problem (which gave
a much faster program, but writing it took a lot of time and fast enough is
fast enough.)

~~~
DarkShikari
Sure, some can be bruteforced, but typically it's intentional.

A good example of what I mentioned was an assignment this week. Last week, the
input size was constrained to a thousand or so, and Floyd-Warshall was able to
solve the problem. This week, it's the same problem, but with inputs up to
~50,000 in size. The old program, run on the new test cases, doesn't finish in
a day.

------
aia
No, they should not. Unless performance was one of the assignment
requirements.

Performance is rarely on the most important requirement in software
development.

If you don't believe me, watch Prof. Leiserson make this point.

[http://ocw.mit.edu/courses/electrical-engineering-and-
comput...](http://ocw.mit.edu/courses/electrical-engineering-and-computer-
science/6-046j-introduction-to-algorithms-sma-5503-fall-2005/video-
lectures/lecture-1-administrivia-introduction-analysis-of-algorithms-
insertion-sort-mergesort/)

------
drallison
Define performance. The question is ill formed since there is nothing
specified to measure. The example wonders who should get the better grade
based upon design decisions each of which are defensible in some context. And
it's the asymptotic and/or typical behavior that is touted, not the
implementation.

At a most elemental level, a program written by a CS student should work, that
is, do what it is supposed to do for all legal inputs and provide meaningful
diagnostics when it fails. That's hard, but that is the nature of programming.

Finding and choosing algorithms to accomplish a task (well defined or not) is
also computer science. That is algorithmics and design. Doing a good job here
is harder to measure and depends both a deep understanding of the available
building blocks and the problem at hand. Moreover, trade-offs are involved and
need to be resolved even though there is little real data to guide decisions.
Seeing and exploiting invariant properties of the problem often enables better
performance in some dimension; insight there is often called cleverness.

Equally important is the ability to see complex things in a simplified way by
introducing abstractions. That's the only way to solve difficult, complex
problems, and it is exceptionally difficult to codify.

------
protomyth
I had a CS Professor who was convinced that he had cheaters in the class(1),
so he decided we should all be graded on performance of our programs. This
resulted in a lot of stress for a lot of students (C code only, most students
taught only Pascal/Modula-2) and some seriously nasty things done in the
programs. I rate it a big fail for what we were doing.

I can actually kind of see this for embedded / small device programming.
Embedded systems that can't meet their performance goals are useless and
buying faster hardware is very costly option.

(1) he also believed not one of us submitted a C program to print "Hello,
World!" that worked. It was a 8AM class and I had just driven 120miles (home
visiting) through a blizzard to make it to class when he announced this fact
in a dramatic fashion. I required calming down.

~~~
xiongchiamiov
How would grading on performance discourage cheating?

------
baddox
Generally, performance isn't interesting in cases where asymptotic running
time isn't dominant. You could sort 3 items with a couple of `if` statements,
which would compile to a handful of microprocessor instructions. That's not
interesting. Surely, in any data structure or algorithms course offered as
part of a CS degree, the sorting assignments are for the "general sorting"
problem. This implies relatively large data sets (and unpredictable, since you
don't expect to be able to use a faster non-comparison sorting algorithm).

------
davidk0101
No. Stuff like this only comes with experience and it's more important to
teach students about the fundamental and overarching themes of computer
science than to pester them with optimization issues.

