
Academically Adrift: Limited Learning on College Campuses (2011) - luu
http://www.insidehighered.com/news/2011/01/18/study_finds_large_numbers_of_college_students_don_t_learn_much#sthash.xiIXcfgd.dpbs
======
nmrm
Standardized assessments are an inadequate way to measure outcomes at the
higher ed level; the methodology discussed in the article is terminally
flawed.

I propose that you _fundamentally cannot measure what a given person learned
or didn 't learn in college from a major-agnostic test_. Learning at this
level is far too specialized.

College is _NOT_ and _never should be_ about getting really good at basic
skills. These tests treat college as an _iteration on elementary and secondary
education_.

For example, I majored in math and now I'm in grad school in a related field.
I'm sure I would score about the same -- if not worse -- on a standardized
mathematics test now as I did in high school. The tests are generally a
combination of calculation/arithmetic and Euclidean geometry. Real mathematics
has very little to do with calculation, and I was definitely better at
churning out Euclidean-style deductions in high school than I am now (I could
trivially recover a lot of the useful clever identities I had internalized in
high school, but I haven't looked at that material in years).

To conclude that this means I didn't improve tremendously in mathematics
during college is just sloppy.

In fact, I would be highly unsurprised to see a negative correlation between
actual outcomes (number of top tier grad school admissions, desirability in
the market place, professional success 5, 10, 15 years after graduation) and
institutions which score very highly on these standardized assessments.
Iterating on high school is an absurd waste of the college years.

edit: computation -> calculation.

~~~
edtechdev
I'd read up on the Collegiate Learning Assessment (CLA). It addresses the
concerns you raised, and it's not aiming to measure some of the things you
assert it is measuring. It's not the same thing as the SAT or the GRE.
Formative (as opposed to summative) assessments like the CLA are helpful in
reforming and improving education. When a high percentage of students are
still making the same basic grammatical and writing errors they were as
freshmen, that's useful to know. The same thing with mathematical and other
skills. It helps convince folks at universities that teaching needs to be
reformed, improved to go beyond the standard lecture, homework, and test
formula which is ineffective for learning:
[http://news.sciencemag.org/education/2014/05/lectures-
arent-...](http://news.sciencemag.org/education/2014/05/lectures-arent-just-
boring-theyre-ineffective-too-study-finds)

See also concept inventories which are tests which are similarly helping
measure and assess student conceptual understanding (and misconceptions) in
various disciplines (including computer science and mathematics). There are
also surveys of student attitudes and engagement and beliefs which are helping
improve student learning and retention (like NSSE - the National Survey of
Student Engagemen). Then there are non-standard assessments which are helpful
for assessing and improving and demonstrating student learning, such as
e-portfolios.

Learning can be measured. It can be assessed. (As can motivation, beliefs,
etc.) And there are some skills we try to help students learn that are indeed
major-agnostic. The ability to write and communicate effectively. The ability
to critically think about and evaluate information. The ability to work with
others. The ability to properly use and apply mathematical and computational
tools. Even digital humanities folks are using computational tools to assist
their work. Employers are complaining about some college graduates having poor
communication skills, having poor programming skills or conceptual
understanding of fundamentals (fizz buzz test), etc. There are things that can
and should be improved at the college level - at all levels.

~~~
nmrm
_(this is a living comment until the first reply. Sorry.)_

I have four major points:

1\. You can assess even major-agnostic, general skills in content-specific
ways. Such assessments are far less likely to have the negative externalities
of more generic tests (e.g. first or last semester "Literature" courses which
are basically CLA et al. boosters. These most definitely exist). But even
still, I think this is a bad idea and too ripe for abuse.

2\. Generic "Critical Thinking" is _often_ not all it's cracked up to be, and
_some experts claim_ it is not statistically correlated with the sort of
critical thinking _purported to be mastered_ in science and mathematics
courses. (Oh, CLA really likes hedge words. They mean you're smart... I
mean... that "you've likely acquired transferable critical thinking skills"
[1].)

3\. Higher Ed shouldn't be responsible for filling in gaps in High School and
even Elementary education. Also, _grammar_!? I rest my case!

4\. The CLA is computer-graded (no, really, it is [1]).

\--

 _1\. A Problem with Generic Critical Thinking_

Generic "critical thinking" is often nothing more than "the right kind of
sophistry". The author of [1] agrees with this sentiment w.r.t the CLA
(although, it should be stated, not in general. See the source; what follows
are my own opinions, not those of the author of [1]).

In addition to pointing out that any answer along the lines of "IDK" is
treated as sufficient, he also notes that the model answers provided for the
CLA are full of logical fallacies.

But the author of [1] really hits the nail on the head in this paragraph:

 _This is exactly the kind of instruction and practice students receive in a
dedicated critical-thinking or informal logic course. Critical-thinking skills
are not statistically significantly enhanced by content-specific courses like
introduction to philosophy or chemistry, or by content-independent courses
such as symbolic logic._

I argue that the what the CLA measures is not what colleges and universities
are or should ever be intended to teach. Instead, it measures a tortured form
of "critical thinking" that is somehow not statistically significantly
improved even by courses in the hard sciences and mathematics. Not even by
courses in formal logic! Only by courses in informal logic. If science and
math don't improve "critical thinking", I'm not sure what "critical thinking"
is supposed to mean. Except, of course, the right kind of sophistry+.

Ergo, it's likely that if performance on the CLA significantly improves, it
will be because colleges and universities are teaching to the test in the
guise of "Critical Thinking and Informal Logic" (although I'm sure
universities will have better names for their mandatory freshman CLA courses).

I'm very concerned that the true humanities -- one place where actual critical
thinking is taught -- will be gutted in order to cover the costs of such
instruction (I already know of one instance where this has happened).

 _2\. Higher Education Should Assume the Basics_

As for grammar and mechanics, things are even simpler. Colleges and
universities are absolutely not responsible for those sorts of basic skills.

[1] [http://www.assessmentupdate.com/sample-articles/a-fatal-
flaw...](http://www.assessmentupdate.com/sample-articles/a-fatal-flaw-in-the-
collegiate-learning-assessment-test.aspx)

\+ My use of the word sophistry is definitely intentional. It's worth noting
that the sort of persuasion rooted in a combination of informal logic and
careful rhetoric ("always use hedge words" is a modern equivalent) is exactly
the sort of thing that the Sophists were known for!

Disclaimer: I'm a huge fan of assessment-driven education. I just don't think
it's right for institution-level assessment at the higher education level. Not
unless it's content-specific.

------
tuke
Before your accept this review at face value, read the book for yourself.

This review, like many, fails to note that the data in question regards first-
and second-year students (i.e., the second sample is in the third semester --
only 1 1/2 years after starting college). Many of us will recall that we made
our big leaps in our junior and senior years, after declaring a major. In a
later study, the authors claim that similar observations can be made regarding
older students. Still, I would read this book with some skepticism.

For a more subtle review, try this one from the New Yorker:
[http://www.newyorker.com/arts/critics/atlarge/2011/06/06/110...](http://www.newyorker.com/arts/critics/atlarge/2011/06/06/110606crat_atlarge_menand?currentPage=all)

~~~
crisnoble
The link in the story to the book was broken, but this one works:
[http://press.uchicago.edu/ucp/books/book/chicago/A/bo1032722...](http://press.uchicago.edu/ucp/books/book/chicago/A/bo10327226.html)

------
nmrm
Comparing my experience in higher education to the constant litany of negative
literature on education sends up red flags. I typically write off
books/articles in this genre as fear mongering.

I've not read this particular book, but a lot like it. The typical pattern is
that engineering majors don't see order-of-magnitude in writing and "critical
thinking", and humanities students don't see order-to-magnitude improvements
in science and math.

But then the author averages everything together and concludes no body's
learning much. Go figure.

Getting a lot of value out of college isn't hard. Attend a state school or
reputable private college (read: typically for-profits and online programs are
a waste of money; mileage may vary). Once there, major in math, a science, or
a core humanity (the latter typically doubled with education or another
"practical" major). Meet people and have a good time, but make sure you're
spending 30-40 hrs/wk on course work.

------
JBiserkov
Q: Why is there so much knowledge in universities?

A: The freshman bring some in, if little; the graduates take away practically
nothing; so it naturally accumulates there.

------
Spooky23
I guess I've become an old man. I can't think of a class where reading less
than 40 pages of material and 20 pages of writing a semester or equivalent
ever happened. Even "Intro to East Asian Cinema" required a grueling schedule
of 2-3 movies a week plus multiple papers.

And I attended a mid-tier public university in the 90s!

~~~
bcaine
I'd be curious how the study measured 40 pages of reading each week. My guess
is that most, if not all of my classes (Computer Engineering student) have >
40 pages of reading associated with the topics taught each week, but for many
of them the reading is never assigned, nor is it ever performed.

In my experience, reading only really is necessary when you are really
confused on a topic from lecture, miss class, or have a terrible professor.

I'd assume its similar across a lot of STEM disciplines (with respect to not
actually reading textbooks much).

~~~
Swizec
I don't know anyone who doesn't read the textbooks or at least the "condensed"
slides version with some 500+ slides per class. Not in STEM not anywhere else.
It is simply impossible to retain that much information just from going to
class, you _have_ to re-read it all before exams. Which is when the reading
happens, nobody actually does the reading every week. That's just silly.

Man, I wish I went to uni in the US. Every time I hear anything about the US
college experience it just sounds like a piece of cake.

~~~
gajomi
>I don't know anyone who doesn't read the textbooks >Not in STEM not anywhere
else

I would almost never "read" (though I would occasionally reference, page by
page) the textbooks in core STEM classes when I was in college. The most
important thing was always the notes and making sure you could work through
exercises. And this only became more true at advanced levels, when oftentimes
there isn't any textbook available. At least in the physicsy/mathy realms I
belonged to, the information density of your typical lecture was such that a
semester should be easily compressible to a dozen pages of typed notes. But
that isn't to say, I don't think, that it wasn't difficult!

>Man, I wish I went to uni in the US. Every time I hear anything about the US
college experience it just sounds like a piece of cake.

There is some truth behind what you are hearing, but it is quite far from the
whole truth. In contrast to many European university systems (at least the
ones I am familiar with) there is a lot of time purposefully built in to take
courses in different areas and build up "perspective". So if you want to take
it easy you can take some bullshit classes and just coast. Or you can take
interesting classes which might be challenging. The variation in student
experience can be quite large. If you go to a top school, however, you can
expect to be surrounded by a bunch of competitive types who are quite good at
what they do, and making good in this environment would certainly not be a
piece of cake.

~~~
nmrm
> the information density of your typical lecture was such that a semester
> should be easily compressible to a dozen pages of typed notes. But that
> isn't to say, I don't think, that it wasn't difficult!

Oh how true!

> Or you can take interesting classes which might be challenging.

Yes. With flexibility comes the possibility of abuse. We shouldn't throw the
baby out with the bath water.

I had quite a large humanities load in undergrad (majored in STEM fields, but
with a BA), and purposefully took in-major, junior and senior level courses in
art, English and history. There were pre-reqs but most instructors would waive
them because the classes never filled. Others decided to take 100-level
communications and business electives to fill the same elective slots. Even
after taking graduate-level STEM courses, a junior-level art history course
remains the most difficult course I've ever taken. The way the course was
taught was a pretty intense combination of "right" and "left" brain thinking.

In retrospect, forcing students to _choose_ to challenge themselves (or not)
is probably the single best way to prepare students for the "real world", and
probably not a bad predictor at success either.

------
lmartel
I think the most important data point here is the average of 12-14 hours spent
studying per week, including "academic" socializing. Of course you're not
going to accomplish much in that time; most employers would fire someone
putting in twice that much time for slacking off.

The rhetoric for as long as I've been alive (I'm college-aged) has been "get a
degree, get a degree" and as it turns out getting a degree isn't very hard. We
need to either make it harder or somehow change the rhetoric to "learn
something."

~~~
Swizec
That's usually 12-14 hours of studying _on top of_ 30-40 hours of classes.

At least that's what's expected. What usually ended up happening for me was
blowing off 30-40 hours of classes a week, doing some 10-20 hours of studying
(including homework and such), and having a near full-time job. Then followed
by some 15 hours of studying per day during exams.

But I never graduated. So that probably wasn't the best strategy. Although I
did get a bunch of solid work experience in my field.

~~~
lmkg
> That's usually 12-14 hours of studying on top of 30-40 hours of classes.

That sounds like an _extremely_ unusual balance of classes vs studying. An
'average' class-load is something like 12-18 credit-hours. That's supposed to
mean 12-18 hours of in-class instruction per week. I'm quite curious how you
ended up taking 30-40 hours of in-class instruction, and if that was normal
compared to your peers.

I was told in orientation (in 2003) to expect 2-3 hours of outside work per
classroom hour, and my experience matched that. If these students are facing
the same expectations that I faced, then they're getting a 1:1 ratio of
classroom:outside hours, which is underworking by a significant factor.

~~~
Swizec
I don't remember what I was told in orientation, but most classes ended up
having 2-3 hours of lectures and 2-3 hours of tutorials per week. You were
expected to take 5 classes per semester. This gives you a minimum of 5 _(2+2)
= 20 and a maximum of 5_ (3+3) = 30 hours of classes per week.

Which means my original estimate was wrong, but coupled with holes in
schedules, potentially taking extra classes in a semester (I think the max I
ever did was 7) you easily ended up spending 30-40 hours a week at school if
you were going to all your classes.

That's how it was at my run-of-the-mill European public University. But I did
always get the impression studying in the US was much easier.

~~~
hueving
Sounds about the same. You are just including tutorials which aren't usually
included in regular credit hours in the US. Those would be in the "2-3 hours
per hour in lecture" block.

~~~
Swizec
Well when you have mandatory presence at tutorials you tend to count them :)

------
gshubert17
[2011] -- but still interesting.

