
State education rankings are riddled with methodological flaws - michael_fine
https://reason.com/archives/2018/10/07/everything-you-know-about-stat
======
abathur
I don't have enough familiarity with this sort of research to evaluate it
quickly, but I was a little curious about the authors' backgrounds:

> Stan J. Liebowitz is the Ashbel Smith Professor of Economics at the
> Colloquium for the Advancement of Free Enterprise Education in the Jindal
> School of Management at the University of Texas at Dallas.

> Matthew L. Kelly is a research fellow at the Colloquium for the Advancement
> of Free Enterprise Education in the Jindal School of Management at the
> University of Texas at Dallas.

Here's another piece by them on net neutrality:
[https://www.insidesources.com/state-governments-drop-net-
neu...](https://www.insidesources.com/state-governments-drop-net-neutrality/)

Here's the site for the "Colloquium for the Advancement of Free Enterprise
Education": [https://jindal.utdallas.edu/centers-of-
excellence/capri/cafe...](https://jindal.utdallas.edu/centers-of-
excellence/capri/cafe/)

Here's a press release about the founding of this entity, which includes some
information on where the funding for it comes from:
[https://jindal.utdallas.edu/news/new-program-at-the-
jindal-s...](https://jindal.utdallas.edu/news/new-program-at-the-jindal-
school-advances-freeenterprise-education)

~~~
amadeuspagel
I don't have enough familiarity with this sort of research to evaluate it
quickly either, but the fact that an ad hominem is the top comment, that no
one on HN found any serious criticism is enough for me.

~~~
abathur
I think this is either an over-reaction or a bit disgenuine. I was careful not
to suggest (and certainly not _argue_ ) their research be _dismissed_ for any
ad-hominem reason.

The fact that they are members of some brand-new academic* unit that claims to
be "for the Advancement of Free Enterprise Education" suggests that there's
some non-zero chance they knew the conclusion about public spending on
education before they started.

I raised a small set of facts without making any claim/appeal/accusation of
anything sensational or conspiratorial.

If researchers at the Colloquium for the Advancement of Drink Your Damn Milk
publish a study concluding you should drink your damn milk, it's reasonable to
weight their conclusions accordingly until you see how others with more domain
knowledge assess them.

------
dsfyu404ed
I have several thoughts on this:

The race adjusted performance is interesting and useful but they're basically
using it as a proxy for wealth. Why can't they just use some metric that
directly represents wealth?

If Massachusetts schools are efficiency with money than the system nation wide
is far more screwed up than anyone is willing to admit.

That state ranking depends in part on money spent is abhorrent. That should be
tracked for reasons that should be obvious but not part of a composite score
meant to represents results.

The bit about unions lines up with my anecdotal experience. I worked with the
education department in college and they did not hold teachers unions in high
regard. If your unions are so bad that college professors at a state school in
a blue state gripe about how they hold back progress then I think it's fair to
say your union is pretty bad.

~~~
rayiner
They're not using race as a proxy for wealth, because the two things are
different. There are unique challenges faced by minorities compared to white
americans of the same income level. (Remember, Trayvon Martin was a middle
class black kid whose dad was a white-collar government worker. A big part of
#blacklivesmatter is educating people that black folks face unique problems
that aren't subsumed within the larger issue of income inequality. And as to
Hispanic students the challenges there are obvious--they're much more likely
to be in families that are English as a second language, and also much more
likely to move around a lot.)

~~~
dan-robertson
I’m not convinced that ESL families is the main difficulty facing Hispanic
children. I’ll accept that there are disadvantages to Hispanics but having ESL
parents seems similar to those faced by almost any other child of immigrants.
Perhaps one could argue that due to the visa system, non-Hispanic parents of
Americans tend to have higher earning potential?

I have no idea whether this is true or not

~~~
rayiner
Due to the visa system, immigrants in the U.S. are not identically situated to
each other. Non-Hispanic immigrants are much more likely to already speak
English and have advanced degrees because they come here through visa
categories that specifically select for those things.

~~~
xyzzyz
Sure, but their children aren’t. Children of non-Hispanic immigrants that are
born in the US do better than US-born children of Hispanic immigrants. While
English ability is clearly important in education outcomes, it’s not what
makes the biggest difference.

~~~
rayiner
The level of English proficiency at home has a large impact on the English-
language ability even of children that are born here:
[https://www.ewa.org/blog-latino-ed-beat/us-students-
struggli...](https://www.ewa.org/blog-latino-ed-beat/us-students-struggling-
english-outnumber-kids-born-abroad). _E.g._ a quarter of California's children
are enrolled in programs for English language learners, even though 93% are
native born.

~~~
xyzzyz
I'd be interested in seeing a causative model for this. My guess is that it's
purely Hispanic vs. non-Hispanic distinction.

The Hispanic kids grow up in mostly uniformly Hispanic environments, so they
have less need to learn English, while latter grow up in more diverse
environments. If hardly any neighborhood kids and classmates speak your native
language, you need to learn English to communicate with them, while if you're
Hispanic, chances are you're growing up in a majority-Hispanic neighborhood
and are going to majority-Hispanic school, so you can do mostly fine with only
basic English skill.

The same thing is true about your parents, so poor English proficiency of kids
is not caused by poor English proficiency of parents, but rather both are
caused by the same factor.

------
Loughla
There is no reason given to why they take per-student expense out of the
picture, other than 'we just don't think it should count'. And that seems to
be the entire basis of their finding that rankings are riddled with flaws.

There is much research (once you sort through the partisan fluff from research
institutions with obvious bias) that supports the positive impact of revenue
per student on academic achievement on standardized tests. It's actually funny
this article popped up, I legit just read a paper from 2015 called "A Cost-
Benefit Analysis for Per-Student Expenditures and Academic Achievement". I
genuinely read that this morning. Weird. The authors found that "There was a
significant correlation between revenues available per student and ACT scores
as one outcome measure of achievement." And just to drive that point home,
they replicated the findings from a 2002 study, further solidifying that
sentiment.

And that's just the most recent one I've read. I'm sure there are more recent.
And that is definitely just one in the series of research related to per-pupil
expenditures.

Also, I'm afraid this piece serves no purpose other than to be self-
congratulatory to the 'lower taxes at all costs' group and right-to-work
proponents. Why I say this: research pieces probably shouldn't include snide
comments like

"high-tax, high-spending progressive utopias."

"punishing taxes"

Maybe that's just me? Am I off base?

edit: also, the comment section on that article is awful. Just awful.

~~~
rayiner
> There is much research (once you sort through the partisan fluff from
> research institutions with obvious bias) that supports the positive impact
> of revenue per student on academic achievement on standardized tests.

The article's methodology looks at academic achievement directly, so there is
no need to use spending as a proxy.

~~~
Balgair
Sort-of.

Per my reading of it, they used only one measure, the NAEP. The GP mentions a
paper that used the ACT as a guide. In the least, we should resolve the
difference between the NAEP and the ACT, if there is any. But, like the SAT or
College Acceptance rates, those are just a few _measures_. Achievement is not
easy to define in this context, let alone measure. All the test scores are
just proxies for 'achievement' as a general term. Should we measure household
income at the 10yr post mark too, College _graduation_ rate, number of
pregnancies, marathon runners? It's all just a proxy in the end for trying to
determine, in granular detail, if education is worth spending tax money on.

~~~
haroldp
Aren't these all the same questions we should be asking when we decide to
_have_ public education? When we decide how much we will spend on it? How many
years our kids will go? What will be taught?

What _are_ our goals?

~~~
Balgair
I mean, that's a question humans have been asking for nearly our entire
species' history. Generally, it's a lot more fun to be around 'smarter' people
than 'dumber' ones, so education is somewhat prioritized.

~~~
haroldp
I'm getting at the idea that maybe we haven't been asking ourselves what are
goals are here, and we're at a point where we're doing it this way because
that's how we do it. And the fact that we are still wondering what a decent
way to measure the results of our efforts are is kind of telling.

------
patorjk
I only have 2 data points of experience on this issue. Which I know doesn't
mean much, but I'm left scratching my head after looking at their list.

I spent the first 14 years of my life in Las Cruces, New Mexico (the state's
second biggest city), and the schools I went to were not very good. I did high
school in Maryland and the school system was a lot better (I initially
struggled because I was so far behind everyone else - a friend of mine who
moved to North Carolina for High School told me he had the same problem). In
the author's ranking list, they place Maine at #48 and New Mexico at #41. I
find this very hard to believe. There are so many problems that New Mexico has
that Maine doesn't seem to have (gang problems, drug problems - at least at
the magnitude that I saw in New Mexico). Unless Maine has some pretty bad
schools, this just doesn't add up. US News lists Maine at #6, a ranking drop
of 42 spots seems really significant.

~~~
CBLT
I'm confused by your comment, would you care to elaborate? Your anecdote has
you moving from (the article's) rank 41 in quality to rank 6. You comment that
the rank 6 state is much better than the rank 41 in education quality.
However, you find their ranking hard to believe?

~~~
patorjk
Sorry for any confusion. New Mexico moved from #50 to #41, while Maine moved
from #6 to #48. This raised some flags with me since I went to school in New
Mexico and find it hard to believe they have better schools than Maine,
especially since Maine was previously ranked very high. So my gut feeling is
their is a problem somewhere in how they’re crunching their data (though I’d
just be guessing if I tried to pick apart their methodology).

------
xphilter
I don't really understand removal of graduation rates. Obviously that's meant
to act as a proxy for learning, especially for those 38 states without
proficiency exams.

~~~
ergothus
> Obviously that's meant to act as a proxy for learning

That is the INTENT, sure. But does it work? If you have someone that graduated
from a school A and someone that failed to graduate from school B that is in
an entirely different area, can you draw any meaningful conclusions?
Particularly knowing that school A has no particular requirements on why you
graduate, while school B does?

I'm far more likely to complain about "But when we disaggregate student
performance scores by racial categories (white, black, Hispanic, and Asian),
the rankings change dramatically." I wouldn't want to go too far with that
before ensuring that I wasn't trying to curb swimming deaths by keeping Nick
Cage out of movies:
[http://tylervigen.com/view_correlation?id=359](http://tylervigen.com/view_correlation?id=359)

~~~
kolbe
The data is pretty useful in penalizing states that don't educate their
students well enough to even get to the standardized testing stage.

------
saitoyeung
This is interesting. What I like about it is that it tries to remove any
criteria unrelated to a student's outcomes. Too often we see esoteric ranking
methodologies that, unfortunately, not many people bother to look at very
closely.

I'm the founder of PolarisList
([https://www.polarislist.com](https://www.polarislist.com)), a high school
ranking based on the number of students sent to Harvard, Princeton, and MIT.

This prompted us to take a stab at generating state rankings based on our own
dataset, and came up with the following list. We calculated this by looking at
the number of students in a state who matriculated to the aforementioned
colleges from public high schools divided by the estimated 2017 population:

1 Massachusetts 2 New Jersey 3 Connecticut 4 New York 5 Vermont 6 Maryland 7
Maine 8 New Hampshire 9 California 10 Virginia 11 Alaska 12 Rhode Island 13
Illinois 14 Delaware 15 Pennsylvania 16 Colorado 17 Minnesota 18 Washington 19
Michigan 20 Oregon 21 Florida 22 Georgia 23 Nebraska 24 North Dakota 25 South
Dakota 26 Texas 27 Montana 28 Arizona 29 Wisconsin 30 Missouri 31 Idaho 32
West Virginia 33 Indiana 34 North Carolina 35 New Mexico 36 Ohio 37 Kansas 38
Iowa 39 Hawaii 40 Nevada 41 Kentucky 42 Wyoming 43 Oklahoma 44 Utah 45
Tennessee 46 Arkansas 47 South Carolina 48 Washington DC 49 Louisiana 50
Mississippi 51 Alabama

A couple thing stand out to me: \- All 3 rankings have Massachusetts and New
Jersey within the top 10 \- All 3 rankings have Oklahoma and Louisiana in the
bottom 10 \- Our numbers differ from the other datasets the most on: \- New
York (#4 on our ranking, #30 on Liebowitz/Kelly, #31 on US News) \- California
(#9 on our ranking, #34 on Liebowitz/Kelly, #44 on US News) \- Alaska (#11 on
our ranking, #42 on Liebowitz/Kelly, #46 on US News)

~~~
masonic
A metric showing that students going to northeastern universities mostly come
from northeastern states isn't very meaningful. Given that it counts only
students _attending_ those schools and not all of those _accepted_ by those
schools (e.g. the full population of those meeting the actual qualification
standard) and doesn't account for _ability to pay_ renders it even less
meaningful.

~~~
dorchadas
It's really a horrible flawed metric. It doesn't mention background of
students, nor the connections they might have made in school/life that help
them get to those places. It also doesn't take into account students who don't
_want_ to necessarily go there (I was valedictorian and had a 4.0, 36 on
ACT...I didn't apply Ivy because I didn't want to go there), etc. It's really
just a "Hey, these schools have more kids here" ranking, saying nothing about
quality of the schools themselves.

------
rb808
Ranking is nearly meaningless. eg compare these:

School A - full of underachieving kids that are pushed to get avg grades

School B - full of avg kids getting above avg grades

School C - full of smart kids getting top grades

School D - full of smart kids getting nearly top grades, but soft skills, nice
personalities and VIP friends

School E - Mix of kids getting relevant grades

Which is the best school for each kid type? They're impossible to rank. Many
schools with top grades are only there because they kick out failing kids, not
because they're better at teaching. Many schools are rated good because they
can improve grades, but you wouldn't want to send a smart kid there. Many
private schools sell themselves on creating well rounded kids, not necessarily
good grades.

~~~
educationdata
The article is about ranking states, not ranking schools.

For the point of ranking schools, there are statistical methods to measure
student growth, instead of achievement, such as SGP (student growth
percentile) and VAM (value-added modeling), which kind of address part of your
question.

------
Floegipoky
This may be unpopular but I really do think that spending should be included
in a ranking system. There are a lot of aspects of education that standardized
testing can't (or won't) measure: access to extra-curricular programs, funding
for art/music departments, technical education, etc. I'm willing to bet a lot
of people here have experienced first-hand the difference that having access
to a decent computer lab can make. You're not going to see that reflected in
tests.

Also, based on my own anecdotal experience of standardized testing in grade
school, these tests rarely include questions about "controversial" topics in
history or scientific ideas that are opposed by Christian fundamentalism. I
suspect these are major areas where southern districts would struggle, given
their disproportionate implementation of _cough_ alternative textbooks.

Overall, this smells of a common trend in American institutions where there's
an ideological interest at play. An issue is raised, and it's decided that we
need more data to decide. The data collection (in this case standardized
testing) is designed in such a way that it produces results that skew toward
the ideological position. Then the bad data is used to justify the ideological
position. Law enforcement works the same way. Actually now that I'm thinking
about it so do our elections.

~~~
amadeuspagel
If the purpose of a ranking system is to help decide questions like how much
should be spent on education, then obviously it doesn't make sense to include
spending in the ranking. (And this is not a strawman, the article mentions
that Paul Krugman for example smartly pointed out that states that spend more
have better rankings.)

> There are a lot of aspects of education that standardized testing can't (or
> won't) measure: access to extra-curricular programs, funding for art/music
> departments, technical education, etc. > these tests rarely include
> questions about "controversial" topics in history or scientific ideas that
> are opposed by Christian fundamentalism.

These are legitimate criticisms of standardized testings, but not excuses for
ignoring its results, if you don't have better data available.

> Then the bad data is used to justify the ideological position. Law
> enforcement works the same way.

Maybe instead of using "bad data", like how many crimes are solved, on law
enforcement, we should just ask how much is spent on it - the more the better?

------
ftwynn
The efficiency measure sure seems dangerously misleading. My SC high school
didn't have the budget to replace broken tables (which were simply overturned
until a handy-ish student managed to unsteadily prop them up) and sent seniors
home home early via free periods to cover for a lack of teachers.

Was that efficient? I suppose. Was it a desirable education? Not really.

~~~
fjsolwmv
The argument goes that if those cutbacks didn't harm outcomes, they were
luxuries not necessary expenses.

------
nkassis
I'm wondering if there are confounding variables that are affecting this
result. For example New York and New Jersey are some of the most regressive
states defined as difference in funding levels between rich and poor school
districts. I found this chart which shows each states:

[http://apps.urban.org/features/school-funding-do-poor-
kids-g...](http://apps.urban.org/features/school-funding-do-poor-kids-get-
fair-share/)

If race is a mediator for income levels and in turn funding level shouldn't
the researchers here include the difference in funding levels between income
levels into their model to determine the true performance per spent dollar?

I might be wrong as I'm no expert in the field but I think these type of
problems are what causal models try to address. I've really enjoyed "The book
of why" by Judea Pearl on the topic. It got me interested in learning more
about causality.

------
KaoruAoiShiho
Is there the per-race data available somewhere?

------
lifeisstillgood
Is it just me that has a hard time understanding the racial aggregation thing.
They claim to have split Iowa and Texas results by race and age. And found
that Texas did better than Iowa for whites blacks and hispanics - but then
aggregating the results ala USNews, Iowa comes out on top

I am struggling to imagine which category is the flip category. If iowa is not
doing better for white black or hispanic, which racial category is somlatge as
to flip the average?

~~~
kolbe
Example using two races and just ACT scores:

Texas:

50% Hispanic who average 25

50% white who average 29

State average: 27

Iowa:

10% Hispanic who average 24

90% white who average 28

State average: 27.6

~~~
lifeisstillgood
texas has 50 % hispanic population ?

[https://en.m.wikipedia.org/wiki/Demographics_of_Texas](https://en.m.wikipedia.org/wiki/Demographics_of_Texas)
It has 84% white population, not noticeably different to UK makeup at 87%

Iowa has 88% White
([https://en.m.wikipedia.org/wiki/Iowa](https://en.m.wikipedia.org/wiki/Iowa))
or maybe 91 not too clear what they mean as difference between white and not
hispanic

So them is some pretty small numbers to play with - somehow the "whites do
better, and iowa has more whites" argument has only got 4-7% to play with

I would like to see the actual figures but it smells to me

~~~
lifeisstillgood
OK I read wikipedia wrong - Texas it seems has 84 % white population _but that
includes_ 34% Hispanic (an odd distinction - is that a common US measure?)

Which makes the Simpson's paradox quite possible. So Texas does not have a
racial makeup like UK. (I only have spent a little time in Houston, mostly I
found it populated by cars)

~~~
xyzzyz
The term of art here is "Hispanic white" vs. "non-Hispanic white".

------
exabrial
This is a headline that raises an eyebrow from me because it seems me odd
someone would believe _they are not_ gamed in some manner.

------
temp-dude-87844
Disaggregating the results by race is real cute but also misses the point:
blacks and asians and hispanics are residents of the state too and so leaving
them in to evaluate the state's overall achievement (relative to other states)
in education is fair. If anything it ought to be more problematic that Texas
fails so badly at educating hispanic students when a fifth of them live in
Texas.

Besides, the race in big-gap states is an unwitting proxy to income, so
disaggregate by household per capita income (relative to local cost of living)
if you want to disdain. This methodology would work well in all states and
expose the challenges of anxiety and the quality of a student's home life in
their education attainment.

~~~
rayiner
"Disaggregation" doesn't mean taking them out--the ranking includes everyone:

> Think about that: White students do better in Texas than in Iowa. Black
> students do better in Texas. Hispanic students do better in Texas. Asian
> students do better in Texas. Given these facts, it is absurd for U.S. News
> to rank Iowa higher than Texas in terms of educational performance.

This, of course, makes sense. Students from disadvantaged backgrounds face
unique challenges. Under the traditional methodology, states can do a crappy
job of educating these students so long as they have few of them. States that
have large populations of disadvantaged students, by contrast, are penalized
in the rankings even if they do a relatively better job educating those
students.

~~~
fjsolwmv
Simpson's Paradox is the relevant keyword here. Averaging a across non-
homogeneous groups (multimodal distributions) makes garbage statisics

