
How to Analyze False Claims about Charter Schools - ph0rque
http://dianeravitch.net/2014/02/28/how-to-analyze-false-claims-about-charter-schools/
======
tokenadult
From the article: "So my correspondent–who requires anonymity– decided that it
would be helpful to reporters and members of the public to explain how to read
stories about charter schools." So we don't know who is advancing these
comments. It is plainly not Diane Ravitch herself, and we have no idea who the
person is, so we have no idea if the person suggesting this checklist is a
person with a conflict of interest in education policy.

I'll go down the checklist item by item.

"Does the story compare the demographics of the student population served by
charter schools to the demographics of local public schools?"

What aspect of "demographics" is most relevant here? There are already charter
school programs (not many, but a few) that do better with mostly pupils who
come from poor homes and parents without higher education than many other
schools do with pupils who come from high-income homes.[1] A lot of public
schools in the United States serve up mediocre results while serving well off
pupils.

"Does it include data on the charter school attrition rate?"

This is a fair item to put on the checklist, and should be reported more in
stories about schools in general. In other words, a story about public schools
should also report on their attrition rate, and on what rate at which pupils
move in and out of the school district, and in general how stable or unstable
enrollments are at all schools being compared. (The United States has a highly
geographically mobile population, and it would be a very unusual school that
has the same group of pupils enrolled by sixth grade as was enrolled in first
grade five years earlier.)

"Does it include data on how the students who leave the charters compare to
students who leave public schools?"

Again, this is a question well worth asking, but we shouldn't guess, until we
have seen the answer to questions like this, whether charter schools or public
schools would come out better in a comparison of this kind. We should be
checking the facts in each individual case all around the country.

"Does it include numbers of students expelled?"

Since the early 1990s, I have been aware of homeschooling advocates who talk
about "push outs" (not "drop outs") from the public school system, kids who
are basically told to like it or lump it in the public school system. People
leaving the public school system should always be looked at closely by
researchers to figure out why they left, and whether they are later able to
find a more fitting school situation in which they can learn better.

"Does it include numbers of students suspended?"

As above, this is a question to ask about any kind of school. Moreover, at the
extremes, a school that never suspends any pupil is possibly a school that is
not trying hard enough to maintain a learning environment for all other pupils
enrolled, so a researcher would want to look at grounds for suspension,
procedures surrounding suspension, what corrective steps are taken when a
pupil is suspended, and so on. I've certainly seen suspension abused in public
school settings.

"Does the story focus exclusively on test scores?"

This is a general defect of reporting on schools. At the broad statistical
level, and especially for international comparisons of school systems, test
scores[2] are mostly what we have to look at. What I also look at, as someone
who has lived in more than one country and who reads more than one language,
is the actual item content of textbooks and the attitudes toward learning
shown by teachers and pupils. I think some school systems in other countries
compare very favorably to those in the United States (not least because pupils
in those other countries begin foreign language study much earlier than pupils
in the United States, a fact not usually reflected in international testing
programs). Educational researchers should be looking more closely at the
actual item content of school textbooks and at the specific teacher practices
in each classroom.[3]

"If so, has someone, with educational expertise, visited the school to
determine if the school focuses on test prep at the expense of a rich
curriculum?"

As above, we should be looking in detail at what actually goes on in the
classroom. Indeed, we should be looking at whether persons employed as
teachers are actually teaching anything, which we shouldn't assume sight
unseen.[4]

"Are the test scores reported outside of school assessments such as the
SAT/ACT or does the story only report test scores of exams that are proctored
in-house?"

This is also important, and also a question that should be applied to news
stories about schools in general, not just about charter schools. So far many
of the worst cheating scandals in school testing programs have originated in
regular public schools.[5]

"Does the story account for the fact that, due to the need to apply to the
charter school, parents of the students at charters are, on average, likely to
be more engaged in education than the parents of students at public schools?"

The policy response to this is to make school choices much more widely
available to many more families. I have seen some examples of helpful reforms
where I live. Minnesota, where I now live and where I grew up, has had largely
equal per-capita funding for public school pupils statewide since the 1970s.
The state law change that made most school funding come from general state
appropriations rather than from local property taxes was called the "Minnesota
miracle."[6]

Today most funding for schools is distributed by the state government on a
per-pupil enrollment basis.[7] The funding reform in the 1970s was followed up
by two further reforms in the 1980s. First, the former compulsory instruction
statute in Minnesota was ruled unconstitutional in a court case involving a
homeschooling family, and a new compulsory instruction statute explicitly
allows more nonpublic school alternatives for families who seek those. Second,
the Legislature, pushed by the then Governor, set up statewide open
enrollment[8] and the opportunity for advanced learners to attend up to two
years of college while still high school students on the state's dime.[9]

And Minnesota also has the oldest charter school statute in the United
States.[10] We haven't had big problems with charter school performance here.

Parents in Minnesota now have more power to shop than parents in most states.
That gets closer to the ideal of detecting the optimum education environment
for each student (by parents observing what works for each of their differing
children) and giving it to them by open-enrolling in another school district
(my school district has inbound open-enrollment students from forty-one other
school districts of residence) or by homeschooling, or by postsecondary study
at high school age, or by exercising other choices.

The educational results of Minnesota schools are well above the meager results
of most United States schools, and almost competitive (but not fully
competitive) with the better schools in the newly industrialized countries of
east Asia and southeast Asia. It's a start. More choices would be even better.

"Does it exclusively or primarily cite reports funded by pro-charter or
conservative think tanks?"

Here it's regrettable that we don't know the identity of the author of this
checklist. Do we know whether or not the author is funded by pro-public-school
interest groups? Disclosure of that would be helpful here.

"Does it include quotes from academic scholars or does it just cite charter
school advocates?"

I can recommend a website full of the writings of actual academic scholar to
Hacker News participants interested in education reform. That is the
_Education Next_ website,[11] which is better supplied than most English-
language websites about education policy with information about education
policy outside the United States, which is important for a reality-check on
issues discussed in this country.

"Does it identify advocates or simply call them 'experts' or 'researchers'?"

This is a funny question for a commentator who insisted on anonymity to ask.

"Does it compare the resources available to charter schools to those available
to public schools?"

In Minnesota, and in all states I have checked, the resources available to
charter schools (funding per student, and availability of buildings) are
strictly less than the resources available to public school disticts, by law.
Yes, lets's check that detail carefully.

It's commendable to receive a suggestion that we should check news stories
carefully. We should always do that here on Hacker News. Let's check stories
on education policy especially carefully, as public school spending is a huge
part of state budgets[12] and a lot of interest groups are lined up behind the
public trough.

[1] [http://educationnext.org/when-the-best-is-
mediocre/](http://educationnext.org/when-the-best-is-mediocre/)

[2] [http://www.oecd.org/pisa/](http://www.oecd.org/pisa/)

[http://timssandpirls.bc.edu/](http://timssandpirls.bc.edu/)

[3] [http://www.amazon.com/Knowing-Teaching-Elementary-
Mathematic...](http://www.amazon.com/Knowing-Teaching-Elementary-Mathematics-
Understanding/dp/0415873843)

[http://www.timssvideo.com/timss-video-study](http://www.timssvideo.com/timss-
video-study)

[4] [http://www.pbs.org/newshour/bb/students-take-a-stand-to-
topp...](http://www.pbs.org/newshour/bb/students-take-a-stand-to-topple-
teacher-tenure/)

[5] [http://www.ajc.com/s/news/school-test-scores/#top-
stories](http://www.ajc.com/s/news/school-test-scores/#top-stories)

[http://online.wsj.com/articles/for-school-tests-measures-
to-...](http://online.wsj.com/articles/for-school-tests-measures-to-detect-
cheating-proliferate-1411752291)

[http://www.newyorker.com/magazine/2014/07/21/wrong-
answer](http://www.newyorker.com/magazine/2014/07/21/wrong-answer)

[http://www.washingtonpost.com/blogs/answer-
sheet/wp/2014/08/...](http://www.washingtonpost.com/blogs/answer-
sheet/wp/2014/08/11/50-ways-schools-cheat-on-high-stakes-standardized-tests/)

[6]
[http://www.mnhs.org/library/tips/history_topics/18public.htm...](http://www.mnhs.org/library/tips/history_topics/18public.html)

[7]
[http://www.house.leg.state.mn.us/hrd/pubs/mnschfin.pdf](http://www.house.leg.state.mn.us/hrd/pubs/mnschfin.pdf)

[http://education.state.mn.us/MDE/SchSup/SchFin/index.html](http://education.state.mn.us/MDE/SchSup/SchFin/index.html)

[8]
[http://education.state.mn.us/MDE/StuSuc/EnrollChoice/index.h...](http://education.state.mn.us/MDE/StuSuc/EnrollChoice/index.html)

[9]
[http://education.state.mn.us/MDE/StuSuc/CollReadi/PSEO/index...](http://education.state.mn.us/MDE/StuSuc/CollReadi/PSEO/index.html)

[10] [http://www.amazon.com/Zero-Chance-Passage-Pioneering-
Charter...](http://www.amazon.com/Zero-Chance-Passage-Pioneering-
Charter/dp/1592984762/)

[11] [http://educationnext.org/](http://educationnext.org/)

[12]
[http://nces.ed.gov/fastfacts/display.asp?id=66](http://nces.ed.gov/fastfacts/display.asp?id=66)

[http://www2.census.gov/govs/school/12f33pub.pdf](http://www2.census.gov/govs/school/12f33pub.pdf)

~~~
japhyr
What's wrong with evaluating the framework (questions) she proposes on its own
merits? These questions strike me as critical thinking questions, but not
biased questions.

~~~
tokenadult
As a courtesy to you and to onlookers, I should point out that your very fair
reply to the original short version of my comment was posted while I was still
expanding the comment with my point-by-point reply to the questions posed by
the anonymous correspondent to Diane Ravitch. You are quite correct that we
should grapple with each question in the checklist in detail. I hope I have
done so in the expanded version of my grandparent comment.

------
simon_
If we stipulate that the charter schools in question have strong selection
biases in their student bodies and better resources relative to public
schools, then:

a) It's true that statistics are frequently misused by journalists and
advocates, BUT:

b) It hardly impugns the schools themselves. Taking a subset of more promising
students without behavioral issues out of failing public schools and giving
them better resources at charters is not necessarily a bad idea.

~~~
mazelife
The article directly addresses why it's a bad idea: "it makes little sense for
the district to heavily subsidize schools [i.e. charters] serving less needy
children that already have access to more adequate resources. It makes even
less sense to make these transfers of facilities space (or the value
associated with that space) as city class sizes mushroom and as the state
indicates the likelihood that its contributions will continue falling well
short of past promises."

Also, the article mentions that many of the charters have suspension rates
between 20 and 50%, which suggests that either a.) the students at these
schools _do_ have behavioral issues and/or b.) the charters use aggressive
suspension policies to get students to leave.

~~~
jack-r-abbit
Heavily subsidize? Last year, the charter school my kids attend got $1300 less
per student than the non-charters in the district. The school has roughly 1200
students. That means we probably _saved_ someone (district? state? not exactly
sure the break down of funding sources) roughly $1,560,000 that year over
having all those students in the non-charter schools. At the very least, the
other non-charter schools got more (per student) than they would have
(assuming there isn't a magical $1.56M sitting around, the schools would have
received somewhere in between the two numbers for ALL students).

Transfers of facilities space? The school recently took out a loan and bought
an empty business complex to renovate as our new campus because the campus we
were using was an old run down school that the district had already closed
several years prior to us moving in. They closed it (and two others in the
district) due to budget cuts. Closing these schools caused teachers to lose
their jobs & class sizes at the remaining schools to increase.

------
clavalle
Basically the author is claiming that we have to hold all variables equal to
compare charter schools and traditional public schools in order to 'be wise'
and 'think critically' about charter schools. Why is that necessarily the
case?

What if the expulsion rate (dropping disruptive students) and parent
engagement is part of the 'secret sauce' of charter school success? Why should
data that is based on these factors be considered 'bogus statistics'?

There is some good, common sense advice in this article but it considers too
many fundamental factors 'bogus'.

This is a bit of a tangent but my fundamental problem with the traditional
public school system is this: why should my child's education be primarily
determined by where our domicile happens to be located?

~~~
scott_s
To answer your first questions: because in the US, we typically assume our
mandate is to educate _everyone_. If the "secret sauce" is to marginalize the
students who are in the most need to help, then that does not achieve our
goal.

~~~
clavalle
Why is it assumed that to educate everyone to the best of their ability that
the traditional public school system, where everyone is thrown together based
on where they live, is the best to achieve that goal and anything else leads
to marginalization?

I used to work at educating blind children. Most of those I worked with also
had other disabilities as well. The school I worked at was a boarding school
and first rate. It concentrated experts into one place and offered continuity
of instruction beyond anything a public district could provide. Student
progress was, on the whole, amazing compared to their progress at a public
school. This is an extreme example, granted, but why wouldn't a similar
approach work with other kids who need extra help and attention? Why not have
behavior specialists at one charter school working with disruptive kids and
engineers teaching kids that are interested in STEM?

~~~
scott_s
We already have that kind of a model in public schools. The difference between
the two is money.

Public schools have special education teachers, and sometimes have entire sub-
schools within existing schools just for special education kids. (My mother
was a special education teacher.)

Since your school was a boarding school, I imagine it also cost significantly
more per-child than we currently spend in public school. A lot of the gain
from your experience, I think, was probably from having good instructors
trained in how to teach students with those particular disabilities, and
probably a good teacher-to-student ratio. If we were willing to pay public
teachers more, public schools could attract the same experts, and if we were
willing to employ more public teachers in total, we could achieve a similar
ratio. We could then achieve the same thing in public schools as in the
boarding school.

~~~
clavalle
>We could then achieve the same thing in public schools as in the boarding
school.

No we couldn't. There are only so many experts to spread around.

------
simeonf
Sentence two of the expert advice is propaganda.

>The majority of these stories praise charters, while often demeaning public
schools.

Charter schools _are_ public schools - they are just less regulated ones.

~~~
phkahler
>> Charter schools are public schools - they are just less regulated ones.

No, they are not. They are run by private corporations. That's like saying
McDonalds is a public restaurant because they are open to the public. Please
don't mix common understanding of what words mean with some plausible other
use of those words, it does not actually help your cause.

~~~
wil421
Not sure where you live but in Georgia charter schools are not run by private
corporation like McDonalds. They take public funding just like every school
but have more freedom in their curriculum.[1] They are not governed by a board
of education but by a non profit board with less rules but higher
accountability by the state.

[1][http://www.gadoe.org/external-affairs-and-policy/charter-
sch...](http://www.gadoe.org/external-affairs-and-policy/charter-
schools/pages/general-frequently-asked-questions.aspx)

~~~
depingus
Charter schools are run by for-profit management companies.

[http://www.miamiherald.com/news/special-reports/cashing-
in-o...](http://www.miamiherald.com/news/special-reports/cashing-in-on-
kids/article1939199.html)

------
snake_plissken
>Reports and stories about charter schools are in the media every day. The
majority of these stories praise charters, while often demeaning public
schools.

Interesting. My experience with how the media presents charter schools is
quite the opposite. It seems like every few weeks there is a story about a
corrupt charter school management team embezzling funds or over-enrolling of
students or suspect academic results. Of course with stories on actual events,
such as an investigation into missing funds, it's difficult if not downright
impossible, if a media outlet so desired, to spin the story into something
positive. But I don't recollect reading anything within the past few years
that actually praised charter schools, and that's not to mean the traditional
public school system is squeaky clean and a model of best practices. I suspect
there is an equivalent level of corruption there. I've wondered if it's
because I live in Philadelphia where this a strong teachers union and a
traditional public school system that is, for all intents and purposes,
bankrupt. However, I would expect that these conditions would lead to the
opposite in how charter schools are portrayed, if only for the sake that they
are an alternative to the current system that is failing miserably.

The charter school debate is a sub-segment of a larger issue, which is this
push towards the privatization of _everything_ that is consumed by the public
and paid for by tax dollars, from public school systems to prisons to health
care programs, such as Medicaid. Personally, I disagree with this push. It
creates perverse incentives and from a political economy point of view, I
think there are some goods which can't and should not be privatized, but that
discussion is for another time.

------
crazy1van
The public school system isn't exactly stellar. I don't claim to know why, but
I don't see how competition does anything other than make the whole system
stronger. Why so much fear if you are confident you have a superior product?

~~~
GabrielF00
It's not clear to me that competition for students does make the whole system
better.

You end up with a phenomenon where some schools get used as dumping grounds
for the kids that the other schools don't want to deal with or whose parents
don't care or don't know any better. I witnessed this when I had a part-time
teaching job in the Boston Public Schools. I spent a year at Jeremiah Burke
High School, which has the reputation as one of the worst in the state. A huge
proportion of kids there either did not speak much English (Cape Verdean
Creole was very common), or had severe learning or behavioral issues.

The problems that exist in public education are incredibly complex, and it's
not clear to me that any simple, ideologically-tinged solution is going to
solve them.

~~~
crazy1van
> The problems that exist in public education are incredibly complex, and it's
> not clear to me that any simple, ideologically-tinged solution is going to
> solve them.

This is exactly why I want competition. A few experts sitting around a
conference table for a year are not going to just deduce the solution, just
like the best VCs invest in many companies instead of just picking only the
Facebooks. These problems are far too complicated for a centrally architected
solution. In the end the best solution might be one that most people never
even imagined.

------
japhyr
Much of the conversation in this comment thread assumes a public vs. charter
school split. If my understanding is correct, these charter schools are public
schools. The distinction is that the charter schools have selection criteria.

A public school with selection criteria is not, by itself, a bad thing. I love
that schools like Stuyvesant HS and Bronx School of Science exist, but that's
because we all know what their selection criteria are.

I don't like charter schools pretending that the only thing it takes to be
part of the school is winnning an entrance lottery. That might be the main way
to get in, but it's not all it takes to stay enrolled. Many charter school
advocates seem to be ignoring, or ignorant of, the fact that these schools
actively weed out students who are difficult to reach.

It's okay to build some schools that require a more serious commitment to
learning; it's not okay to build schools like this and pretend they serve
everyone, or do a better job than schools who have to keep every eligible
student enrolled.

~~~
tokenadult
Charter schools by law do not have selection criteria. The example you
mention, the Bronx High School of science, is a public school with an entrance
examination, usually called an "exam school" for short. No charter school
anywhere in the country can operate on that basis, as all charter school
statutes everywhere explicitly ban examination requirements for entrance.

~~~
japhyr
Thanks for the clarification, I'm glad I qualified that comment!

------
mbubb
This is really interesting - to be honest I am at work and am resisting being
pulled into reading this until tonight...

I have a good friend who I have had since college days. Has been an NYC public
school teacher for a while. Loves to teach - onel of those folks that is
constantly formulating ideas, reads voraciously, argues passionately, etc.

I see him as caught between the attempt to corporatize education and a
bloated, syphillic (sp) union... My ideas not his.

On my side I had kids in public school until 2nd grade where I saw disturbing
things that made me jump ship to a charter where my boys thrive.

This is anecdotal - I do not want to try and prove/ disprove anything by this.

I wish we had a viable pub school option locally but we don't at the moment.
Our school survived a scummy attack on charters locally in which false claims
were made about segregation. This was disproven but we were made aware of the
fact that the other side has more resources and deeper legal pockets then the
charter school does.

As I said I will sit down and read this later. I am interested in something
that comes up when my buddy and I talk about this stuff.

Somewhere I picked up a concept from French which I will anglicize as
"Deformation Professionelle" (my apologies for butchering it - I will properly
google it later).

ed.
[http://en.wikipedia.org/wiki/D%C3%A9formation_professionnell...](http://en.wikipedia.org/wiki/D%C3%A9formation_professionnelle)

I see this when tech and teachers discuss education.

I sit and talk to him about tech issues - 'net neutrality', privacy issue,
(going back a decade) attacks on open source software, etc. And he doesn't
know about these things.

And I (at times) wax enthusiastic about things like Khan Academy or charter
schools without seeing possible drawbacks.

I look forward to giving this a proper read later - am glad to see this on HN.

------
squozzer
Not sure why the author would bother analyzing claims she has already decided
were false.

~~~
crusso
Because it's a political hit piece, designed to help nip this nasty threat to
the established public education power structure in the bud.

At the end of the day, school choice doesn't matter. The fact that Charter
schools are relatively new and just finding their way doesn't matter.

I'm all for apples to apples comparisons. But when you have a huge basket of
fruit you're trying to normalize to apples, it's really telling which ones are
picked.

~~~
pdabbadabba
Your comment would be more persuasive is it were accompanied by evidence that
the errors described in the article were not, in fact, pervasive and that
there were real data to indicate that charter schools outperform ordinary
public schools. As it is, your comment reads like a similar attempt to nip a
nasty threat to charter schools in the bud.

~~~
crusso
Your comment indicates that you didn't really even read or attempt to think
about mine. You're asking me to nitpick statistics that frame the exact
worldview the article was pushing.

I'm saying that in the big picture, those probably aren't the specific
statistics that many people even care much about at this point.

It's like an article comparing a Tesla to a Ford pickup truck that focuses
solely on 0-60 speed and fuel costs but never bothers talking about towing
capacity, load capacity, ability to refuel anywhere, etc.

~~~
pdabbadabba
I still don't get it. The article only points out that, in determining whether
a Charter School provides a superior education to ordinary public schools
(which is the claim made by the Economist and many other sources in the media)
we need to know more than raw test scores. We need to control for a number of
variables, such as the resources of attendees, etc.

If Charter Schools are attended by (e.g.) wealthier students than public
schools, or students with more involved parents, then a charter school's
superior aggregate performance on a standardized test tells us precisely zero
about the educational quality of the school unless one can somehow compensate
for this confounding variable. Yet this is routinely overlooked in the media.

You seem to be saying that we shouldn't care about controlling for variables
like this and should, instead, look at other ones. OK. But you don't tell us
why, or what statistics would be preferable, or give any evidence to suggest
that publications like the Economist actually consider your preferred measures
in lieu of those suggested by the article (remember: the article is about how
to evaluate the claims made in the media about charter schools, not about the
merits of charter schools themselves -- though I readily grant that the author
clearly has a view on the latter topic as well).

~~~
crusso
Meh, I'll let TokenAdult post the very long and detailed refutation of the
statistics themselves and the validity of the questions being asked.

 _You seem to be saying that we shouldn 't care about controlling for
variables like this_

I'm saying that those variables and the questions that this "expert" chose to
put forth miss the mark on the questions that should be asked and the
variables that should be examined.

 _But you don 't tell us why, or what statistics would be preferable_

I did tell you why. My guess is that if you thought that choice was important
in education then my thinly veiled sarcasm about choice would have caught your
attention.

Regarding statistics, the statistic alone that typically Charter schools
receive less per student than mainstream public schools renders questionable
every other statistic that this article mentioned.

------
xname
The author is silly. Studies have been conducted to account for selection
bias. Because, by law, students have to be randomly selected to get into
charter schools with lotteries, it is not hard to control for the selective
effects. Check this:
[http://ies.ed.gov/ncee/pubs/20104029/](http://ies.ed.gov/ncee/pubs/20104029/)
and this [http://seii.mit.edu/wp-content/uploads/2013/10/Charter-
Schoo...](http://seii.mit.edu/wp-content/uploads/2013/10/Charter-School-
Demand-and-Effectiveness.pdf)

Also, I don't understand why so many leftists hate charter schools so much.
First, charter schools are public schools, not private schools. Second,
charter schools are created to help the poor, not the rich. What's wrong with
those leftists? I just can't understand. Do you really love teachers union so
much that poor kids mean nothing to you?

~~~
depingus
>First, charter schools are public schools, not private schools.

Charter schools are funded with public money, but they are run (and often
owned) by for-profit management companies.

[http://www.miamiherald.com/news/special-reports/cashing-
in-o...](http://www.miamiherald.com/news/special-reports/cashing-in-on-
kids/article1939199.html)

Many people see them as just another way to get public money in the hands of a
few private companies.

~~~
mattmcknight
public means open to the public for free.

