
The reviewer’s fallacy: when critics aren’t critical enough - wormold
http://www.slate.com/articles/arts/books/2018/01/the_reviewer_s_fallacy_when_critics_aren_t_critical_enough.html
======
duxup
Roger Ebert used to call things "a great entertainment". The author of the
article uses that term too. When Roger said it I always understood it to mean
that there was nothing special about the film.... but it was entertaining.

It felt like he was identifying a separate category of films. It's not really
great among film, but it gets its own job done "entertainment", and in that
way it is plenty good. It leaves room for praise for what it is, but also
space for the few really "great" things.

~~~
lobster_johnson
Ebert's reviewing philosophy was to review a film relative to its ambitions,
genre and target demographic. If the film accomplished what it apparently set
out to do, Ebert viewed it favourably. Many of his highest scores were for
pure, disposable entertainment, as opposed to high-brow arthouse film such as
Bergman or Antonioni.

~~~
SideburnsOfDoom
> review a film relative to its ambitions, genre and target demographic

So long as those assumptions are also stated in the review, that seems a good
idea.

------
alexkavon
I think Ash Williams (Bruce Campbell) said it best in season 2 of Ash Vs. the
Evil Dead, something along the lines of, "critics have shitty taste."

I blame mostly the mindset that a job like that can create. A critic/reviewer
is often pressured to agree with pop culture in order to have their opinions
validated. Movies and music are a good example of this ( _cough_ Star Wars
_cough_ ). Actually look at most critics vs audience aggregations on Rotten
Tomatoes or the fact that very few outfits even write music reviews anymore
beyond Pitchfork.

There are very few outliers in this case and are often viewed as malicious,
for example Jim Sterling.

I personally never read reviews before I ingest some entertainment I have
interest in. At most I read the closing paragraph of a review which is likely
to contain no spoilers and sums up the critics opinion. Scores are irrelevant
to me as well, I have my own scale.

~~~
ClassyJacket
>very few outfits even write music reviews anymore beyond Pitchfork.

The is probably a pretty natural result of it now being so easy to simply
listen to music you're curious about. Reading a review of an album will take
as long as just going to YouTube or Spotify and listening to the first song.

~~~
zelos
On the flip side, for most consumers there are probably several million
albums/television series/songs/films a few clicks away. Choosing is still
hard.

------
yesenadam
>"Theodore Sturgeon, who once observed, “It can be argued that 90% of film,
literature, consumer goods, etc. is crap.” The “It can be argued” part usually
isn’t quoted, and the figure is very ballpark."

Actually it sounds like the author wasn't critical enough. They believed
wikipedia. Strange. But I remembered reading about this recently, that
Sturgeon used the word "crud", not crap.

The rest is pasted from wikiquote:

"I repeat Sturgeon's Revelation, which was wrung out of me after twenty years
of wearying defense of science fiction against attacks of people who used the
worst examples of the field for ammunition, and whose conclusion was that
ninety percent of it is crud.

The Revelation: Ninety percent of everything is crud.

Corollary 1: The existence of immense quantities of trash in science fiction
is admitted and it is regrettable; but it is no more unnatural than the
existence of trash anywhere.

Corollary 2: The best science fiction is as good as the best fiction in any
field. - _Venture Science Fiction_ (March 1958)

The original expression of this has often been declared to have been "Sure,
ninety percent of science fiction is crud. That's because ninety percent of
everything is crud." According to Philip Klass Sturgeon made the remark during
a talk at New York University around 1951. It has also commonly appeared in
variant forms such as "Ninety percent of everything is crap" and is often
referred to as "Sturgeon's Law" — though he himself gave that title to another
phrase:

Sturgeon's Law originally was "Nothing is always absolutely so." The other
thing was known as "Sturgeon's Revelation".

~~~
yesenadam
Wish I'd noticed before that about 90% of the sentences I quoted at the top
are crap! They got the name right and the number.. apart from that, nothing.

------
indubitable
Interesting point the article indirectly hits on:

\- _" Chummy logrollers—a perception heightened in the social media age. In a
2012 Slate piece called “Against Enthusiasm,” Jacob Silverman wrote, “if you
spend time in the literary Twitter- or blogospheres, you’ll be positively
besieged by amiability, by a relentless enthusiasm that might have you
believing that all new books are wonderful and that every writer is every
other writer’s biggest fan.”"_

I've always assumed studios had simply managed to, through some means or
another, 'corrupt' most reviewers in a process similar that happened to video
games. I wonder how much of the problem might be that people are now just too
close to the things, and in particular the people behind the things, that
they're reviewing? It's far easier to be "critical" (read: truthful) about
things and people you have no connection to. But in today's world of social
media, reviewers and the creators of the products they're reviewing are going
to be in near direct, if not indeed direct, communication. Knowing full well
that speaking honestly about a product could hurt people you've formed
relationships with is not something easy to do.

For megafilms like "The Last Jedi" (currently at 90% on rotten tomatoes
contrasted against 49% for viewers), how many of the reviewers have at least
one direct relationship to somebody involved with the project? It doesn't need
to be corruption. Plain old reluctance to be honest to those close to us or do
them soft favors with no expectation of direct personal benefit -- that basic
aspect of human nature would go a long ways towards explaining the
deterioration of honesty in reviews. Or taking it to another level, we often
wear rose colored glasses towards those close to us. This entire process might
be entirely subconscious.

~~~
iainmerrick
_megafilms like "The Last Jedi" (currently at 90% on rotten tomatoes
contrasted against 49% for viewers)_

Be careful, The Last Jedi is definitely part of the "culture wars" effect
somebody mentioned elsewhere. I don't think it's a very useful example for the
effect this article is talking about (although it's very useful if you're
interested in the culture wars).

If you actually read the Rotten Tomatoes reviews, many of the "viewers" are
people complaining about gender representation. It's not a representative
sample, it's just been flooded by "men's rights" activists.

Personally, I think the 90% is too high (I didn't like it very much) but the
49% is way low (most people I know liked it just fine, no quibbling).

~~~
Goladus
_Be careful, The Last Jedi is definitely part of the "culture wars" effect
somebody mentioned elsewhere._

I don't buy it. _The Force Awakens_ is at 88% and culture warriors fought over
that one, too. Both films have severe problems from a storytelling
perspective, but the problems with _The Last Jedi_ are much deeper, are of the
sort more likely to turn off fans of the franchise as well as fans of _The
Force Awakens_. There are plenty of culture-war-neutral analyses of _The Last
Jedi_ that absolutely eviscerate the film. For just one example:
[https://www.youtube.com/watch?v=vw7pcCj0ORk](https://www.youtube.com/watch?v=vw7pcCj0ORk)

While I can't speak to the 49% rating, where I'm seeing the major divide is
between Star Wars fans who care about world building, character development,
and a coherent story (historically important aspects of Star Wars); and fans
who are content just to enjoy the emotional ride from one scene to the next
while enjoying the nostalgia.

As from the culture wars perspective, it seems hard to deny that _The Last
Jedi_ takes an unnecessarily overt stance on the culture war, daring that sort
of criticism. TLJ includes: a postmodern deconstruction of Luke Skywalker and
the Jedi mythos, a purple-haired female admiral dominating the only talented
white male on their side for displays of what they probably imagine to be
toxic masculinity, and all the bad guys are white males with overt Nazi
references. I mean, seriously, why can't they just make it Star Wars?

Take another Disney movie as a contrast: Moana. Moana is a hero's journey,
just like the original Star Wars. Moana is about a girl character, her mentor
is her grandmother, the story is set in the South Pacific and features non-
European cultures and sources their mythology for the story. The twist at the
end might be considered feminist. But nobody but serious cranks are
complaining about it, because none of it gets in the way of the story and for
the most part actually enhances it. Moana isn't made some kind of wunderkind.
There's no artificially forced interaction between an SJW stand-in and a white
male stand-in. There's no postmodern deconstruction of the myths or themes.
Disney is just doing their usual thing of mining ancient myths and fairy tales
and adapting them for entertaining mass consumption by modern audiences.

~~~
iainmerrick
I think you went on a little too long there.

~~~
Goladus
I had 3 points to make. Was it too much for you? Frankly I could have written
a great deal more.

1\. There's excellent non-culture war criticism of the movie.

2\. TLJ made its own bed with regards to the culture war.

3\. Disney has demonstrated they can make progressive movies that don't piss
off culture warriors.

------
ebbv
A simpler and obvious explanation for critical complacency; 90% of critics are
also crap/crud.

~~~
stochastic_monk
Or at least, the vast majority of published reviews are.

Recently, I was reading a review of Samuel Beckett [Nobel Prize in Literature
1969], from the NYTimes from 1958 [0], and at first I was shocked by how much
higher the quality was than what one might read in the Times today.

Turns out that the author was a Stephen Spender, CBE, [1], a poet who was
knighted for his contributions. It's not when or where something is written so
much as by whom. You are exactly right that it's determined by who the critic
is.

[0]: [http://www.nytimes.com/books/97/08/03/reviews/beckett-
unnama...](http://www.nytimes.com/books/97/08/03/reviews/beckett-
unnamable.html)

[1]:
[https://en.wikipedia.org/wiki/Stephen_Spender](https://en.wikipedia.org/wiki/Stephen_Spender)

------
j9461701
An interesting article, most certainly.

I suppose the immediate response is the critic sphere is heavily self selected
for people who _really_ like books / movies / comics / etc. So what to most
normal people would seem like flowery platitudes and a cultural aversion to
tough talk might actually be a completely genuine love of the medium itself.
If you just love to read, it's not vapid pleasantness to call every book fun
and a joy - it's just how you genuinely feel.

>I hate to sound like a philistine, but audience-critic discrepancies often
occur when a work is less than pleasant to sit through, whether because of The
Sorrow and the Pity–like length (a growing problem, pun intended) or grim
subject matter. Take last year’s Best Picture winner, Moonlight, which has a
98 critics’ and 79 audience score on Rotten Tomatoes, and which I haven’t
seen. The Rotten Tomatoes blurb calls it “The tender, heartbreaking story of a
young man’s struggle to find himself, told across three defining chapters in
his life as he experiences the ecstasy, pain, and beauty of falling in love,
while grappling with his own sexuality.” I can get that at home.

I think this is another element of the critic sphere being very self selected.
For whatever reason it selects for literary realist types who honestly care
more about depressing stories featuring puppies with broken legs than any kind
of sci-fi, fantasy, etc. genre film that's merely fun and not "meaningful".
The critics aren't behaving fallaciously or malevolently, they as a group
simply have genuinely different preferences for entertainment.

>Here’s the heart of the problem: The set of critics’ and audiences’ interests
do not perfectly overlap but rather form a Venn diagram.

Ultimately, this is the real problem whatever is causing the divergence in
interests. It makes it hard to use critics, and harder to identify the genuine
diamonds in the rough for you personally.

My personal solution is simply look to the average person directly. The
internet is full of forums, and many of them discuss whatever medium you
happen to care about, and those forums are invariably going to have plenty of
threads along the lines of "What is your favorite lost gem?", "What movie do
you think flew under the radar?" "What deserved more credit than it got?".

~~~
Fnoord
> Ultimately, this is the real problem whatever is causing the divergence in
> interests. It makes it hard to use critics, and harder to identify the
> genuine diamonds in the rough for you personally.

> My personal solution is simply look to the average person directly.

IMDB is basically a layman's scale vote (from 1 to 10) while mentioning
Metacritic's average critic review, whereas Metacritic contains both critic
review and user review. You can pick either on Metacritic, but if you _never_
care about critic review then perhaps IMDB is a better solution. I tend to use
both since IMDB has more titles. Especially old, obscure, and international
titles.

------
pessimizer
It might be simpler to say that the interaction between the reviewer's
motivation to believe that 50% of everything is crap and the fact that 90% of
everything is crap has emergent effects.

The major one is that bad qualities that randomly happened to occur more
commonly in products that have previously made the 50% cut become
misindicators of quality, distorting the evaluation of a constantly drifting
subset of products that have or lack those misindicators. This occurs
asymetrically (between misindicators of positive quality and misindicators of
negative quality) because there are more products misidentified as being good,
therefore it's relatively far more common for particular bad qualities to
randomly be more associated with works above the median then the opposite.
Works that fall below the median in a world of 90% crap would have very few
good qualities at all.

------
BosunoB
This isn't a completely sufficient argument, though. Surely, any person into
an artform will at some point be able to recognize the difference between what
is good because it is good and what is good because it is different. A good
critic would be able to differentiate these two things and give a review with
some nuance, but it seems incomplete to say that the critic problem is mostly
the fault of seeing way too much junk. Anybody who is actually in to anything
has seen a ton of junk, but not everybody has lost their ability to clearly
judge things.

------
hitekker
Barely related: Rotten Tomatoes summaries feel like they've been paid for.

Sure, if a movie is rotten, they'll trash it. But when a movie is barely
fresh, they seem to accord it more praise than the critics gave themselves. I
don't have a particular instance in mind so I'd be curious if anyone else has
noticed this.

~~~
cf498
I dont think its that straight forward, but there have been instances lately
where they behaved rather calculated

[http://www.chicagotribune.com/entertainment/movies/ct-
rotten...](http://www.chicagotribune.com/entertainment/movies/ct-rotten-
tomatoes-justice-league-review-20171117-story.html)

------
HarrietJones
If there's more than 20 points between critic score and viewer score (in the
viewers favor) on Rotten tomatoes, then the movie is usually good. There are
exceptions to the rule, and recent inclusions in the culture wars (Justice
League, Ghostbusters) can break this, but generally it's a good rule of thumb
to follow. Reviewer scores come from people forced to watch hundreds of films,
and as a consequence they reward poor quality idosyncrocy over good quality
but predictable tropes. People crave variety, and when you review movies for a
living, you lose touch of what variety is for normal people.

------
partycoder
See also:
[https://en.wikipedia.org/wiki/Groupthink](https://en.wikipedia.org/wiki/Groupthink)

From the article:

> Groupthink is a psychological phenomenon that occurs within a group of
> people in which the desire for harmony or conformity in the group results in
> an irrational or dysfunctional decision-making outcome. Group members try to
> minimize conflict and reach a consensus decision without critical evaluation
> of alternative viewpoints by actively suppressing dissenting viewpoints, and
> by isolating themselves from outside influences.

------
pedz88
You have touched on one of the main reasons we decided to launch www.bad-
about.com. In a world full of positive hype from reviewers and sales spin from
the companies themselves, we thought it would be helpful to collect and
display JUST the criticisms to help balance it all out.

------
crondog
Is there a service like this: rate a bunch of media, then filter for critics
that largely share your preferences, then subscribe to recommendations from
just those critics? If there isn't, there totally should be.

~~~
jacobush
There is but it's from a limited selection of media and you can't know the
identities of the reviewers and the set of reviewers changes all the time, as
well as the set of available media. (I am talking of course, about Netflix.)

~~~
sgt101
Also by consuming media preferences could/ should be changed, also new
friends, change of life and political context... I don't think that jamming
people in a bubble is a good plan

~~~
crondog
everybody's always in a bubble, since there is a finite number of hours in the
day and far more media than anyone can consume. The best we could hope for is
to personalize the bubble while minimizing the influence of outside interests
with divergent goals - e.g. advertisers and spammers.

I imagine the change of preferences can be tracked over time by periodically
rating new works I consume and having critics fall in and out of the personal
overton window.

~~~
jacobush
The best for... what. Some of the best movie experiences I've had was when I
saw something someone else picked, something I would never have picked on my
own.

~~~
crondog
let me counter your anecdote with mine: I saw something somebody else picked,
and it was complete trash. In fact, this is trivially the case every time I
see anything awful that has at least one positive review. Judging by, say, RT
page for the last jedi, this is a common experience.

It's all about the signal to noise ratio. It is more likely that you'll enjoy
a work that was enjoyed by people similar to you. It is also possible that
you'll enjoy a work recommended solely by some random pseudonym on the
internet, but less likely.

~~~
jacobush
ahh.. there are more parameters I didn't consider:

* high median (many good films, low tolerance for trash)

* finding jewels (can wade through lots of trash to find that outstanding film)

* high baseline (a minimum quality is important, no trash please)

