
Why Wine Ratings Are Badly Flawed - tshtf
http://online.wsj.com/article/SB10001424052748703683804574533840282653628.html
======
ericb
A good friend teased her husband for spending only $7 for a bottle of wine in
a facebook status post. So I bet her I would eat a cork if she could
consistenly tell the difference. I won.

We held a blind taste test for 12 people with 14 wines ranging from 3.99 to
$50. For each variety, there was one wine that was under $7 and one that was
$11-50. We bought most, but not all, of the wine at Trader Joes. Participants
had to guess if each wine was 'cheap' or 'expensive.' No one got more than 8
right. Aside from guessing the price category, people rated how much they
liked the wine from 1-5 so they could see what they liked after the reveal. A
50 dollar Cabernet was one of the lowest rated.

This 3.99 bottle had the 2nd highest rating:

[http://cheapwineratings.com/2008/04/10/villa-cerrina-
montepu...](http://cheapwineratings.com/2008/04/10/villa-cerrina-
montepulciano-dabruzzo/)

Highest rated was a 6.99 Pinot Noir whose name escapes me at the moment. I
don't claim the results necessarily prove anything about the wider world,
except that my friends have been overspending on wine.

~~~
rdtsc
I am convinced that many wineries should sell the same wine for different
prices. They could sell the same wine for $7/bottle, $30/bottle and
$125/bottle. Of course the $125 should have a different label and should be
marketed a lot differently but the wine itself doesn't have to change.

I personally just buy a big jug of dark red cooking wine. I can't tell the
difference between it and the other more expensive wine, and I refuse to spend
more money for snoberry (note: I have nothing against others paying that much
if it makes them feel good/better).

~~~
carbon8
Well, there's a big difference between the issue of wine rating silliness and
the wide variety of flavor you'll find in wine. A $10 bottle of gabbiano
tastes almost nothing like a $10 bottle of monticello rioja and neither will
taste anything like your jug of wine. Saying they taste the same is like to
saying orange juice tastes the same as lemon juice, and that both taste the
same as grapefruit juice.

~~~
rdtsc
I was saying that I couldn't taste the difference and according to the
article, many other people can't either. They are essentially tasting the
label and the marketing, which is why the marketing and the label should look
different in both cases.

They are people more sophisticated and better at wine tasting, but there are a
lot of people who just follow irrational marketing cues.

~~~
carbon8
_"I was saying that I couldn't taste the difference and according to the
article, many other people can't either."_

This isn't what the article is talking about. The article is arguing that
subjective, linear wine ranking is faulty, that preferences are not objective
or static, that it's difficult to distinguish between similar wines and that
identifying individual flavors and scents (eg, "lavender, roasted herbs") are
tied to the taster's imagination.

If you can tell the difference between grapefruit juice and orange juice, you
can tell the difference between a gabbiano, a monticello rioja and a generic
merlot. They taste dramatically different and you can't give a merlot to
someone who knows what chianti tastes like and have them believe it's a
chianti. Different varietals have very distinct flavors.

And when it comes to quality, there is a much, much more significant
difference between a poorly stored, very low-quality wine and a mainstream
~$10/bottle wine than there is between a $20/bottle wine and a $100/bottle
wine. After all, a number of the pricer wines would be simply $20/bottle if it
weren't for the ratings systems critiqued by the article.

------
jcl
_Mr. Hodgson said he wrote up his findings each year and asked the board for
permission to publish the results; each year, they said no. Finally, the board
relented -- according to Mr. Hodgson, on a close vote -- and the study
appeared in January in the Journal of Wine Economics._

I love it. He knew from his findings that the wine review board's decision
would vary from year to year.

~~~
sophacles
Thank you. It so rare that a statistics joke makes me actually laugh.

------
stcredzero
When I lived in Cincinnati, the "Everybody's Cooking" radio talk show was like
my "Car Talk" for foodies. One of their soapbox things was wine ratings. They
thought it was way too complicated. "C'mon, are you _really_ going to taste
the difference between an 88 and an 89!?"

Those two chefs proposed the "Siskel and Ebert" system. Thumbs up/thumbs down.
"Do you like it? Do you not like it? It's _only_ fermented grape juice!" They
advocated ditching the mysticism and just enjoying it with dinner.

~~~
silentbicycle
The style / grape type is probably more important than some number. "I like
really crisp whites", "I like really thick, fruity (but not too tart) reds",
etc. will narrow down a search better than just looking for high ratings.

If you really like steak, you're probably better off with a cabernet sauvignon
than a riesling, rating nonwithstanding.

~~~
stcredzero
Probably the biggest problem is the _packaging_. With produce, you can go to
the store, and feel and sniff the wares. If you don't like it, you go to a
different store. With wine, you have to wait until you get home with the
bottle. And even if you find a good wine, you have no idea if it will be the
same year to year, or even a few months later. You have no idea when
marketdroids are going to convince the executives to cash in on the name.
(Same happens to beer.)

Whole Foods does right by its customers by handing out samples of chosen
wines. I like being able to taste before I buy!

~~~
silentbicycle
Well, with beer, it's much cheaper to get a variety of singles. After my wife
and I did that a couple times, we had a very clear idea what styles we like.
(And that we prefer beer to wine.)

Incidentally, West Michigan is one of the best places in the US for craft
ales. There's Founders (<http://www.foundersbrewing.com/founders/>), Bell's
(<http://www.bellsbeer.com/>), and the Hopcat
(<http://hopcatgr.com/main/About.aspx>), for starters. :)

~~~
stcredzero
The Pacific Northwest was marvelous for great, innovative microbrew when I
lived there in the early 90's.

Cool, those guys are still around! <http://maritimebrewery.ypguides.net/>

------
silentbicycle
Something I was reading lately* noted that those numeric ratings are usually
reached tasting wine alone, rather than with food. That can bias ratings
toward wines that stand out after tasting several similar cabs (or whatever)
in a row, including wines that utterly lack subtlety. "This tastes like
getting whacked with a giant oak plank."

* One of the Culinary Institute of America texts, I forget which.

~~~
pbhjpbhj
I find that the taste of the second glass of a red is often more rounded and
enjoyably palatable than the first glass. I don't think this is inebriation
but something possibly to do with how the taste buds themselves work.
Temperature will alter the flavour a lot, as will the food being consumed
alongside (as you noted).

~~~
jeroen
Also don't underestimate the influence of oxygen.

------
Flemlord
This is one of my pet peeves. I can't figure out any way to find good wine
outside of personally sampling it. Ratings don't work, and price only has a
rough correlation to quality. My current system is to buy a random sampling of
low-mid priced wines and when I find a good one, buy a case.

------
fhars
Reminds me of the experiment run by a german tv program where they let some
people compare a real, but mediocre wine with a liquid composed from single
substances to match the gas chromatograph readings of a good wine. The
artificial wine won.
[http://www.wdr.de/tv/quarks/sendungsbeitraege/2009/1110/002_...](http://www.wdr.de/tv/quarks/sendungsbeitraege/2009/1110/002_arena_taeuschen.jsp)

------
msluyter
Fascinating. I don't know much about wine, but I've noticed that my own
appreciation of the same type of wine will vary greatly. Sometimes it seems
highly dependent on what I was doing earlier in the day and what I've recently
eaten. If I was cleaning, for example, perhaps some trace residue of ammonia
on my hands might interfere with the taste/smell. Similarly, if I had spicy
food, an otherwise enjoyable red might taste strange to me. And of course,
there's a fair bit of variability between bottles for various reasons, making
the whole process even more uncertain.

------
tel
The strategy I've always employed is to pay no more than $6-9 unless it's a
special varietal. Some of the best wines I've ever had were South American
bought as one-offs for $10-15 though much of the enjoyment came from the
novelty. Much better than honing your pallet to the taste of expensive wine is
to learn to pair wines with food. Learn what varietals you like and then play
with foods spicy, sweet, and sour. The most enjoyment from wine comes from
learning how the tastes combine and change wildly.

Pairing wine is a game of combinatorics. The tricks tend to be to maximize the
number of other flavors you can try with your wine. In this way, you can
transform a single bottle into 3 or more different flavors. Some tricks:

Bread, cheese, and wine is a common thing but don't go overboard on the bread.
Much of the time the heavy starch is going to wipe out the flavors. Invest in
more variety in the wines and cheeses.

Spice comes in many, many varieties (from peppercorn to bright red ground
cayenne, to Chinese sichuan pepper) and each one often has a different
reaction with wines.

Steak and wine is a great combination because well-prepared meats have tons of
interesting flavors thanks to the "maillard reaction". Sausages are a
particularly good choice.

~~~
jfischer
The Economist recently had a article about the scientific basis of the "only
white wine with seafood rule":
[http://www.economist.com/sciencetechnology/displaystory.cfm?...](http://www.economist.com/sciencetechnology/displaystory.cfm?story_id=14743767).
Apparently, the key factor is the level of iron in the wine -- too much iron
in the wine/seafood combination results in a "fishy" taste. It would be
interesting to see more research into the chemistry of wine/food pairings.

~~~
silentbicycle
From a chemistry standpoint, the biggest factor is the acidity of the wine.
Acidic wines (particularly whites) can be very crisp and palate-cleansing,
while less acidic wines taste softer and smoother.

I don't have my copy of Harold McGee's _On Food and Cooking_ with me, but
that's the first place I'd look for more info.

------
DanielBMarkham
I'm not so sure that applying statistics to wine tasting isn't like applying
physics to haircuts: yes there's probably no relationship between the money
you paid and the way you feel about your haircut, but if you thought there was
you probably were missing the point.

Wine is an immensely subjective and varied experience. That's one of the
reasons people like it so much. There's one feeling you get when you're dining
out and a friend turns you on to a 10-dollar red that tastes great and another
feeling you get when you go to the cellar for a special occasion to get a
hundred-dollar bottle -- even if the wines are statistically
indistinguishable.

What can I say? It's a great product. All of human experience goes into the
process: peer pressure, prior experience, atmosphere, cost, anticipated
pleasure, etc.

When I started trying to learn how to taste wine I went through Wine Spectator
and only ordered wines of the very highest rating. I understand this is just
an anecdote, but I found that about 1 in 3 were actually distinguishable by me
from grocery-store wine. I took that to mean that yes, there are differences,
but the amount of signal is vastly overstated.

But that misses the point: it's emotional, not statistical.

~~~
HeyLaughingBoy
_it's emotional, not statistical_

You just said a mouthful :-)

When I started making my own wine a friend sent me a review of a Barolo -- the
variety I was making -- and I mentioned that the wine review seemed awfully
similar to high end audio equipment reviews (e.g., $15k CD transports +
another $10k for the DAC and preamp!) and sent him a few links.

He was absolutely astounded (and amused): both kinds of reviews used language
so similar that in some cases they were almost interchangeable.

What's my point? Both products can be very expensive with little objective
differentiation leaving the buyer with a great deal of incentive to defend his
purchase vigorously to anyone who asks the question, "why the hell did you
spend so much money on _that_ "

~~~
DanielBMarkham
lol. Your are correct. The less actual substance and more emotion there is,
the more tenaciously people will defend their positions. Again it's a feature,
not a bug: one of the reasons I decided to look into wine collecting was
observing a couple nerd friends of mine argue (discuss) for 30 minutes what
kind of wine to order at a fancy restaurant.

 _Both products can be very expensive with little objective differentiation
leaving the buyer with a great deal of incentive to defend his purchase
vigorously to anyone who asks the question, "why the hell did you spend so
much money on that"_

To which the appropriate reply is: "I didn't spend my money on _that_ , I
spent my money on _me_ (or you, or our dinner, or your graduation, etc)

It's like weighing the steak at a fancy restaurant and comparing it to the
price-per-pound down at Costco

It reminds me of my step-father's view on haircuts: for a certain amount of
money, you deserve to get at least so much hair removed. :)

Having said that, I've got a bottle of 98 Donnoff Riesling that has my name on
it for Christmas Eve. And I've been saving a 1996 Opus One for gosh, I don't
know, perhaps the end of the world or my first million :) And I'm thinking
that I've sat on those 4 bottles of 97 Dow's Port long enough.

To my mind, I'll take these wines and statistically match them up anywhere.
But who cares if I'm wrong and it's just me? I've enjoyed collecting them, and
I'll enjoy the special feeling of consuming them when I think the time is
right.

But damn it, I'm right. (grin)

------
hop
Vodka is an even worse when it comes to peoples perception of quality. The
higher the price, the better it must be! Twice I've had bartenders will up
shot glasses with well vodka and Grey Goose, then number the napkins - my
friends have got it wrong both times.

I've been to Mt. Hood distillery, they import 180 proof ethyl alcohol by train
from the midwest where its distilled from grain, then the use it to make many
different popular brands of vodka, gin, liqueurs, and others. The Vodka bought
in America is Ethyl alcohol cut with water!

------
chasingsparks
Similarly: _Judgment of Paris: California vs. France and the Historic 1976
Paris Tasting That Revolutionized Wine_

[http://www.amazon.com/gp/product/0743247515/ref=cm_rdp_produ...](http://www.amazon.com/gp/product/0743247515/ref=cm_rdp_product)

------
vaksel
the idea of wine ratings seems weird to me.

Everyone has different tastes, different taste buds, how come we don't have
any rating systems for candy? or steak? or eggs? or icecream?

This Snickers bar, really brings out the taste of Artificial Red #503

~~~
dagw
Is it really any weirder than a rating on films or books or music? That is
also largely about taste. And there are plenty of competitions and ratings on
most kinds of foodstuff (including candy and ice cream), so wine is in no way
unique in that respect.

~~~
oikujhgfvg
In books and films you have a little more information. An even then the only
realistic scale is one or two thumbs up.

Suppose the only thing you knew about a film was it's director (the grape) the
studio (vineyard) and a score out of 100

~~~
dagw
I'd also know who gave it the score. If a director that I love makes a film
that a critic that I respect gives a high score I'd be happy to see that film
knowing nothing else.

------
bruin4tw
I don;t know how to rate wine at all. The way I rate wine is by the burn. If
there is no burn and goes down smooth it is good wine.

~~~
sebg
The burn comes from the alcohol. Smoothness could be due to lack of tannins in
the red wine. (Tannins cause the feeling of astringency in your mouth and
inner cheeks.)

