

Online recipe ratings are broken - kbrower
http://blog.recipepuppy.com/online-recipe-ratings-are-broken/

======
pedalpete
This is not only a problem with recipe sites, but with most user input.
[http://youtube-global.blogspot.com/2009/09/five-stars-
domina...](http://youtube-global.blogspot.com/2009/09/five-stars-dominate-
ratings.html)

YouTube noticed a similar issue. Getting users to rate on a star rating just
isn't effective. You may have more luck with a simple 'yes/no' option. Would
you make this again? And offer the opportunity for comments.

I find the comments in the reviews on epicurious to be very influential in how
I follow the recipe, and if I cook it at all.

~~~
decadentcactus
Agreed, and thanks for the link.

I think this applies to almost all online rating systems, in particular ones
where it's just pick a number out of five or ten. Most will be either very
high (perfect, or perfect with a slight problem), or the lowest (completely
shit). Barely anybody will mark something as 2/5 because it was utterly
terrible, but this one feature was good. They'll remember the bad parts. No,
no data at the moment and it's just a hunch, but one I hope to be able to
measure soon.

I'm hoping to apply this to webhosting, rather than simply rate out of ten and
leave a 2 sentence review, it'll be much more indepth, with multiple options
(support, uptime, features etc). I'm betting that the more effort that you go
through to leave a thorough rating will be balanced by the fact you won't have
to rate that often (at most maybe once a year once you find a good host).

Oh well, still in the idea stage but I think people should reconsider just a
simple rating out of five for anything non-trivial (like rating videos).
Anything important (major purchase decision) should be more detailed and have
more information for the prospective user.

~~~
pedalpete
I think you have another challenge when dealing with something like rating
hosting. The problem, I believe, comes in where you will likely have many
people leaving reviews after a bad experience, rather than a good one. For any
negative experience, a person will tell 10 people, for a positive experience,
they will share with 2 people.

Videos are a bit different, because people review as they are watching.
However with something like hosting, you need to give considerable thought to
how you get people to report positive or average experiences equally as often
as negative ones.

That's just my initial reaction.

~~~
decadentcactus
Yeah, being filled with negatives would be annoying although I'm hoping to
come up with something, probably along the lines of discounts/coupons
regardless of whether they left a positive or negative one (as long as it's an
informative one).

It's definitely something to think about and it's a pretty difficult balance
I'll have to achieve but I'm confident I can figure something out.

------
jkkramer
"So how does the user actually find the best recipe for a particular dish?"

Users may have better luck finding quality recipes by going to curated sites
Cooks Illustrated rather than mob-rule sites like RecipeZaar. As with social
sites (Digg, Reddit, HN), the chemistry of the community and the influence of
the admins can have a powerful effect on the quality of the content. It also
depends on what you're looking for. If all you want is cheesy enchiladas,
recipes from the popular sites will work just fine.

There are many sites (not to mention books and periodicals) which have thought
and care put into their recipes, often with supplementary material which helps
you understand what you're doing and become a better cook, rather than just an
ingredient-assembling automaton.

Another aspect to consider is that many people like to read (or watch) things
about food but don't actually cook much. That's why the Food Network has
become so popular (ever notice how all their commercials are for ready-made
products?). A pet peeve of mine is to see 5-star reviews with a comment like,
"Looks delicious! Will try making it sometime." Pretty annoying, but some
people just like to watch. Maybe that's why recipes with pictures get higher
ratings.

------
halostatue
Yes, online recipe ratings are broken, which is why I don't pay attention to
them. Of course, I'm a fairly good cook, so I know how to look at a recipe and
tell whether it makes sense or not. Often, I'll end up with 60% of one recipe,
15% of another, 10% of a third, and 15% my own experience to pull together a
recipe for a dish I want to try to make.

If you're looking for something on how to make a dish (like I did recently for
baguettes) and find several recipes to choose from, I find a few things make
it easy to choose from:

1\. Real, individual ingredients are better than composite ingredients (that
is, flour, baking soda, oil, etc. rather than Bisquick™). 2\. Simpler recipes
are usually better than complex recipes, especially if you've never made the
dish before. There are exceptions, of course. (I'm _certain_ that the King
Arthur baguette recipe that I found would be better than the one that I tried,
but it requires making a sourdough starter some fourteen hours in advance. I
will try it some day.) 3\. Well-written descriptions suggest better recipes.
This is a bit like code comments, though—there is such a thing as too much
description. 4\. Recipes that depend on exact timing of multiple pots and pans
are probably not a good idea to experiment with if you're not already good at
making similar dishes.

That said, I'd pay attention to recipe ratings if there were only two yes/no
questions: I made this; I'd make it again. Report the resulting value as a
simple ratio of (repeat)/(made once). Seriously, I don't make recipes more
than once if I wasn't enthusiastic about the results or weren't convinced that
I could make it better by changing the ingredients or cooking instructions
better. I don't know anyone who does. Star ratings are a waste of time.

------
bmcleod
I'd suggest that a bell curve in this case isn't a particularly good goal.
Most people who cook already know what collection of flavours they're likely
to enjoy and will make more dishes that combine these flavours in interesting
ways.

A good example is dishes containing coriander, a large number of people hate
it with a passion while others love it. People who hate it won't make the dish
and won't review it. Those who love coriander will make it and most likely
enjoy it.

Applying statistical analysis with the goal of getting a bell curve isn't the
correct direction to reason in.

~~~
kbrower
I guess I just want a recipe rating system that is consistent and has some
sort of meaning. I want to know of the people that like coriander and made a
dish with coriander in it, what recipes did they find the tastiest (the ones
with extra cumin or not). This is not possible when coriander loving people
rate the coriander containing recipe 5/5 9 times out of 10.

~~~
bmcleod
There was a post a while back suggesting having a few of your previous ratings
display beside the rating scale.

It requires more of a slider approach though, which is something you have to
spend more time teaching users about.

~~~
kbrower
I just found the thread. Interesting
<http://news.ycombinator.com/item?id=910976>

------
matrix
I don't think recipes themselves can really be rated; we probably have to rate
the person who created the recipe or the person evaluating the recipe. Let me
explain:

The number of people that come back and rate a recipe is probably small. And
how many people who had a middling experience with the recipe would make the
effort to come back and rate it? Or even a good experience, for that matter? A
bad one? You bet they're going to rate it.

So right there, you have all kinds of bias problems with the data. And just to
make it more fun, how many of those people rating the recipes prepared it
properly? How many substituted ingredients that just completely ruined the
dish? You see this all the time on these sites - "This recipe is horrible. I
vomited for 3 hours straight. BTW I didn't have any red onions so I used
pickled cocktail cherries instead."

Somehow you have to find a more indirect way of evaluating a recipe's worth.
Maybe it could be a measure of how many times a person comes back to look at
it (on the theory that each hit is presumably them coming back to look up that
recipe again)? Or perhaps how many people chose to make it.

~~~
amr
I think your are on to something. Report how many people saved a recipe (and
unsaved it?), how many printed it and how many views it received.

Additionally, I think the mentioned websites ignore an important fact about
recipes: they are not static. Most recipes have variations in technique or
ingredients. Many cooks, as you stated, substitute ingredients that subtly, or
not so subtly, change a recipe. If a mechanism exists to capture those
variations (wiki style, maybe?), those people might have an incentive to come
back and share their experience rather than their impressions.

------
jakarta
Recipes are interesting in that most reviewers actually alter the recipe,
making star reviews misleading.

So someone might post a bad recipe and have other members alter it and rate it
highly after their own modifications. You can only figure this out by reading
the comments.

So some kind of yes / no on whether you have modified the recipe is pretty
essential.

~~~
pie
You can observe this same multi-factor effect in restaurant or product
reviews.

When a restaurant serves one customer unsatisfactorily or a product is
delivered in poor condition, the rating suffers substantially. This rating no
longer represents the quality of the thing itself but also the individual's
experience.

The same is true with a cook's experience preparing a recipe. Depending on the
reviewer's bias, a recipe rating might describe circumstance and culinary
skill better than the quality of the instructions.

As other folks here have mentioned, there's also the issue of average
reviewers being not-so-great at representing their true opinions with linear
rating systems, but that's a whole other topic.

------
prawn
Slight tangent: anyone built a recipe site that works by presenting a common
item (chicken, flour, milk, etc) with "Got this? Yep or Nup" and so on to
build a list of search terms?

CookThing seems similar I guess, but not quite the same.

~~~
saturnine
<http://www.supercook.com/> is a front-end to allrecipes.com

~~~
prawn
Not sure if it wasn't working when I tried it, but I entered 'capsicum' and it
showed no recipes. What I envisaged is a site that would randomly pick a
common ingredient (rice, potato, coconut milk) and confirm that the visitor
had it at hand. Then go from there, rather than starting by having them name
something they had.

------
sjsivak
I do not use the sites listed in the article, but I have found that Epicurious
(<http://www.epicurious.com/>) has plenty of bad recipes that are rated as
such. I have had good success following the ratings and checking the comments
on the recipes before trying them.

Of course, with any crowd sourced system it is good to take the time to really
read through who you are getting your trusted advice from. I still remember
reading a review of a Korean restaurant on yelp that had two or three one star
reviews because "they did not have sushi".

~~~
kbrower
I just checked and I unfortunately do not have the epicurious rating data. I
did just check their website and they have a guide for rating 1-Okay 2-Good
3-Delicious 4-Exceptional.

Their rating system is not broken to the degree of these other websites.

------
teilo
There is a factor at work in this recipe ratings systems that is a common bane
to all rating systems which present a user with a range of choices. One
commenter (pedalpete) noted already that people gravitate high or low. But the
larger issue is the number of people that will gravitate toward the current
average rating.

This is a natural desire to conform, and it isn't always conscious. If a user
perceives that most people have rated this item X, he will naturally want to
rate it X, and will even re-evaluate his judgment based upon the ratings of
others.

Of course, for an online-rating system with instant feedback, this gives the
earliest ratings the highest weight, immediately skewing all the results which
follow.

But what effect does the average rating have on a person who disagrees? They
will naturally tend toward the opposite extreme. They want to make their
disagreement obvious. If the average rating is high, the negative rater will
gravitate toward the lowest ratings, when in fact he might have gone 2 or 3
stars had he had no such prior feedback.

This is a rather intractable problem, since about the only way to prevent this
sort of feedback would be to hide the rating until the person has actually
tried the item in question and rated it themselves. But that, of course, makes
a rating system useless.

------
jrockway
Another thing you can't do with recipe ratings is prove the the rater actually
made the item in question. At least on shopping sites, you can say "jrockway
bought this item on 5/19/2009", so you have some idea whether or not the
person has any clue.

------
aresant
Taking a page from the OKCupid marketing department? Bravo! Smart way to
attract people to your service.

~~~
Frazzydee
This is a good thing. The public usually does not have access to such detailed
data, so I'm glad that other companies are copying what OKCupid pioneered.

