
We Experiment On Human Beings - dochtman
http://blog.okcupid.com/index.php/we-experiment-on-human-beings/?utm_source=rss&utm_medium=rss&utm_campaign=we-experiment-on-human-beings
======
nostrademons
When I was still on OKCupid, I once wrote a Chrome extension to hide the
pictures. I found that I had, in general, a much better experience with the
site - I'd actually read people's profiles, I sent better messages, and I got
more responses. I eventually gave up on it when a site redesign changed some
of the #ids I was depending on and I didn't feel like revamping the code. But
interestingly, my eventual girlfriend had _terrible_ pictures - the grainy,
multiple-people-in-the-background sort you're never supposed to put on a
dating site. Setting up our first date, she was like "We don't need to
exchange numbers, you know what I look like", and I was like "Actually, I'm
not sure I do, here's mine." (Okay, I didn't actually tell her her pictures
were terrible until we'd been dating for six months or so, but that was the
general sentiment.)

I think it's one more example of when people's emotions and desires lead them
to suboptimal outcomes. Most of the cues we associate with beauty and sexual
desire evolved back in the savanna when health and fertility were very real
risks; being able to pick up on which potential mates would be able to carry
healthy offspring to term and nurture them until adulthood was very important
then. Nowadays, the far greater risk is that you'll hate each other and fight
all the time, but this has only been a concern since people started living
long enough and in close enough proximity to care.

~~~
jessriedel
> my eventual girlfriend had terrible pictures - the grainy, multiple-people-
> in-the-background sort you're never supposed to put on a dating site.

This is not what people mean when they say someone looks terrible in their
pictures.

The point of the experiment is to withhold information about whether someone
is physically attract, not about whether they have good photography skills.
The only reason to avoid putting up grainy pictures is because people will
assume you're not good looking and are trying to disguise this fact.

~~~
icefox
According to this research done by the okcupid guys here:
[http://blog.okcupid.com/index.php/page/7/](http://blog.okcupid.com/index.php/page/7/)

"The flash adds 7 years"

"The type and brand of camera you use has a huge effect on how good you look
in your pictures."

"There are peak times of the day to take a good picture." ala you are more
attractive in photos taken at sunrise and sunset.

Between that blog entry and the other ones on the topic the conclusion really
is that how attractive you are in real life and how attractive you are in your
photo can have very little correlation.

------
blt
Heh heh. I recently got a message from okcupid saying "Because of a diagnostic
test, your match percentage with <user> was misstated as 31%. It is actually
91%. We wanted to let you know!" I had a feeling something like this might be
involved. (I messaged that person anyway, because they had a funny username,
so I guess I supported their point.)

------
ssl_love
Hey OkCupid – How about some SSL Love?

"For the hundreds of thousands of users searching for that special someone
through one of the largest free online dating sites, the love fest may be
coming to an end. OkCupid is putting users’ privacy in danger by failing to
support secure access to its entire website through HTTPS. Every OkCupid
email, chat session, search, clicked link, page viewed, and username is
transmitted over the Internet in unencrypted plaintext, where it can be
intercepted and read by anyone on the network."

[https://www.eff.org/deeplinks/2012/02/hey-okcupid-how-
about-...](https://www.eff.org/deeplinks/2012/02/hey-okcupid-how-about-some-
ssl-love)

The Heartbreaking Truth About Online Dating Privacy:
[https://www.eff.org/press/releases/heartbreaking-truth-
about...](https://www.eff.org/press/releases/heartbreaking-truth-about-online-
dating-privacy)

~~~
3pt14159
Is this really still the case?

~~~
handsomeransoms
Oh, yeah :/ The login page is over SSL, but everything after that is not (!!)
This protects your password but not your OkCupid session cookies. It's an
improvement over no SSL at all (which was the case for a _long_ time) but
still leaves a lot to be desired.

------
rockdiesel
Nice to see a new blog post from OkCupid.

They haven't updated their blog since April 2011.

~~~
cschmidt
Interesting that the author has a plug at the bottom for his new book, which
he says he's been working on for the past 3 years. I guess his book writing
got in the way of blogging.

~~~
chimeracoder
> I guess his book writing got in the way of blogging.

There were a number of factors (which I've explained elsewhere in my comment
history). That's part of it, but another factor was that, in 2010, there were
2.5 people working full-time[0] on doing research for OkTrends.

The blog posts took a _lot_ of work. "The Real Stuff White People Like"[1]
took almost two months of my time, plus some from Max and Christian as well.
(Much like the product design process, since we didn't start each post off
with a clear end result in mind, not all the work was visible in the final
product).

I left to go back to school. Max ended up taking on more responsibility for
other data/stats work, which slowed the pace a bit, and he left at the
beginning of 2012 to do his own stuff.

People asked me for the last three years whether the reason OkTrends hadn't
posted since 2011 was because of the Match.com acquisition and whether Match
shut them down and I had to tell everyone "No, trust me, they're still around!
It's just a coincidence!". Thankfully I no longer have to. :)

[0] 2.5 full-time means: Two of us full-time, as well as Christian, though he
split his work time between OkTrends (the blog) and other stuff.

[1] [http://blog.okcupid.com/index.php/the-real-stuff-white-
peopl...](http://blog.okcupid.com/index.php/the-real-stuff-white-people-like/)

~~~
harimau
I had only heard about the studies but never took time to read the blog posts.
I think I've spent the better part of the past two hours on there. Great stuff
and always interesting to see people doing experiments like this (and making
them open and 'fun'). A blog post every few months with as much in-depth
research is something to look forward to.

------
zach
I thought Experiment 2 was a great example of the pain of trying to fit your
users' feelings into an "ideal" model.

When we started LALife.com, it was with the idea of making a real estate site
with lots of great statistics and quantitative data. We gave grades for how
safe things were, how good the schools were, multiple heatmaps, assessor data,
census data, nearby amenities -- a buffet of data. And sure, people who also
fit in the mold of trivia collectors thought this was amazing.

But the more we talked to people, the more we found out that they really
didn't use this data, even when they said they appreciated it. We had really
overshot the market -- users almost never delved into the statistics, yet took
our grades as absolute authority. What people wanted first of all was just
insight about whether it was a "nice place" or not, a maddeningly vague
concept.

Ultimately, we trashed countless tables and statistics and scaled it back to
one number. Yes, one 0-99 number that shows you, well, how "nice" a place is
to live. We were so afraid of generalizing things like this because everyone
is different and has different priorities and so on. But trying to make things
custom for everyone is a losing game, and it turned out "is it nice?" is
something everyone already knew intuitively.

So insanely, we went from having 50 extra data points on a home for sale to
having one "superscore". But a funny thing happened, which is it became much
more successful in the site's actual mission, to help people understand a
home's neighborhood without having to visit in person. And now we're providing
an insight that is compact enough that we can put it everywhere and people can
digest it instantly.

So although we would ultimately like to give people a more data-conscious
mindset, the tool and the user need to agree on that commitment. So we're
accommodating the user's actual mental model while we work towards expanding
it.

PS: We have our scoring model working nationwide, but as you have seen from
the Zillow thread, it's hard to get nationwide home listings. We're working on
getting homes for rent on [http://www.padrank.com/](http://www.padrank.com/)
so you can sign up there if you want to see how it works throughout the US.

~~~
boomzilla
This phenomenon is actually quite common. In retail, it's called the customer
funnel:
[http://en.wikipedia.org/wiki/Purchase_funnel](http://en.wikipedia.org/wiki/Purchase_funnel).
The idea is you want to capture as much awareness with something really
simple, usually it's a brand name, e.g. Apple. When customers express more
interest, by coming in Apple store/website, they are presented with more
design/technical details. Finally, only when they show purchase intent, they
are shown price/configuration/delivery options.

------
andrey-p
I really enjoyed reading this.

On the flip side (as an OKC user), I've also had moderate amounts of fun
trying to figure out their algorithms by trial and error.

For a while, I discovered I could figure out who scored me highly on their
Quickmatch [1] feature by visiting it repeatedly - the ones who gave me a high
score always came up first. This doesn't seem to work anymore.

[1]: If you haven't used OKC, the idea is: you score potential matches 1-5.
You can tell if someone scored you highly (but not who) and if both of you
give each other 4 or 5 you get a message "introducing" you both. If you're an
A-list (paid) member, you get to see who scored you highly without scoring
them.

Edit: tweaked phrasing as per comments.

~~~
Chinjut
I believe you are supposed to get notified when rated 4 or 5 regardless of
whether you rated the other person at all.

~~~
wldcordeiro
Didn't they recently remove the 4-5 star system in favor of just a like
button? I've been using the mobile version lately but it just has a star now
for like and no longer shows a 1-5 rating.

------
interstitial
OKC experimented with the matching process -- the express purpose of their
site. Facebook experimented with mood manipulation -- something they have no
permission to do -- regardless of the fine print. Facebook is a very bad
landlord, but people don't want to move.

~~~
azakai
Facebook didn't experiment with "mood manipulation" any more than OKC.
Facebook experimented with changing the site layout and which content and how
much to show. Exactly the same stuff as all sites test.

Facebook did theorize that the changes it was making would affect people's
moods in a certain way. That might make it sound like intentional mood
manipulation.

But OKC's changes _also_ affected people's moods. More than Facebook's, I
would guess.

All these sites are constantly experimenting on humans. That's what changing
the site content means. Everything that affects us affects our moods and
everything else.

~~~
webmaven
I'd also note that the proxy they are using ("apparent mood of subsequent
content") hasn't been shown to correlate with the subjects'/users' actual
mood.

All the FB experiment actually shows is that by manipulating the mood of
content seen, you can affect the mood of content produced.

Here's a couple of contrarian hypotheses: "when some users see more 'happy'
content, they feel worse about themselves in comparison, but post more 'happy'
content to pretend that isn't so." Or, "when some already-sad users see more
'sad' content, this does not affect their mood directly, but does give them
tacit permission to share how they are already feeling. Subsequently, their
mood actually improves."

------
galfarragem
I feel very "bearish" towards dating sites. People convince themselves they
are saving time meeting more possible "matches" in less time but they forget
that meeting online is not the same as in person, "turn-based" chat is
different than "real-time" chat.

Some years ago randomly I met a girl online, that most of men (based on her
picture) would wife up. I was living abroad at the time and we were only able
to meet in person some months later. After few minutes in the date I could see
that we would not go anywhere. There was a lack of "real-time" empathy and I
hated some details in her personality that I couldn't realise before meeting
her (not her look, surprisingly she was even more beautiful in person than at
photos). If I had met her in person instead of online, I would have saved
months of my life.. I will not do it ever again.

edit: Because of her I met my gf of 5 years, but that was just serendipity
working, not because of a dating site.

------
ianstallings
For the record not _every_ site does these type of experiments. I've been
programming websites since the 90s and I've never experimented on my users
like this. The most I've done is swap in and out user interface features to
see which one has a better response in a typical A/B test. It's not the same
as telling a lie to a user no matter how you split hairs about it. They're
overstating their case because they happen to be in the business of matching
people up.

------
adamzerner
Is anyone surprised that only 1/5 people who are 90% matches have a
"conversation"? I don't use dating sites, but I'd think that people who do
want to converse, and who better to converse with than 90% matches?

But I guess there's a lot of matches, and you can't talk to everyone, which
would explain the low numbers. Still a bit surprising to me.

Edit: What you guys said makes sense, thanks. My lack of knowledge of dating
sites lead to a bad intuition.

~~~
egypturnash
Here's a screengrab of my okcupid inbox.
[http://imgur.com/jsKVyCi](http://imgur.com/jsKVyCi)

One-liners and no picture are instant 'don't bother replying', and most people
with more details tend to get weeded out for other reasons.

~~~
searine
It is kinda fascinating how awful most guys are at pickup lines.

Be brief, be funny, and always as a question. That's how you do it folks.

~~~
kstenerud
Yeah, and spend hours and hours coming up with funny, witty, engaging prose,
individually tailored, for people who are 90% unlikely to even respond. After
awhile you start to feel like a putz, switch to simple one-liners, and
discover to your surprise that it doesn't really affect the response rate by
too much.

~~~
67726e
As a guy, this... this is pretty much it. I tried OKC for around half a year,
starting out doing just that. After the first half I just said "Fuck it" and
switched to bullshit two sentence intros that were fill in the blank types. My
response rate was practically the same.

Thinking about it now, it seems like the numbers game recruiters play. Send
out a lot of semi-decent introductions and hope for a response or two. Your
best bet is getting out there and "networking", but in lieu of that, sending
out a deluge of messages is the next best, albeit shitty, thing.

I'd be interested in hearing how women deal with the deluge of introductions
and what makes them decide to respond to someone. Is it the intro? The profile
picture? Do they read the person's bio or use that compatibility meter?

~~~
scarmig
This is simple to come up with a couple tests for: switch a great profile
picture for a bad one, switch a profile text you put effort into with an empty
or horrific one, and switch thought out messages with really shitty copypasta.

I've done this, and the profile picture is what matters. Text matters a bit--
empty hurt, but the half-assed one, if anything, outperformed the one with
effort, though they were basically the same.

As computer types, we like to focus on the text, because the text is all and
controllable. Good pictures are more difficult, because they take soft skills.
Note that, contrary to expectations, it's probably easier to get great
pictures of an average looking person than a great profile from an average
writer: just using a well-constructed photograph taken with a decent camera as
your main photo puts you ahead of most people, even those significantly more
attractive than you.

~~~
Xcelerate
Does photo quality really matter that much? I have never found that the
quality of the photo makes a difference in how attractive I find the person.
It can be a cheap cameraphone or a DSLR, but it makes no difference, at least
to me.

If it does to other people, then I should put more effort into my
photographs...

~~~
the_af
Someone else posted this in another comment:
[http://blog.okcupid.com/index.php/page/7/](http://blog.okcupid.com/index.php/page/7/)

It turns out photo quality, and even which camera you used, matters a lot.

------
rabbyte
I don't think the sticky point is whether or not we should be experimenting
with people to find what works and what doesn't. It's about respecting users
as humans and not simply data points. How does OkCupid know that conversations
"went deeper" or how often emails and numbers were exchanged? Were the
participants aware of that level of scrutiny or is that tucked away in the
ToS? Most people I know aren't even comfortable talking about their OkCupid,
they use it as a means to an end and place trust in the service to be
discrete. Do these experiments fall in line with user expectations?

I'm not saying you have to find these experiments upsetting just that when
people are upset it tends to be about treating users like honorable guests and
not about whether or not we should be allowed to tinker with services to find
the right solution.

~~~
herge
For the emails and numbers, they can probably search for those in the messages
quite effectively without many false-positives.

I'm all for the expectations of privacy, but if you type something into a
search box on a website owned by somebody else, well, you get what you pay
for.

~~~
rabbyte
Sure, my comment wasn't to say they're doing anything nefarious. I'm actually
not taking a position here either way. I'm just pointing out that people
aren't upset that services use data to improve their services which seems to
be the tone and premise of OkCupid's post.

------
tomjen3
>But we took the analysis one step deeper. We asked: does the displayed match
percentage cause more than just that first message—does the mere suggestion
cause people to actually like each other? As far as we can measure, yes, it
does.

Hm, that is unexpected. Wonder how long it will last.

~~~
sliverstorm
Nah, we like people who like us. So if a trusted source tells us someone else
either likes or will like us, we tend to like them.

~~~
surement
From Modern Seinfeld [0]:

 _George swipes right for every woman on Tinder. E: "What if you're not
attracted to her?" G:"If she's attracted to me, I might be!"_

[0] [https://twitter.com/SeinfeldToday](https://twitter.com/SeinfeldToday)

------
jessriedel
> I found a similar thing: once they got to the date, they had a good time
> more or less regardless of how good-looking their partner was. Here’s the
> female side of the experience (the male is very similar).

I'm skeptical it was similar, at least if you were to measure the thing that
actually matters: whether or not the person wants to go on a second date.
("Did you have a good time" is probably equating to "Did the person avoid
doing something terrible" in the survey.)

It's ludicrous on its face to think that men, if they choose a blind date
partner based on conversations and profiles, will want to go on a second date
just as often with a beautiful woman as with a plain woman.

------
lifeisstillgood
Basically, people are exactly as shallow as their technology allows them to
be.

Just a quote that resonates ... The design and architecture of our online
environments will affect our happiness as much as anything else.

------
pervycreeper
I'm surprised no one has commented on the ethics of "Experiment #3". Feeding
false data to users is not something that should be taken lightly.

~~~
spott
I think the premise you are assuming is wrong: You assume that it is sending
false data to them to say "this person (which our algorithm has deemed is a
bad match), is a good match". The problem is that _until the experiment is
done, you don 't how good a match that person actually is_. So you aren't
sending them "false data," because you don't know that it _is_ "false data".

~~~
pervycreeper
If it is being reported as known, but is actually unknown, that still would be
considered as 'false information'. For instance, if I claim to be clairvoyant,
and predict that the next coin you toss will come up 'heads', that would be a
false claim, regardless of the outcome of the coin toss.

------
wcdolphin
But you never did so with the intention of eliciting a negative psychological
response.

That to me is a distinct difference, both in practicality and morality.

------
tragomaskhalos
TL;DR; people are shallow idiots. Cool article though.

------
bshoemaker
#clickbait

------
krallja
Doesn't load completely in IE11.

------
iamleppert
What a bunch of pointless data and hubris to prove that, yes, people are
predominately fueled by the laws of physical attraction.

~~~
xj9
It might be hard to accept, _but people are shallow._ Everybody has an
attractiveness threshold that must be reached before someone will spark their
interest. It might be lower or higher, depending on the person, but everybody
has one.

Obviously, sustained interest and the development of romance is more
complicated and involves aspects of personality, shared interests, &c.
However, that doesn't change the fact that if they weren't good looking enough
for you in the first place, you never would have made the effort to find out
about the stuff you actually care about.

~~~
rabbyte
Being able to relate attractiveness to a linear scale doesn't mean attraction
operates on linear terms. In reality, there is no threshold nor is there a
higher or lower. I agree with you in the sense that people will disregard
personality or shared interests if they don't find the person physically
attractive but there's no answer that's more correct than another.

------
foolrush
I suspect that the cavalier sentiment OKCupid has approached the discourse
with will not go well for them.

Will be interesting to observe the sociological takedowns of this post.

~~~
tomjen3
Unlikely. Facebook went overboard with it after hiding it from their users.
OKC has always done this with their data, although I am more than a little
disturbed by them showing deliberately bad data to their users.

Also who the hell though it was a good idea to have a dating site without
pictures (even for 24 hours?)

~~~
Mandelbug
Unfortunately, while showing what we would consider "intuitively bad data" may
seem like a bad idea, only actually empirically testing it can we measure and
quantify the exact impact that bad data (either shown purposefully or
accidentally, or unknowingly) will have on users.

It may have in fact turned out that what we intuitively think as bad data
results in better matches or better experiences. I think experimenting is
worthwhile, so long as it is done in the open as they have been doing.

~~~
nobodysfool
I remember reading about this concerning Netflix. They used to have a
competition on who can come up with a better recommendation algorithm. They
eventually decided not to implement the best one because what they were
getting was good enough, and since most people were streaming instead of
getting a dvd, the more recommendations the better. They could just try and if
they don't want to see the whole movie, rate it and instantly pick another.

~~~
thelonelygod
I'm pretty sure they didn't implement it because it was an extremely
impractical solution.

Edit:

"We evaluated some of the new methods offline but the additional accuracy
gains that we measured did not seem to justify the engineering effort needed
to bring them into a production environment." \-
[https://www.techdirt.com/blog/innovation/articles/20120409/0...](https://www.techdirt.com/blog/innovation/articles/20120409/03412518422/why-
netflix-never-implemented-algorithm-that-won-netflix-1-million-
challenge.shtml)

~~~
nobodysfool
>Streaming has not only changed the way our members interact with the service,
but also the type of data available to use in our algorithms. For DVDs our
goal is to help people fill their queue with titles to receive in the mail
over the coming days and weeks; selection is distant in time from viewing,
people select carefully because exchanging a DVD for another takes more than a
day, and we get no feedback during viewing. For streaming members are looking
for something great to watch right now; they can sample a few videos before
settling on one, they can consume several in one session, and we can observe
viewing statistics such as whether a video was watched fully or only
partially.

I wouldn't say that this makes you wrong, but I'd say you are partially right.

