
Overwatch Randomized Trial on Gender - Aissen
http://danluu.com/overwatch-gender/
======
jasonshen
This is about as well done of a study as we could ask for from a non-academic
research level. Basically while the number of sexual comments were relatively
low, the amount of questioning/challenging of his abilities went up a lot when
he played with a feminine name.

Reminds me of a professor at Stanford who underwent a sex change and
experienced very different interactions from his peers:
[https://www.nature.com/scitable/content/does-gender-
matter-b...](https://www.nature.com/scitable/content/does-gender-matter-by-
ben-a-barres-10602856)

We recently did a survey of 240 collegiate esports players and found that
women were 9x more likely to be "very concerned" about sexism and 2x as likely
to be "very concerned" about bullying in competitive gaming culture.

I think Dan's research can help explain why.

[https://www.midgame.gg/survey](https://www.midgame.gg/survey)

~~~
ToFundorNot
This study is missing one important factor, a baseline. The author should have
provided an androgynous named account to see how that fared.

~~~
Dylan16807
An A/B test doesn't need to attempt a "middle" option. And there's no reason
to assume an androgynous name would be in the middle or in any way qualify as
a "baseline".

~~~
zaarn
It would still be very interesting to see how people would react to neutral
usernames like randomly generated number sequences. It could provide some
useful insight.

~~~
seqastian
Gamers just default to male.

~~~
zaarn
That's just an assertion without data backing it up.

The above experiment with neutral names would give some insight if "gamers" do
indeed "default to male", as you express it, or not.

------
jordanthoms
If I took up soccer, I'd be unlikely to join a team with some 8 year olds -
even though I'd probably be at their skill level. Instead, I'd find a group of
people who I'd get along with better. So why do games do matchmaking based
only on skill, not taking social cohesion into account? There's much more to a
good gaming experience than just having skill-balanced teams.

CS:GO is a classic case of this - the player base is noticeably more mature at
certain times of day, and there's a distinct period when all the kids have
just got home from school where things are particularly bad. Occasionally
you'll get a team where you everyone works together well, but to have that
every time you'll need to spend a lot of effort maintaining a huge friends
list in order to be able to put a team together at any given time - rather
than just pressing the matchmaking button.

(I think you could might get decent results on this by having each player rate
the others at the end of the match - but you don't use the ratings to rank the
players, instead you'd use them to cluster players based on who they mutually
like to play with. You'd then use that as a variable to optimize on when doing
matchmaking. Hopefully with enough data you'd be able to predict if players
would have a good experience even if they've never played together before.)

~~~
AnIdiotOnTheNet
Since matchmaking has taken over, I basically don't even play online
multiplayer games anymore. I feel fortunate to have grown up in a time where
there were dedicated server communities. It's a bit like Cheers, sometimes you
wanna go where everybody knows your name and you don't really mind that they
routinely wipe the floor with you.

------
madrox
I do wonder if toxicity is inherently built in to the nature of Overwatch.
It's a 6v6 team game that incentives you into a play mode that punishes you
heavily for leaving a game you aren't enjoying. I realize non-competitive
modes are a lot less punishing, but anyone who's played a lot of Overwatch
knows if you want to "play Overwatch the way it was meant to be played," you
have to go to competitive. Speaking psychologically, I'm curious what that
kind of experience does to a person's behavior (short term and long term).

I've played hundreds of hours of Overwatch on console with my wife. We don't
use mics, because we had enough bad experiences early in our career to make it
not worth the risk. I never have problems when I play by myself.

Surprising number of people questioning the methodology in this study. I
would've expected that if it ran counter to anecdotal experiences described by
women (and men who group with them) who play, but I'm not convinced more
rigorous methodologies would significantly change the results. If someone does
believe that, I'd seriously challenge their reasoning.

~~~
eropple
I don't think toxicity is built in, though I understand where the thinking
comes from. I play Overwatch (albeit on PC) and pretty actively use a mic, and
I make a point of being the first person to talk in a channel. I'm friendly
and I'm even-keeled about it, and that sets the tone. Once that tone is set
it's very difficult for your garden-variety douchebag to start mouthing off.

The LFG system also helps.

~~~
JimboOmega
Comparing overwatch to League of Legends, League is sooo much worse, and I
don't know why.

Also FWIW the voice you use in OW can cause drama. I've definitely had games
where I get flak for my voice - when it was more androgynous often people
would want to know my gender.

I've seen people get exploded on for their accents, sounding young, etc.

~~~
eropple
Yeah, that can make sense. I'm a white American male with a deep voice, people
aren't going to give me much shit. But by being friendly I can make things
easier for other folks, and I occasionally get PMs from people who tell me
they appreciate that.

------
dev_dull
> _For one thing, when I was playing video games in the 90s, a huge fraction
> of the online gaming population was made up of kids, and online game
> communities were nicer than they are today_

Maybe it’s because I grew up playing FPS, but for as far as I remember people
in these games were extremely hostile to women.

~~~
slavik81
Overwatch is the only First-Person Shooter I have ever played where I have
heard women speak in voice chat and other players continued to focus on the
game itself. In other games I've played, a woman in voice chat would _always_
become the center of attention, whether she wanted to be or not. The attention
wasn't always negative, but it was unavoidable (and thus probably pretty
frustrating).

That happened in Overwatch sometimes too, but I was surprised it wasn't more
often. My suspicion is that it's the a result of the competitive mode putting
focus on the gameplay and a larger than average female player base for an FPS.

~~~
danso
Came here just to say this. Before OW, my other multiplayer games were Valve's
(CS, TF2, L4D), and I didn't know any girls/women in real-life who played
them. Given the hours/years of time I played, I would have to guess that I
played in a game with a female player at least a few times, but I could not
for the life of me ever recall a single time. It would be such a rare
occurrence that if it did happen, I think I _would_ remember it generally.

With OW, I'll hear a female voice at least a third or fourth of my random
games, as well as see female-gendered screennames who may not actively
participate in chat. I haven't played many games between old-school CS and
TF2, so I don't know if the overall demographic has changed. But I wouldn't be
surprised if OW's diverse and welcoming cast and lore was a major factor in
attracting female gamers. As much as I liked the older games, OW was the first
game that made me realize how much diversity could itself be a feature -- not
just in attracting a playerbase, but just more creative designs (though I
understand it may be too cartoony for some, of course). If only Blizzard's
writing could be as clever and less clichéd-ridden than Valve's...

~~~
Gaelan
It's not that you've never played with a woman—you've just never _known_
you've played with a woman.

~~~
danso
That's what I mean. If I had played with any, they may have not been active in
voice chat. In OW, it's not uncommon for me to hear women players.

------
SolaceQuantum
"If that result generalizes to OW, that would explain something which I
thought was odd, which was that a lot of demands to switch and general vitriol
came during my best performances with the feminine account. "

This isn't statistically analyzed, but I think it's a fairly significant
detail that's not emphasized more in the paper.

~~~
qiqing
Agreed. Especially this excerpt:

> Per above, I didn't want to code tone of messages to avoid bias, so this
> table only indicates the rate at which people told me I was playing
> incorrectly or asked that I switch to a different character. The qualitative
> difference in experience is understated by this table. For example, the one
> time someone asked me to switch characters in the masculine condition, the
> request was a one sentence, polite, request ("hey, we're dying too quickly,
> could we switch [from the standard one primary healer / one off healer
> setup] to double primary healer or switch our tank to [a tank that can block
> more damage]?"). When using the feminine name, a typical case would involve
> 1-4 people calling me human garbage for most of the game and consoling
> themselves with the idea that the entire reason our team is losing is that I
> won't change characters.

------
ggggtez
Obviously, a number of issues present in the study, but I can appreciate they
tried to account for some of them. Lots of variables are unaccounted for like
time of day, win/loss, character, what _the researcher_ said in chat, etc.

Similarly, the opening preamble points out that often the worst harassment is
potentially related to when players _hear_ female voices. Obviously there is a
different way you could do this study, which would be to invite ~300 players
of various skill levels to play a single game and record the results for
analysis. That type of study would have it's own problems, but you could at
least be more likely to control for some of the problems in this study.

~~~
mactrey
Yeah, the voice is the key. The gender ratio is skewed so much at higher ranks
that an obviously feminine name will simply be assumed to be ironic. I would
imagine the toxicity/sexual comments would increase by an order of magnitude
if the voice was female-presenting along with the account name.

------
Nexxxeh
Really interesting read. I appreciate that it highlights the problems with the
data and doesn't reach ridiculous conclusions based on junk data.

Anecdotally I've found a noticeable improvement since Endorsements, which this
data missed.

Also the rise of Fortnite and more recently Apex Legends has led to a further
decrease in toxicity that I've seen. A lot of toxic kids left for BR pastures.

That's on PC in the EU region on a high-silver current, low-gold peak SR
account, casual QP-heavy play but some comp, pure flexing (as I prefer to fill
rather than focus on maining one or two characters. Maining is arguably the
best way to climb out to Plat).

I'm an almost ridiculously positive player with a gender neutral name and I've
found being overtly positive vastly improves the experience.

It does prompt a couple of toxic responses ("kys", "fuck off" etc) but the
majority is really positive. (I generally just report any toxicity.)

Playing without voice in Comp is sub-optimal, as winning is much easier with
some kind of synergy. Even a basic PharMercy combo, or ultimate combos like a
EMP+DVa Bomb can be highly effective.

I normally Solo Q, but playing paired with a friend in QP yesterday I had a
surreal experience. One player on the enemy team insulted my friend (sexual
but not specifically-gendered insult). She was playing with a non-gendered
name, as Moira, and the toxic enemy player complained she was carrying us.

All the players on both teams turned on the toxic player, calling them out and
highlighting why they were actually losing. I have not seen anything like it
before but it was hilarious.

Stay As A Team is great in theory, but a team of 6 randomised will get rekt by
a pre-made 6 stack on Discord with each other, and the matchmaker doesn't seem
to take that into account.

I wonder what region author was playing on?

~~~
ben_jones
Overwatch remains one of the most toxic games I've ever played. I've put well
over 1,000 hours into that game but haven't played for about 9 months now, and
honestly its a breath of fresh air (to not deal with it anymore). There's
something about the rank structure of the game (and others I've heard about
like LoL and Dota) that encourages beratement.

Apex Legends conversely has been amazing in comparison. Maybe its because its
new, the tagging system, that its a free game, or that there is no ranked
structure (yet), but I've met nothing but cheery people and the occasional
crude drunk. I've played with a couple shy women, but (anecdotally) have seen
not one iota of what i've seen in Overwatch (aside from crude usernames that
are prevalent everywhere).

~~~
kfwhp
>that its a free game

Wait... Shouldn't it be the opposite? Paying for the game should make you more
reticent to being an ass because losing access to it means you've lost money.

~~~
ben_jones
Optimistically I believe humanity is generally good such that if you get a
massive number of users the good overwhelms the bad. Contrast this with
Overwatch who (IMO) has a shrinking userbase of increasingly hardcore/invested
players, and the worst tends to shine through.

I wonder if this suggests that hardcore/invested gamers tend to be less kind
than casual gamers or if this is anecdata.

~~~
doktrin
> I wonder if this suggests that hardcore/invested gamers tend to be less kind
> than casual gamers

I can't prove this scientifically, but yes _absolutely of course_ "hard core"
gamers are more likely than casuals to be raging toxic asshats in any kind of
competitive environment.

------
NullPrefix
>There are a lot of games that weren't included in the experiment because I
wasn't in a mood to listen to someone rage at their team for fifteen minutes
and the procedure I used involved pre-committing to not muting people who do
that.

------
tdb7893
The hardest thing about this study to me is that these tendencies aren't only
applicable to Overwatch. It's obviously worse in video game culture (because
it's more abusive and immature as a whole) but this matches with what I've had
women engineers tell me of their experiences (especially in college where I
know people who got asked out constantly and also were disrespected in group
projects)

------
nickpsecurity
Great study! Only gripe is that most trolling I get in online gaming is
through voice chat. They also seem to troll me a tad bit more if I have a mic
plugged in but I'm not talking. I suggest a re-run of this experiment
recording the audio in case we get wildly different results. In a one-party
consent state if you want to play it safe. ;)

I also think it will vary game by game. I don't play a lot of games. Most are
shooters like Modern Warfare, Battlefield, and Mass Effect 3. Modern Warfare 2
and 3 on Xbox brought in more trolls in voice chat than all my others combined
except maybe GTA V. In the last one, you play as a vicious thug causing
problems for everyone around you. Them being pricks makes more sense. The
difference in shooter games, though, was more interesting given it might
reflect different sub-cultures in gaming. Specifically, MW is designed to
appeal to largest number of people with lowest barrier to entry and patience
required. Might self-select for higher percentage of jerks. Whereas,
Battlefield 4 and Mass Effect 3 push people to work together more for best
results.

~~~
towaway1138
I don't understand why people ever play with voice chat enabled. I don't want
to hear streams of ungrammatical foul language, and I certainly don't want to
hear people slapping their kids, etc. TMI.

~~~
munchbunny
I've found that when you are matched with a team of good communicators, voice
chat is a force multiplier. I've had games of Overwatch, Battlefield, etc.
where the faster coordination really let us steamroll the other team.

Maybe the other players were just really good and I was getting carried? But
from my anecdotal experience, good voice chat tends to add a ton to team
momentum.

With someone who's playing TV in the background, someone who doesn't bother
with callouts, and someone who's tilting hard... I'd rather mute them and
myself.

------
macrael
This video series from a gamer who plays with voice chat enabled was pretty
eye opening.
[https://www.youtube.com/watch?v=jL0aVqVslSE](https://www.youtube.com/watch?v=jL0aVqVslSE)

~~~
34qg4g
Cherry picked recordings are very eye opening

~~~
macrael
The series comes out regularly enough to give you all the information you need
about the frequency of this shit.

------
devindotcom
>Examples of typical undirected comments that would occur in either condition
include ""my girlfriend keeps sexting me how do I get her to stop?", "going in
balls deep", "what a surprise. _strokes dick_ [during the post-game
highlight]", and "support your local boobies".

I think it's notable that although these are not directed they are highly
gendered. This is part of the problem as well.

------
AcerbicZero
This is an interesting study, and helps highlight why Overwatch needs a
scoreboard. With 6 random people on a team, very limited personal statistics,
and no game length team statistics, everyone tends to get toxic once you start
losing. I've played everything from high gold (tanked rank for a few seasons)
to low masters, and its the same pattern every time. Someone makes an
accusation about a player not pulling their weight, demands a swap in a rude
way, and the player being accused defends themselves. Then the name calling
starts.

It would be interesting to see what roles the author was playing, and the
exact SR of the accounts, as there seems to be a great deal of variation in
what people will say on voice comms (usually higher SR, because voice comms >
chat) and what will be said in chat. People also like to blame DPS first, then
demand a Reinhardt (if they don't have one) and then finally blame support.

Without hard data as to what is going wrong with a team composition, toxicity
is almost guaranteed when losing.

~~~
wlesieutre
_> People also like to blame DPS first, then demand a Reinhardt (if they don't
have one) and then finally blame support._

As someone who mains support, I don't think this is true. I've had plenty of
games where I do a shitton of healing, get the healing card at the end, and
still had people blaming me for "0 heals" all game.

A scoreboard would help with that, but it'd be a trade-off with having 5 angry
people shitting on you if you ever have a bad game. I'm not sure that's
better.

EDIT: Plus there's all the times when your stats are bad but you're not the
right one to blame, like when the whole team can't move up through a choke
because the tanks are too chicken to go first, but it gets reflected in poor
DPS eliminations.

Ultimately it's a team game, and failures of teamwork aren't reflected well in
a stats scoreboard. I think you'd incentivize poor team play by people trying
to pad their own numbers at the team's expense.

And no matter what information is available, some people are just looking for
someone to blame when they lose. I'm not sure how to fix that. If matchmaking
is working right you're going to lose about half the time, there's no point in
getting worked up about it but people still do.

~~~
greedo
Exactly. I average around 10k healing/10min, but if the team isn't working
well, that can drop 90%. If no one peels for me, or if we just get rolled
because of an infinite number of variables, my individual stats can look like
crap.

------
MobileVet
Lots of one off gender swaps that have been shared and showed similar bias. Eg
[https://blog.olark.com/live-chat-gender-equality-
experiment](https://blog.olark.com/live-chat-gender-equality-experiment)

------
antisthenes
Perhaps I missed it in the article, but did the author control for receiving
comments on 'how to play' in winning games vs losing games?

Because if not, the whole study seems sort of moot.

------
rossdavidh
So, it seems like you could probably get a fair number of people to volunteer
to participate in a study like this, in order to up the number of games
played. Also, I wonder if you could (Laurie Anderson-style) change your
voice's gender, so that gender could be manipulated even for games with live
voice. If there's a researcher out there who was willing/able to organize, I
think there would be a fair number of volunteers.

------
gpm
Lots of comments here about voice chat.

I wonder if there is software that can convincingly change the gender of a
voice in real time? Could make for an interesting experiment.

------
zemo
The entire premise of this study is not useful, because what you're asking is
"what is the numerical occurrence of quantifiable harassment" and not "what is
the rate and which players abandon playing the game as a consequence of
feeling unwelcome and how does it vary by gender". What do the numerical rates
of harassment tell us, and how are they actionable? If a group of players does
not play because they feel unwelcome, THAT is the thing that matters, and your
study does absolutely nothing to analyse this criteria.

> I avoided ever using voice chat and would call things out in text chat when
> time permitted.

Again, this betrays a deep flaw in methodology, since voice chat is an
integral part of competitive Overwatch. The important things to consider here
are: what percentage of players are comfortable using voice chat and how does
that vary by gender; does availability and usage of voice chat confer any
competitive advantage to the player; and do the two factors interact in such a
way as to confer a competitive advantage to one gender over the other?

I find it disturbing when men do scientific studies to determine if women's
stated experiences are real.

~~~
minitech
> If a group of players does not play because they feel unwelcome, THAT is the
> thing that matters

What, in terms of profit? (And in the next paragraph, competitive advantage,
really?) That’s a very narrow view of what it means for something to “matter”.

~~~
zemo
no, in terms of the fact that they literally ran away because of harassment.
That's important, and if I have to explain to you why people leaving due to
serial harassment is bad for communities, we have nothing to talk about.

~~~
minitech
We agree that people leaving due to serial harassment is bad for communities.
Do you disagree that harassment is an issue even when it doesn’t cause people
to leave? Because that’s what it sounds like, and it sounds ridiculous.

