
We built voice modulation to mask gender in technical interviews - HaseebQ
http://blog.interviewing.io/we-built-voice-modulation-to-mask-gender-in-technical-interviews-heres-what-happened/
======
benkuhn
I would really have liked to see them check whether the gender-blinding
actually worked. Even if someone's voice is modulated, they may have gendered
behavior patterns that could influence someone's performance ratings. I
wouldn't be surprised if I could guess the gender of a voice-modulated person
pretty accurately from other cues.

To combat this, they could have asked the interviewer during the performance
assessment whether they thought the applicant was definitely male/probably
male/unsure/probably female/definitely female. Then you could use voice
modulation as an instrument for perceived gender and get a better estimate of
the true effect of perceived gender when controlling for actual gender.

Hope they do something like that for the next round of experiments!

~~~
leeny
Aline from interviewing.io here. Great idea!

~~~
t0mbstone
You all should record the audio from the original modulated interviews, then
play them in _both_ male _and_ female versions to sample sets of separate
observer groups that rank the performance of the person being interviewed.

That way, you would be isolating the thing you are testing to just the voice,
and you would be able to establish controls and a baseline for the exact same
interview.

The way your test is described right now, there are still way too many factors
that could explain the results. Your controls are far too loose.

~~~
leeny
You're absolutely right re controls. Re your suggestion, totally agree, and
please stay tuned :)

~~~
drunkpsychmajor
I have read a lot of the comments here and your article and I find this
fascinating. On controls and such: on reddit someone pointed out that more
prosocial people may be more likely to sign up but I believe you controlled
for this using random assignment so I think you are good there.

The biggest thing, and I believe it has already been mentioned, is trying not
to bias the raters. The best way fmpov being to pre-record interviews and play
them to several different people.

The above would prevent testing for attrition but I think one variable at the
time is the way to go. You could test attrition later by having ratings go a
certain way and then seeing how subjects respond when they know that they
interviewer knows their gender vs when they dont etc.

If you have any interest in trying to get this published academically, and I
want to make sure you understand I am a very lowly research assistant, I could
talk to some of the professors at my university to see if there is any
interest.

------
xienze
I honestly feel like this sort of thing is a waste of time.

Let's pretend someone invents the perfect "gender bias-free" system: no names,
no faces, voices are flawlessly transcribed on-the-fly so you can't pick up on
speech patterns. What if it works perfectly and you still end up hiring more
men than women? I think exercises like this may end up giving people answers
they don't want to hear.

So, stop wasting time with stuff like this. If you're bothered by the fact
that there's more men in your company than women just hire more women until
you get your desired ratio. You've already made your minds up ("there's too
many men here, clearly we're biased against women during the hiring process")
and I doubt anything will make you think differently, so just go ahead and fix
the imbalance.

~~~
CobrastanJorji
I'm no expert, but isn't "let's just hire only women for a while to balance
out the numbers" going to violate a whole bunch of employment regulations?

~~~
bionsuba
Yeah, but who cares?

I'm not making a normative statement here, I'm simply stating the fact that no
one will come after you if you do this. Companies and colleges say all the
time that they're looking for a specific ratio of races and genders and
nothing happens to them.

~~~
NetTechM
I think you are talking about affirmative action, not so much a specific guide
to hiring process. The law states that x ratio of specific minorities have to
be hired/enrolled per year as a catch up mechanism that is largely outdated in
2016.

As far as University enrollment, women have led the numbers for years so that
isn't really at the heart of it. More so the subjects they choose as a
major/minor being the issue.

Marketing STEM majors to minorities and women has been a priority for a short
amount of time, so we will see how it pans out in the next decade.

~~~
killface
And that also means that at some point (ideally in the very near future),
we're just going to have to be OK with the idea that not all women want to be
STEM majors. This necessarily and explicitly means that the ratio will never
be 50/50.

We're all ok with the idea that men don't really want to be early childhood
educators (Pre-k through grade 2). It's a big deal when a guy applies for
those jobs, according to teachers I know. Then, of course, come the endless
jokes about the pedophile trying to get a job with kids, how he is going to
need a female escort when dealing with the kids, etc.

------
Jemaclus
Hi Aline! Great read, as always.

I had a couple of thoughts while reading:

1) One of the things you mentioned was that men whose voices were modulated to
sound like women tended to score better than women whose voices were modulated
to sound like men. Do you think that's because the men actually performed
better than women, or do you think it's possible that interviewers have a
lower set of expectations for female interviewees? Or some other reason?

My hypothesis at this point is that, all other things being equal, it seems
like if an interviewer expects you to do poorly because of an unconscious bias
against women, and you actually do perfectly average, the interviewer might
rate you higher simply because you surprised him. Does that make sense?

2) How did your partners (Mattermark, Yelp, etc) react to your findings? Will
this change how they hire in the future or how they use interviewing.io?

3) Given enough time and resources, do you think it's really possible to
eliminate gender bias in interviews (through techniques like this, or
otherwise), or do you think the best we can do is minimize it?

4) Piggybacking off #3, do you do follow-ups with the companies after you've
placed people? I'd be interested to see whether women who did well in the
interview actually do well at the companies long-term -- perhaps those biases
extend beyond the interviewing process, as well!

Keep up the good work! :)

~~~
tekcor
> "1) One of the things you mentioned was that men whose voices were modulated
> to sound like women tended to score better than women whose voices were
> modulated to sound like men. Do you think that's because the men actually
> performed better than women, or do you think it's possible that interviewers
> have a lower set of expectations for female interviewees? Or some other
> reason? My hypothesis at this point is that, all other things being equal,
> it seems like if an interviewer expects you to do poorly because of an
> unconscious bias against women, and you actually do perfectly average, the
> interviewer might rate you higher simply because you surprised him. Does
> that make sense?"

I was actually expecting this even before reading the results. I believe this
is because most companies try to have gender balance which tend to result in
females being hired with lesser "score" than males, simply because of
disparity between in pool male and female candidates available (even if you
consider they perform equally well).

Also I would say this is a bias towards females not against.

------
smokeyj
I'd be more interested in quantifying productivity and seeing if there's a
systematic discrepancy between productivity and gender. Either the market is
accurately pricing talent or there's a delta to be exploited.

I think if a paper was released tomorrow that said "female programmers 30%
undervalued as observed by double blind coding challenge" head hunters would
fix that problem overnight.

~~~
stale2002
If I were to start a company today, I'd definitely be tempted to only hire
women, and pay my entire workforce 80 cents on the dollar, instead of 70 cents
on the dollar.

I couldn't be sued for pay discrimination if ALL of my underpaid engineers are
women, since there is no benchmark for a lawsuit to compare the discrimination
too!

I jest, but it really does surprise me that more companies aren't trying to
aggressively poach underpaid women from other companies.

~~~
stcredzero
_If I were to start a company today, I 'd definitely be tempted to only hire
women, and pay my entire workforce 80 cents on the dollar, instead of 70 cents
on the dollar._

Except that correcting for experience, the pay gap shrinks to about 94 cents
on the dollar.

[https://www.youtube.com/watch?v=it0EYBBl5LI](https://www.youtube.com/watch?v=it0EYBBl5LI)

I'm with John Green here: Much of the strong negative feeling around the
gender pay gap comes from women as a whole feeling "squeezed" because they
still do an out-sized amount of the unpaid work in our society. This also
contributes to the absolute numbers pay gap, because women need to ask for
more flexibility to be able to accomplish that unpaid work.

I wish discussions around that issue were more nuanced and fact based. The
answer is more complicated than the simplistic notion that one gender is being
crappy to the other, in either direction.

~~~
bduerst
This is deceptive because it's an opinion piece on Youtube.

The BLS study that controlled for occupation, industry, part-time vs. full-
time, education levels, and other factors found that there _is still a gender
pay wage gap_ of 94 cents to the dollar [1].

It's not-fact-based to keep touting the 77% number, but it's just as much not-
fact-based to claim there isn't a gender wage gap because women feel
"squeezed".

[1]
[https://www.shrm.org/Advocacy/Issues/CivilRights/Documents/G...](https://www.shrm.org/Advocacy/Issues/CivilRights/Documents/Gender%20Wage%20Gap%20Final%20Report.pdf)

~~~
Spivak
But he brought that up in the video with the "unexplained pay gap" that still
remains when you control for known discrepancy causing variables.

I don't think people are being deceptive arguing for the non-existence of a
gender pay gap since controlling for reasonable variables wiped away 80% of
the discrepancy. There's no strong explanation for the remaining 20% so it's
really just an argument over whether or not the speaker believes that
remaining pay gap is the result of hiring or wage discrimination.

~~~
stcredzero
_I don 't think people are being deceptive arguing for the non-existence of a
gender pay gap since controlling for reasonable variables wiped away 80% of
the discrepancy._

I would say that the numbers very strongly show there is a 6% pay gap. If we
assume the job market is efficient, then either there is some bias against
women or women slightly under-perform men or women are worse at negotiating
salaries or they are receiving some renumeration in the form of greater
flexibility.

 _There 's no strong explanation for the remaining 20%_

Yet. I don't find it intellectually satisfying to say something doesn't exist
just because we don't have a theory that covers it well. The disparity is
smaller, but it does exist. I also observe how hectic the schedules of
professional women with children appear to be. I'm not sure we completely
understand what's going on there, but I don't think the sentiments of women
around this issue are entirely disconnected from objective reality. From what
I can see, women are "squeezed" in terms of the resource of time.

------
ranko
Gender blinding in orchestral auditions (which usually involves screening the
performers from sight and thus is pretty-much foolproof) has been shown to
improve the percentage of women who are hired.

See, for example, 'Orchestrating Impartiality: The Impact of "Blind" Auditions
on Female Musicians', Claudia Goldin, Cecilia Rouse:
[http://www.nber.org/papers/w5903](http://www.nber.org/papers/w5903)

~~~
Kevin_S
That's interesting, I would have thought being able to see actual finger
movements to analyze mechanics is a part of the evaluation process. Could be
totally off base though I suppose.

~~~
qq66
Good mechanics are a given at that level of performance. If you can play like
Yo-Yo Ma with bad mechanics, more power to you, an orchestra is not going to
care.

~~~
Bartweiss
If anything, I would imagine it's like the old story about drafting athletes.
Given two equally successful point guards, you draft the one with crappy
technique because he has more room to grow.

If you're hiring for some mid-level orchestra, bad mechanics on a
sufficiently-skilled player might be enticing. And as you say, bad mechanics
among top performers are basically nonexistent.

~~~
apl
That'd be interesting, but once you reach an orchestra you're generally beyond
technical training. Players are expected to know how to play; individual
training is rare and focuses on style aspects more than technical things.

~~~
Bartweiss
True enough; I had that in mind when I said mid-level, but even at that point
I imagine individual training is largely over.

Perhaps a better example would be a teacher identifying the most promising
school/college-age musicians?

------
gravypod
Why not just use text? There are many speech characteristics of gender that
will never be found and masked by modulation.

Simple placements of words, parts of speech referring to the self, and other
things are all much better clues to gender then voice frequency and pitch. For
example, when speaking to some people who I know who are either naturally very
high or low pitch, or are trans-gendered, there is a lot more information that
can be contextually extracted from the content of their speech.

Really, just do text-based interviews. It will allow people to revise what
they are thinking and really allow them to mask their gender.

~~~
BoysenberryPi
Because your actual interview will not be text-based and if their goal is to
simulate an actual interview scenario where you will be speaking out loud an
expected to form coherent thoughts to questions on the fly this comes closer
to doing so than text based.

~~~
mwcampbell
And why shouldn't the actual interview be text-based? Yes, that would be a
drastic change, and it would render on-site interviews pointless. But then
again, it would eliminate a lot more biases, e.g.based on accent and
disabilities.

~~~
jacobr
Many jobs will use mostly written communication, then it doesn't matter much
how you perform verbally, and a text-based interview would probably work well,
or even better.

If you're expected to cooperate closely and communicate verbally with
colleagues, you would want to know how the candidate handles that before
hiring them.

------
randyrand
Someone off-topic but: Laws forbidding sexual and racial discrimination in
hiring are a waste for everyone's time involved.

Do I really want to work alongside someone that "isn't racist" or sexist just
because the law forbids it? No of course not. I'd rather work next to people
that aren't racist and sexists regardless of if the law forbids it. I want to
work at companies where people actually want me. How difficult is that to
understand? =(

Most people want less racism/sexism. But this is not the way to do it. I'm in
favor of removing this law and letting discriminators be upfront about it. So
that I can knowingly avoid them without wasting hours and potentially years of
my life.

~~~
xenadu02
You are clearly completely incorrect... Like so obviously incorrect I'm almost
shocked to see such a comment here on HN.

Making discrimination illegal sends a social signal that the behavior is bad
and shouldn't be tolerated. It emboldens those who are slightly anti-
discrimination (but part of the majority group) to speak out against it
instead of following the herd. The few cases where actual discrimination cases
are prosecuted serve to warn others that there are negative consequences to
bad behavior. Over time this change in attitude transfers to younger
generations that see it as normal. That leads to fewer and fewer racist/biased
people in positions of power.

Often institutionalized racism and sexism are enforced by a small group of
busybodies... Take Cory Booker's family story where the Real Estate agent
tried to enforce a "white only" town but once they bought the house (by using
a white stand-in couple) all the neighbors were totally fine with a black
family moving in. It took only one bad actor to effectively segregate an
entire town. I have a suspicion (but no proof) that this is generally the case
- a vocal minority wants to enforce the status quo but the majority would be
fine with change.

We can't stop some people from discriminating, but we can declare that we as a
society think that is bad behavior and that declaration alone has positive
consequences.

~~~
randyrand
I disagree.

I don't think by making it illegal very much is done to actually combat the
way people feel. Look at Trump's success. If anything the social justice type
movements over the past few decades have only _increased_ Trump's success.

So for questionable gain, you are forcing people to waste everyone's interview
time, and -- god forbid if they are hired by racist/sexist people -- waste
potentially months and years of their life.

yes you can prosecute after the fact. But You will never get those years back.
It would be much more preferable to have taken another job in the first place.

------
mmastrac
I had a demo from Aline on this last week after seeing it on Twitter and
reaching out. It was a really interesting experience - it masks the voice
pitch but leaves the person's characteristic of speaking (is there a word for
that?) intact.

You can also access the recordings after-the-fact to hear how you did. I think
this is valuable for candidates who want to improve their interviewing skills.

FWIW, I heard it both ways - my voice as female and her voice as male. The
pitch shifting was very convincing both ways.

~~~
aylons
> person's characteristic of speaking (is there a word for that?)

I think you're talking about prosody:
[https://en.wikipedia.org/wiki/Prosody_(linguistics)](https://en.wikipedia.org/wiki/Prosody_\(linguistics\))

------
Jemmeh
_> "Contrary to what we expected (and probably contrary to what you expected
as well!), masking gender had no effect on interview performance."_

I've seen quite a few studies pointing out this "confidence gap" now. They
have shown women to be less confident, and it shows in their actions and
speaking patterns. I still hear these speaking patterns even in the voice
modulated versions-- statements sounding like questions with the upward
inflection at the end. Vocal fry. Lots of "uhm, well, erm" flustered speak. So
it doesn't surprise me that even with a modulated voice these women still
sound unsure of themselves and thus they still didn't interview as well.

As women there are some actions we can take. It's good to be aware of these
speaking patterns and try to break them--but ultimately there are so many of
these patterns that show that the only real solution is to try to internalize
confidence. It's easier said than done. And we can't ignore the fact that
there are reasons outside of ourselves for why women are not as confident as
men. It's not just an internal issue. While we can take some actions for
ourselves, we should still be looking at the external reasons that caused the
issue in the first place. Why are large amounts of women less confident
compared to men? Let's tackle those issues, too.

When women try to be confident, it can get frowned upon. A man being assertive
is a leader, a woman is bossy. Women are quicker to hit walls for just how
confident they're allowed to be before they get the "bossy/bitchy" label
thrown at them. The below article makes good counterpoints for the confidence-
gap, and provides sources to some studies regarding women and confidence. And
I've seen women call each other bitchy/bossy, too. This is something that
society as a whole needs to work on.

[https://www.theguardian.com/commentisfree/2014/apr/23/female...](https://www.theguardian.com/commentisfree/2014/apr/23/female-
confidence-gap-katty-kay-claire-shipman)

~~~
mpweiher
> "less confident"

This is not a gendered issue. Men who act less confident are at least as
disadvantaged.

> "confidence / frowned upon"

Also not primarily a gendered issue. Men who are not perceived as having the
status to be confident get pounced on as hard or harder. This appears to have
a lot to do with physical size. Tall men are "leaders", small men have a
"napoleon complex". etc.

Considering women are statistically shorter than men, I wonder if sex-
discrimination is fully explained by height-discrimination. Or maybe even
over-explained.

See also:

[http://www.jonathanrauch.com/jrauch_articles/2004/08/short_g...](http://www.jonathanrauch.com/jrauch_articles/2004/08/short_guys_fini.html)

[https://en.wikipedia.org/wiki/Height_discrimination](https://en.wikipedia.org/wiki/Height_discrimination)

~~~
Jemmeh
Women in general have less confidence than men. (Study:
[http://www.apa.org/pubs/journals/releases/psp-
pspp0000078.pd...](http://www.apa.org/pubs/journals/releases/psp-
pspp0000078.pdf)) That is a gender-specific gap in confidence.

That doesn't mean that there are no men with less-than-average confidence out
there. Nor does it mean that there are no men who have a hard time because of
said lack of confidence.

~~~
mpweiher
> Women in general have less confidence than men.

Women are also generally slower than men (I was once faster than the fastest
woman on earth, and I am not a particularly fast man). Not winning 100m races
against men is not a sign of gender discrimination, it is a sign that men are
faster.

So if men are better at being confident, and being confident is important,
then that's just that. Women can try to learn to be more confident, but
discrimination it ain't.

~~~
Jemmeh
That's exactly what my post said. It says women can work on internalizing
confidence, but we should also try to figure out why large amounts of women
have less confidence compared to men, and work on that too.

Discrimination comes into play when women are frowned upon for being
confident. For example when asking for a raise. A man can be seen as a go-
getter. But for women, that assertiveness can be seen as bitching or whining.
One of the studies from the article covers this. Here is directly linked:
[https://www.cfa.harvard.edu/cfawis/bowles.pdf](https://www.cfa.harvard.edu/cfawis/bowles.pdf)

~~~
mpweiher
Should "we" also try to figure out why large amounts of women have less speed
compared to men, and work on that, too?

~~~
Jemmeh
Sorry, I went back and edited my comment but you had already posted.

The reason we need to figure out the confidence gender gap issue is because it
leads to less opportunities for women. Anyone who is less confident is going
to have less opportunities. But for various reasons, women tend to have a lot
less confidence than men. Yes, that's something that is partially internal to
each person. But the fact that large groups of women tend to be less confident
than large groups of men means there are external factors for this too.

And it's hard to say, "Okay ladies, well just be more confident then" because
when we do become confident, it's frowned upon. That's one of those external
issues. No matter how much confidence I could build up within myself, if other
people only view it as "bossy", that's an external factor.

~~~
mpweiher
It is just as hard, or harder, for men who are not confident to just say
'well, be more confident', so again, not a gendered issue.

~~~
Jemmeh
But women are more frowned upon for being confident than men.

Again, see this study regarding women vs men asking for raises:
[https://www.cfa.harvard.edu/cfawis/bowles.pdf](https://www.cfa.harvard.edu/cfawis/bowles.pdf)

~~~
mpweiher
Hmm...study seems a bit flawed.

For example they ask college students to _imagine_ they are bank HR managers
and whether they would hire a candidate that asked for more money. From that
they conclude that a bias exists in the real world. However, all they have
shown is that college students _expect_ such a bias to exist in the real
world.

------
balls187
> it’s not about systemic bias against women or women being bad at computers
> or whatever. Rather, it’s about women being bad at dusting themselves off
> after failing, which, despite everything, is probably a lot easier to fix.

This conclusion (imo) should have been bolded. I disagree that getting women
to "dust themselves off" is an easier fix than if there were inherent bias
against women in interviewing.

My theory is that this is tied to (In the US) men being more tolerant of
taking risks, which leads to being more tolerant of failure, which leads to
being able to more easily "dust oneself off."

I'd imagine if you looked at women in this study who played competitive
sports, they would be less likely to leave the platform.

Risk-aversion might also contribute to women being less likely to ask for a
promotion, raise, or negotiate salary.

I disagree that "dusting yourself off" as being an easy fix similarly for why
I think "pick yourself up by your bootstraps" is not an effective way to
elevate people out of poverty. There are more cultural biases to overcome than
a simple behavior change.

~~~
zeveb
> My theory is that this is tied to (In the US) men being more tolerant of
> taking risks, which leads to being more tolerant of failure, which leads to
> being able to more easily "dust oneself off."

There are some pretty obvious evolutionary psychology reasons for men to take
more risks. Whether evolutionary psychology is just a bunch of just-so stories
or not, that's another discussion.

~~~
cauterized
There are plenty of enculturation reasons too. Doesn't need to have an
evolutionary or biological basis.

~~~
generalledger
I don't know if this is true, but intuitively it seems that something is more
likely to be ingrained in a culture if there is already a biological basis for
the behavior.

~~~
cauterized
I still strongly question that. For instance, it could be that our double-
standard expectations of women create more cognitive dissonance - which might
impact confidence or give women a sense that whichever choice they make will
be judged as "wrong". Or perhaps girls are less likely to be given the benefit
of the doubt when they make mistakes in childhood - teaching them to be less
resilient or risk-taking - without there being any _biological_ reason to
judge them more harshly.

------
elcapitan
I want applicant-side voice modulation to mask my insecurity and non-native
language skills.

~~~
malcolmgreaves
Both maturity and strong communication are important for any professional
position.

~~~
ubernostrum
_Both maturity and strong communication are important for any professional
position._

So, shall you start firing executives at VC firms, or shall I?

~~~
sluggg
Peter Thiel ... ubernostrum is coming for you

------
minimaxir
> Contrary to what we expected (and probably contrary to what you expected as
> well!), masking gender had no effect on interview performance

The main issue with gender and hiring is unconscious bias _on the interviewer
's end_, which voice modulation does make sense as a tool to avoid bias. Per
the grading criteria, they are judging on skill, problem solving, and
communication. I'm not sure how voice modulation _would affect the
interviewee_ in those three criteria, which makes the results of the
experiment not surprising.

~~~
StavrosK
The interviewer is the one grading the interviewee.

~~~
minimaxir
Right, which highlights another problem: interview.io's interviewers are
presumably trained to be more objective and stringent on the criteria. This
may introduce a selection bias.

Untrained interviewers, like the ones at startups randomly pulled into
interviews _and also hiring for culture fit_ , may have a bias that is more
conscious.

~~~
Karunamon
> _interview.io 's interviewers are presumably trained to be more objective
> and stringent on the criteria._

The common thought is that bias is an unconscious and unavoidable thing, hence
the voice masking. If that's truly the case, all the training in the world
doesn't help.

------
gmarx
The result does not surprise me at all. I realize this was not the kind of
study that would pass rigorous review. I am surprised the author thinks this
is a surprising result. The kind of people who do technical interviews have
been taught since childhood that the system is biased against women and that
they need to look out for this bias. If anything I would think the occasional
super tech woman would be enough of a surprise that the average interviewer
would have an unconscious bias in favor of her

------
hasenj
Honestly the modulated voice sounds like a gay man, or a transgender, or
someone who is physically male but is trying hard to speak like a female. It's
probably not the pitch itself but the manner of speaking.

------
blisterpeanuts
_"...it appeared that men who were modulated to sound like women did a bit
better than unmodulated men and that women who were modulated to sound like
men did a bit worse than unmodulated women..."_

The author admits it's not statistically significant, but nonetheless it
somewhat makes sense, if you take human nature into account.

If there's any bias in the technology field, it's probably that males are
assumed "smarter", but women are still liked and desired.

So, if a female is not answering questions as sharply, but is masked as a
male, the bias is to deduct extra points. If a male is answering Q's while
masked as a female, the interviewer is pleased and awards extra points.

Just my theory, but it seems to fit. Again, though... a very small sampling.

------
onetwotree
So I don't think that hiding factors that (we think might) influence biased
people is the right way to address bias.

Let's say a woman uses this voice modulator in an interview and gets a job
that she wouldn't have otherwise. Now she's going to work in the same shitty,
sexist environment that would have denied her the job had they known she was a
woman. What problem have we actually solved here?

I think that the problem isn't so much that women can't get jobs in tech as
that they don't want to work in an industry that is (correctly or not)
perceived as being a sexist sausagefest.

We have to change attitudes and culture to address bias, not simply put on
blindfolds.

------
mxfh
Can you pick your own filter?

Would go for Laurie Anderson's _Voice of Authority_ anytime.
[https://www.youtube.com/watch?v=YajQNIAY78k&feature=youtu.be...](https://www.youtube.com/watch?v=YajQNIAY78k&feature=youtu.be&t=37)

Used to great extent here: The Cultural Ambassador - Live
[https://open.spotify.com/track/3uM0L8r6lJGaU54v9RjwJK](https://open.spotify.com/track/3uM0L8r6lJGaU54v9RjwJK)

~~~
dang
I didn't realize it had a name. How wonderful.

------
teamhappy
Note to self: Write screen play about Equilibrium-esque future were everybody
wears burkas with built-in voice decoders.

------
yummyfajitas
It would be great if the author posted the raw numbers, so we could check the
statistical analysis for ourselves. I'm concerned by the low number of females
in the sample - was it high enough for the study to have significant
statistical power?

------
humbleMouse
This is a nice idea but seems like political correctness taken to the extreme.

------
breitling
Planet Money did a great podcast on this recently [1]. It's pretty cool how it
works.

[1]
[http://www.npr.org/sections/money/2016/04/22/475339930/episo...](http://www.npr.org/sections/money/2016/04/22/475339930/episode-697-help-
wanted)

------
hharnisch
This is interesting because it starts to peel another layer of the problem.
Yes there are still cues that can give away gender, but this seems another
step closer to making the interview process a science. The next logical
question from here is "why are women more likely to quit after 1-2 bad
interviews".

~~~
jbmorgado
I don't think you understood the article. The conclusion is that there is _no_
problem. This bias against woman in IT interviews people keep talking about,
doesn't actually exist according to the study.

~~~
imh
This misunderstanding is kind of the core breakdown in these conversations.
People don't agree on what the problem is. Is the issue about equal pay, or
about equal pay for equal work? In hiring terms, is it about equal hiring or
equal hiring for equal ability?

If we don't have equal pay for equal work (and equivalent skill, etc), I think
we can agree that's unfair and a problem. But if we do have equal pay for
equal work, but still don't have equal pay, then I'd say we may still have a
problem. Why aren't women making the same work decisions as men? If social
pressure is keeping women from doing equal work, that's bad. If social
pressure is forcing men to make shittier sacrifices for pay, that's bad. In
the article linked, women churn more. That's probably bad.

If there's inequality without unequal pay or unequal hiring, then the problem
isn't the fault of the firms, but it's still a problem.

------
rottyguy
I always felt they should do something like this for politicians during a
race. No names, age, gender, political leanings, etc. Just interviews and
debate.

~~~
alphapapa
Not sure exactly what you mean. But some kind of anonymous/pseudonymous debate
or interview process would not be a good idea, even though it would have some
nice benefits in theory. For example, it would be easy for a malicious
candidate to profess views opposite to what he believes in order to be
elected. Then he could pursue his actual policies after taking office.

Oh, wait, this happens now... :( But it would be even easier without true
identity and non-verbal cues, etc. Words on a screen/page could be written by
anyone.

Real identity is crucial, because what matters more than words is actions, and
only with a real name, face, and history can past actions be examined to
determine whether the person lives up to their rhetoric. If political history
teaches us anything, it should teach us to pay less attention to what
politicians say and more attention to what they have done.

So, as interesting as debates can be, maybe we need less of those and more
examination of candidates' past. If we really want to know what a candidate
believes and what he will do if elected, we should look at what he's done in
the past.

------
fractalsea
In the intro you say that data from >1000 interviews shows:

> men were getting advanced to the next round 1.4 times more than women

> men [...] had an average technical score of 3 out of 4, as compared to a 2.5
> out of 4 for women

The conclusion of your experiment was

> masking gender had no effect on interview performance

So the question is: why do women perform worse (by the above metrics) even if
the interviewer does not know they are a women?

Unfortunately you spend the remainder of the article showing that women are
more likely to quit after bad interviews, and hypothesise that this is due to
lower self confidence. This is interesting and all, but it does nothing
towards explaining the discrepancy you described in the introduction!

------
vegabook
So the point here, hopefully, is to allow truly talented female coders to
overcome the bias against women generally, which exists because of the fallacy
of averages.

Hopefully, the idea is not to push the erroneous view that women are _on
average_ equally good at coding than men. That is objectively untrue.

I can, with enthusiasm, support a technology which helps the (smaller number
of) women who are genuinely good coders to shine through the bias. I cannot
accept any technology which seeks to mask the reality that in this particular
domain of modern life (coding), men are usually better performers than women.
Because that would be regressive.

~~~
infinite8s
Do you have a reference or any data to support your assertion that women on
average are not equally as good at coding as men?

------
j0e1
> Maybe tying coding to sex is a bit tenuous, but, as they say, programming is
> like sex — one mistake and you have to support it for the rest of your life.

Hilarious! But I wonder if it is true.

~~~
sluggg
Less and less true as most companies move towards a more agile development
cycle

------
alarge
I guess the results here would surprise you if you assumed that hiring bias
was primarily the result of an explicit bias against {gender, race, etc.}.
While there is certainly this class of bias, I personally haven't found as
much of it in the technology field as there is purported to be in general
society.

What I have found, instead, is a more subtle bias that goes something like
this: "I view myself (or that guy over there) as the model employee. I'm
looking for someone who thinks and acts like we do -- not just on our level,
but views problems the same way, is likely to arrive at the same solutions,
has the same energy level, etc.". I believe this attitude forms the
fundamental underpinning of both company culture and hiring bias. I've seen it
work successfully over the short term (particularly in startups where you're
trying to build a small team of really smart people that can build something
quickly), but I think it is counter-productive over the longer term.

When I'm hiring, I simultaneously try to answer three questions: (1) Is this
person capable of doing the job I'm hiring them for? (or grow into being
capable), (2) Is this someone I could work with and who could work with my
team?, and (3) What unique life experiences or perspectives does this person
have that they could bring to the team and improve our culture or decision
making?

What I've found is that some folks over-focus on (1) and sometimes (2) without
adequately recognizing the value of (3). I personally believe that you don't
remove bias by trying to legislate against it - you remove it by getting
people to desire the alternative - diversity.

------
ajmurmann
Are there actually studies that show that the interview process rather than
the pipeline is the issue? At my previous job we ended up making an offer to
every female candidate who made it to the phone screen. This wasn't because we
preferred them because of their gender but because they were really good. Many
men never even made it past the phone screen. We still ended up with a
majority of men on the team.

------
kafkaesq
Which (by design) would unfortunately mask character, warmth, and
authenticity, also.

------
nxzero
If gender bias is the issue, focus on bias detect and mitigation; not masking
gender, which to me sends the wrong message.

------
zw123456
I think this is a great effort and I applaud their efforts. There are also a
lot of very good points made in all the comments below, summary:

1\. Simply changing the frequency of the voice does not necessarily mask
gender, other intonations are still present or make the person sound perhaps
gay or otherwise still feminine sounding.

2\. There are other socialization issues as well as biological/chemical that
make men more aggressive than women.

One of the key things I have learned over the years is an appreciation for the
different way that women approach problem solving than men, both technical
problem solving as well as organizational. Both have merits, but a blend can
be very powerful, a diverse work group has intangible value.

I think rather than attempting to create a male avatar for women this lesson
needs to be embraced more widely.

------
andrewclunn
I wonder if the voice modulation had any discernible impact on communication
ability across the board.

------
vonklaus
Hiring is a complex process that is extremely subjective accross role,
organization and sub-sector of the tech industry. To the extent this result is
correct _within the population of interviewing.io_ ; it is in fact quite
positive on many levels, if unfortunate on others. Specifically-- again
accepting the methodology used here, interviewers are unbiased in their
reviewing.

If we consider simply this subset however, I suspect there are several factors
which contributed to this result:

* As indicated by others, it was unclear if this actually successfully modulated voice AND that other factors of influence did not leak gender to the interviewer.

* interviewing.io may simply attract men who are better than women.

* The current climate provides a lot of resources and support for women entering tech. Many organizations have made a large push to hire women and thus the top and mid tier women (who are significantly less numerous than male counterparts) are hired into organizations and thus not applying here.

* places where candidates learn about interviewing.io could differ based on gender. A contrived example being males learning about it on HN, while female counterparts learning about it through a short part-time coding bootcamp.

* there was an experience gap or significant skill gap between genders. This was alluded to above.

* women on interview.io are generally worse programmers or perform worse in technical interviews.

* not enough data for statistical significance.

This is a pretty interesting result and could actually be a positive thing for
interview.io. It is possible they are objectively evaluating candidates and it
is simply a marketing problem which they can adjust for.

There are also some non-trivial differences between men and women which likely
matter even in this context. Amy Cuddy does an amazing TED talk (and a longer
one as well) about poses and cues and their effects on perception.

If the idea of this investigation was to make a larger observation about the
industry, it would be interesting if they could correct for experience & skill
level possibly by something completely objective like HackerRank for example.
If seperated into 3 skill bands, it would be interesting to compare the actual
interview results accross similarly skilled populations. To correct for bias
it may be useful to tell interviewers the candidates will be anonymized
modulating both voices to try and have both genders voices sound alike as a
single neutral voice, ideally while still allowing for pitch and intonation.

Be interested to see a follow up as the site receives more candidates and
exposure and grows their organization, ect.

------
alansmitheebk
This is ridiculous. At some point we need to face up to the reality that most
women in the US are just not interested in working in technology. There is no
evil conspiracy to keep women out of tech. Maybe high school guidance
counselors and parents don't encourage women to go into tech. Maybe there is
someone to blame, but it's not the tech industry's fault.

~~~
balls187
Strawman argument.

The article delved into whether or not there was systemic bias against women
in tech interviews, given that interviewing.io's data showed that men
continued onto next rounds 1.4x more often.

The conclusion based on the experiment, that it wasn't due to bias, but
because women were more likely to not continue to refine their skills by
continuing to interview.

> When you do meet the rare American born female developer she is either: A. a
> designer who realized that there are no design jobs so she became a front-
> end dev; B. a first-generation Asian-American (B/C Asian parents steer their
> children into STEM); or C. it's a woman with blue hair and a pierced face
> who is there to "shake up the patriarchy" and make a statement about gender.

This is a sexist comment (and borderline racist).

------
dghughes
There's more to a person than the tone of their voice men and women speak
differently use different phrases, words and cadence.

------
auganov
Could you redo the classic study where you just replay the same performance to
every interviewer but change up the name to be strongly associated with either
gender? It's been widely replicated but still cool to see redone in a modern
setting.

------
bitL
So, is it like interviewing vocodered Smurfs?

------
thaumasiotes
How are they getting attrition numbers? I signed up with interviewing.io many
months ago, and never heard from them again. Sure enough, their website still
advertises "join the waiting list" rather than "join". Any significant
attrition should leave them completely devoid of users... right?

------
NTDF9
Here's an alternate hypothesis. Maybe, just maybe, men and women are
different? Maybe men and women have different strengths and aptitudes?

------
NetTechM
I liked this article, well written and thought provoking. I am curious if this
attrition effect plays a large part in graduate rates from STEM programs.

------
scotty79
Did women that quit after one or two bad rewiews have same avg score as men
who did the same thing?

------
yanilkr
Sweet that some one is trying new things with ideas.

But Oh boy, Life does not revolve around tech interviews. Make peace with it.
An old school psychologist would observe us and say that the people obsessed
with tech interviews faced an emotional toll of rejection during early career
and are trying to get over it.

Google, Facebook and other companies are like parents of the tech kids and the
kids are seeking some kind of approval or validation from them via passing
their technical interviews. You are not cool enough if you don't crack our
interview.

Its possible we are doing something very wrong here.

~~~
kylestlb
Maybe interviewing is over-important because of the frequency in which tech
workers/engineers change jobs.

~~~
titanomachy
Maybe un(der)employed programmers, who are naturally more concerned about
interviewing, have extra free time and are thus over-represented on hacker
news.

~~~
prodigal_erik
Even when I'm not the candidate, I'm asked to conduct a hell of a lot of
interviews at work, and it's a huge time sink when the signal/noise is poor.

------
koolba
_Interviwer: "Hi, my name is Roger. I'll be interviewing you today."_

 _Interviewee (modulated voice): "Hi Roger. Nice to meet you. My name is
Alice."_

Hmmm ... Speech hints aside, this may not work 100%.

~~~
onetwotree
Yeah, also any linguist can tell you that women use language differently than
men. Even in written communication you can usually, consciously or not, tell
unless someone is very deliberately trying to appear to be the other gender.

------
virgil_disgr4ce
I was excited about this until I realized the sample size is incredibly small
and there are few controls :/

------
sevensor
As we keep learning to our dismay, you can't solve social problems with
technology. I've got to give them credit for trying, and for reporting the
result, but I'm not at all surprised.

------
grb423
Is anybody studying the reasons behind the huge gender disparity in roofing,
welding or kindergarten teaching? I think there is a tremendous gender bias in
those and other fields that is going unstudied, because nobody cares or
because those fields aren't as _cool_ or _important_.

~~~
rcpt
Surgery has a large gap to but I rarely hear it brought up in these
discussions.

[http://www.ama-assn.org/ama/ama-wire/post/medical-
specialtie...](http://www.ama-assn.org/ama/ama-wire/post/medical-specialties-
vary-gender)

~~~
Bartweiss
That's a particularly interesting one, because it's not susceptible to the
usual charges of "no one cares because it's low pay/status". If you look to
radiology rather than surgery, the gender imbalance is enormous.

Suddenly we're looking at gender skew in two highly technical, prestigious,
lucrative fields, and seeing that they prompt totally different types and
amounts of discussion.

~~~
aphextron
It's almost as if men and women have _some_ fundamental differences in
psychology that affect their preferred choice of career.

~~~
bduerst
> men and women have some fundamental differences in psychology

This was largely debunked from a biological perspective [1]. Any psychological
perceptions you may be referring to are largely cultural and subject to the
biases that this product is trying to combat.

[1]
[http://www.pnas.org/content/112/50/15468.full](http://www.pnas.org/content/112/50/15468.full)

~~~
Udik
The study you linked doesn't absolutely say what you are trying to make it say
(that there aren't fundamental differences in psychology between men and
women). It seems to say that male and female brains and cognitive abilities
aren't as dimorphic as their genitalia, but instead there is a continuum and
overlaps for many distinct cognitive traits. In other words: (almost) all men
have normally formed penises, (almost) all women have normally formed vaginas.
Mixed situations are rare. Brains are another matter, genders mix and overlap.
That doesn't mean that there are NO genders though.

~~~
bduerst
>That doesn't mean that there are NO genders though.

The study isn't saying that there are no genders - specifically it is saying:

Our results demonstrate that regardless of the cause of observed sex/gender
differences in brain and behavior (nature or nurture), human brains cannot be
categorized into two distinct classes: male brain/female brain.

~~~
Udik
"most brains are comprised of unique “mosaics” of features, some more common
in females compared with males, some more common in males compared with
females, and some common in both females and males"

Nothing in this view says it's not possible that, for example, an interest or
a skill for engineering, while rare in both genders, might be n times more
common in men than in women.

------
anon2016
So it masks gender by making everyone sound "like a dude" according to the
videos. That's not masking gender. Make everyone sound like a robot and the
blog post title will be more accurate.

~~~
vonmoltke
The point of the experiment was for the interviewer to _think_ the interviewer
was a particular gender to see if they as it based on that impression. Making
everyone sound like a robot does not accomplish that goal, so I don't see how
it makes the title more accurate.

~~~
StavrosK
It makes everyone sound the same. Changing only women's voices is a tell-tale
sign. If you get a guy who sounds "computery", or "off", it's a girl.

~~~
lotharbot
They didn't only change women's voices. They also changed some men to sound
like women. And they made some voices sound processed without pitch changes.

~~~
sliverstorm
That could be a fun thing to break out at a party. I wonder what me sounding
like a woman, sounds like.

------
awesomepantsm
So their conclusion is that the people who do worse, use the product they
happen to sell less?

Sounds like bullshit to me.

------
jarmitage
Amazing how the author believes they have proved that there is 'no systematic
bias' with This One Simple Trick!

Edit: sorry for the tone, but I just find this whole article bait. The
headline is bait. The writer acknowledges the limitations of their study
(basically that it doesn't prove much) but then makes a whole bunch of
extrapolations anyway, and makes those guesses the meat of the article rather
than investigating why their methods didn't work or how they could be improved
or what would be required to produce conclusive evidence.

~~~
StavrosK
Yes, that's exactly what they say:

> On the subject of sample size, we have no delusions that this is the be-all
> and end-all of pronouncements on the subject of gender and interview
> performance.

But let's ignore science to fit our pre-existing biases!

~~~
dang
That comment was snarky, dismissive, and inaccurate, but please don't respond
with snark of your own.

------
myohan
cool but it's sad to know that this is required to maintain gender equality

~~~
huherto
It actually proves the opposite. That there was not systematic bias.

~~~
myohan
i think it is pretty obvious there is no systematic bias against women it's
more of reminiscent of an ancient culture of mistreating women...i think it is
rare to find in a modern/western corporate world, group of folks who
systematically and in organized way trying to oppress women

------
AndreyErmakov
So we all realize technical interviews serve no practical purpose and should
be quietly abandoned, but instead of looking into the future and discovering
better ways of identifying talent, people keep inventing stuff that makes the
torturing mechanism even more sophisticated so that the suffering can be
prolonged.

I'd like to see the people behind this project apply their technical skills to
something more useful to the industry and the society in general.

>> to get to pipeline parity, we actually have to increase the number of women
studying computer science by an entire order of magnitude

Any woman who's gone through that process will likely want to get another
career, that where people are treated with more respect. And I suppose many
men are having the exact same thoughts.

The more you mock your talent pool, the more actively that talent is running
in the opposite direction, just to get out of this mess.

~~~
MustardTiger
Why do people in tech think "proving you can do your job" is unique to tech?
Tons of industrues has practical interviews. Do you really think a roofer
shows up and says "I know what I am doing, just take me on my word" and gets a
job?

~~~
MustardTiger
throwaway_xx9 I don't know why your post is dead, but your example is
obviously silly. That is not applying for a job, that is contracting to an
individual. You don't face a technical interview contracting to an individual
in programming either. But if you apply for a __job __as a roofer with a
construction company, you absolutely have a practical interview.

