
Defending Against Hackers Took a Back Seat at Yahoo, Insiders Say - apetresc
http://www.nytimes.com/2016/09/29/technology/yahoo-data-breach-hacking.html?_r=2&mtrref=t.co
======
luso_brazilian
From the article:

 _> The "Paranoids," the internal name for Yahoo’s security team, often
clashed with other parts of the business over security costs. And their
requests were often overridden because of concerns that the inconvenience of
added protection would make people stop using the company’s products._

That's the best summary of the problem for the industry as a whole, not only
tech but any industry where failures are uncommon but with grave consequences.

A quote from Fight Club that illustrates that problem:

 _> Narrator: A new car built by my company leaves somewhere traveling at 60
mph. The rear differential locks up. The car crashes and burns with everyone
trapped inside. Now, should we initiate a recall?_

 _> Take the number of vehicles in the field, A, multiply by the probable rate
of failure, B, multiply by the average out-of-court settlement, C. A times B
times C equals X._

 _> If X is less than the cost of a recall, we don't do one._

That's the current mindset of the technological world, estimating whether the
cost of atoning for the problem is lower than the cost of securing the
systems.

~~~
Angostura
Calling them the 'paranoids' probably seemed like a fun idea at the time, but
I wonder if it set up a subconscious bias against their work. I wonder if they
had been called 'The Guardians' or 'The Defenders' there would have been a
different outcome.

Seems trivial, but words matter.

~~~
dsacco
The Yahoo Paranoids chose their own name. It was designed to be light-hearted
in a way that didn't make them seem stuffy so that engineering teams would be
more receptive to their work. In my experience, this is incredibly important
from the outset.

Anyone who has worked in information security for a _month_ knows that the
relationship between product engineering and security engineering defaults to
antagonistic. It takes a lot of work to make it friendly and productive, and
as a security professional I think "Paranoids" is much better for overall
collaboration than something like "Defenders", which in my opinion reeks of
self-importance.

The more pertinent issue here is management not fostering the culture enough.

~~~
Twirrim
Where I'm working now, we've got security engineers assigned to seating in
each development team.

They're not managed by, or working for, our teams. They have their own manager
and security work that they're getting on with.

Having them sitting amongst the team, however, is resulting in a much
different narrative than any I've been around before. There's a much higher
quality, and less antagonistic kind of engagement going on. They've become
someone you chat with at the watercooler, or at their desks, instead of having
to file tickets, or wait for scheduled reviews to raise things.

People can quickly consult with them and deal with a whole heap of small
potential risks way early on in the development process, and it's paying
serious dividends down the road.

~~~
bananarepdev
That approach Works well with Q&A too.

~~~
andersonmvd
You're talking about Squads basically. Bring different people in the same
group. And yeah, QA is very similar to Security in some points, but if you
think straight QA should include security. Weird to say that a software has
quality without security included, but the truth is that security is specific
that the regular QA usually can't handle.

~~~
rexpop
You've capitalized Squad, but it's hard to Google. Where did you get that
term, and where is it defined outside your head?

~~~
bananarepdev
As xxr said, Squad is how Spotify names their (previously Scrum) teams. Other
interesting concepts they use are "Tribes" and "Guilds". Take a look at the
Spotify engineering practices, they are really inspiring.

------
darklajid
Funny timing. My in-laws (both 70+) just had to change their passwords (both
using @yahoo.de email addresses) and my mother in law probably botched it /
managed to type the wrong thing twice or something.

Password reset requires 2 security questions (ugh - already ugly) and while
she's 100% certain that she knows the answer to both the second one isn't
accepted - probably another spelling issue (think St. Marlo vs. St Marlo vs.
Saint Marlo vs. Sankt Marlo vs ..).

All of this is her fault, not yahoo's. But now she's stuck. There are no ways
to contact support, at all, and by now her 'resolve this problem' links
already contain a "In rare cases like these, we suggest creating a new
account" line.

Anecdotal moral of the story: Yahoo has no customer support at all. Migrate
your elder family members away while you still can. :)

~~~
stronglikedan
I just had to convert my online bank account to their _new_ (2nd in a year)
system, which now requires five security questions. As a single man who
doesn't have a "favorite" anything, this proved to be a challenge. All
questions were either regarding spouses, favorite somethings, or questions
about descendants which I have no clue about.

I ended up just picking random questions and setting them all to the same
answer, which has nothing to do with any of the questions - something that I
can't even misspell if I tried. This is going to be my new strategy moving
forward.

And don't even get me started on their silly username and password "security"
rules.

~~~
kyleknighted
I use 1Password and randomly generate answers for each question and log them
in the "Notes" of the account.

I'm sure other tools like KeePass have similar sections to do the same thing.

That way the answers aren't reproducible and you have them safely stored
somewhere.

~~~
ascagnel_
As an added bonus, this means someone armed with the real answers to those
questions won't be able to get access.

When Sarah Palin's "personal" email was hacked during the 2008 election, the
attacker used her Wikipedia page and recovery questions.[0]

[0]: [http://nypost.com/2008/09/19/dem-pols-son-was-
hacker/](http://nypost.com/2008/09/19/dem-pols-son-was-hacker/)

~~~
spacehome
'hacking'

~~~
Shanea93
"to circumvent security and break into (a network, computer, file, etc.),
usually with malicious intent"

Just because they didn't impress you by finding a side channel timing attack
for the password hashing algorithm used by Yahoo, doesn't make it any less of
a hack.

Why spend millions investing in a network of computers to break encryption,
when the key can be gained far more easily with a $20 tire wrench applied with
sufficient force to the DBA's knee caps?

------
apeace
I can relate to a company not putting value on security, or thinking the cost
of securing systems may be higher than the cost of getting hacked.

I once worked for a company where I inherited a RESTful API. It stored the
company's core data, including private customer information. It had no
authentication, completely open for anyone on the internet to read or update
any of our data.

I alerted my manager about this and that made its way to the highest levels of
the company. The decision was to create a backlog item. It took about a year
before we got to it.

The reason we ended up finally fixing it was because we were contacted by a
security researcher one day. He said he had found a vulnerability in our
system, but wouldn't tell us what it was until we disclosed our bug-bounty
terms (basically promising to pay him if he had found a real vulnerability).
If we wouldn't do this, he was going to write a blog post about it.

My manager used some delay tactics to buy us some time, while we spent the
next 24 hours slapping a bandaid on the API. Once we had fixed it and agreed
to pay the researcher, he disclosed his vulnerability and it had nothing to do
with our API. It was a minor XSS that couldn't leak any sensitive information.

~~~
robk
That sounds a lot like blackmail

------
chollida1
I wrote a few days ago about how easy it is to compromise your ethics when
trying to save a company. The problem is that once you compromise once, its
very easy to do it again.

[https://news.ycombinator.com/item?id=12557163](https://news.ycombinator.com/item?id=12557163)

The problem is, it's way too easy to look past the action you are taking
because you can talk yourself into believing its for the greater good.

And this is a huge ethical breach by Mayer, if she did this way back then,
it's pretty reasonable to assume there are some more skeleton's hiding in the
closet.

I don't really think I'd be wanting to give Verizon a reason to reconsider the
takeover......

~~~
nordsieck
You might be interested in the research on "normalization of deviance". Most
of the solid research in in safety critical operations like aerospace, however
plenty of people have made parallels to other industries.

------
StavrosK
Can I just inject some perspective and say that the question would (should?)
have gone something like this?:

"So we got 500 million passwords stolen. We're using bcrypt with an adequate
number of rounds, so we only anticipate 1000 of those passwords ever being
broken. Should we issue a mass reset?"

It's never black and white, you have to weigh things against each other.

~~~
Bartweiss
Didn't the breach disclosure say "most" passwords were hashed with bcrypt?
Obviously I don't know what everyone else got, but it can't have been better
or they'd have said so...

I don't mean to detract from your point, good prevention beats reactionary
resets. It just raised my eyebrows at the time as a strange weasel word in a
claim that users were safe.

~~~
StavrosK
Now that you mention it, I remember that too. Seems weird, I don't know why
you'd have some passwords hashed in other ways. Even if you've migrated, why
not migrate everyone at once?

~~~
astrange
You need the user to login once to get their raw password to rehash it. Unless
you like rewrapping old hashes in every new one as it comes along.

~~~
StavrosK
Yep, exactly. You wrap them all in the new one, and migrate when the user next
logs in.

------
jgrahamc
Referring to the infosec team as "paranoids" is a really bad idea. I have our
infosec team report into me and they terrify me on a regular basis but they
are not paranoid. They worry, the poke around, they find stuff and they fix
it.

~~~
mtmail
The team calls themselves "paranoids" since at least 1999. I worked with them
and have only praise.

"We try to be somewhat lighthearted about security," [head of department]
said. "As important as it is, I also think it helps adoption if it is not too
serious." [http://www.zdnet.com/article/at-yahoo-it-pays-to-be-
paranoid...](http://www.zdnet.com/article/at-yahoo-it-pays-to-be-paranoid/)

------
CaptainZapp

      “At Yahoo, we have a deep understanding of the threats facing our users and continuously strive to stay ahead of these threats to keep our users and our platforms secure,” 
    

Why do I always get the almost unresistable urge to yell at my flat screen
whenever a corporate spokesdrone opens his or her mouth?

Is the ability to talk plattitude-gibberish a requirement for such a job?

~~~
dagw
_Is the ability to talk plattitude-gibberish a requirement for such a job?_

It's not just a requirement, it is their job. How would you re-phrase that
sentence in a way that A) isn't an actual lie, B) doesn't admit to any wrong-
doing, C) keeps your customers calm and D) keeps your shareholder calm.

~~~
Diederich
I generally agree with this, and it's been my consistent experience too.

But since I'm in full-on Elon Musk fanboy mode right now, have you seen in him
the exact opposite? He's extremely up-front about things, good, bad and ugly,
and it's so refreshing.

Additional tangent: I suspect that's what drew so many people to Trump and
Sanders. Though it's hard for me to put both of them in the same sentence
together, one thing they have in common is, at least, the appearance of being
straightforward. In Sanders' case, I think he _is_ truly being
straightforward.

</tangent>

~~~
dagw
_He 's extremely up-front about things, good, bad and ugly, and it's so
refreshing._

As much as I love Elon Musk as well, I have to admit that some of comments
around Autopilot and the accidents and possible problems surrounding it have
had a distinct whiff of corporate new-speak to them.

~~~
Diederich
I think I know what you're talking about, but to be clear, are you talking
about his 'stern' tone about driver error? I won't disagree with that point.

~~~
selimthegrim
Do you remember Ferdinand Piech telling Americans they didn't know how to
drive when the Audi acceleration thing was happening back in the late
eighties?

~~~
Diederich
I don't.

------
tptacek
I am not sure what end-to-end encryption would have done to defend Yahoo's
users against the entity that broke in and hoovered up its databases.
Similarly: the password reset situation is sad (understandable --- it would
have cost them millions of users at a point where their declining user base
was being carefully watched by the market --- but infuriating) but again, what
difference would it have made with respect to the most recent breach?

There are just a few companies in the whole world who both run tens of
thousands of servers _and_ are equipped to go head to head with serious
attackers. Yahoo isn't one of them. Has it ever been? No.

~~~
dsacco
In the interest of discussion, would you be up for naming those companies, in
your opinion?

~~~
tptacek
Google, Microsoft, Facebook, Apple.

Maybe 3-4 other "smaller" big startups that have anomalously great programs,
which I'm not going to name.

~~~
wmfiv
Seems like the banking industry has at least a degree of competency here as
well. Who would bother with identity theft if you could just hack the banks
and steal the money directly?

~~~
tptacek
Strong disagree. The reason you don't see tons of bank compromises --- apart
from the fact that banks don't routinely disclose breaches --- is that it's
harder to monetize a breach than people tend to assume it is.

Think it through. So: you "hacked" a "bank". You're on their internal network.
Talk me through "stealing the money directly".

~~~
wmfiv
Isn't that the point? If you can't monetize a breach of their network then it
seems like they're doing their job.

~~~
placeybordeaux
There are many ways to monetize a breach.

~~~
wmfiv
Yeah. I'm not suggesting that it's an A+ job well done. But at the same time,
relative to the target they are, seems like they're doing something right.

------
blakesterz
>>"...said the company spent $10 million on encryption technology in early
2014..."

What does that mean, exactly? Is it really possible to spend $10 million on
encryption or is that some kind of marketing spin on things? I'm genuinely
curious about this.

~~~
Naritai
I would guess that the bulk of the costs is installing the processing power
that enables the encryption.

~~~
blakesterz
Ah, I just reread that article and now I see:

"The current and former employees say he inspired a small team of young
engineers to develop more secure code, improve the company’s defenses —
including encrypting traffic between Yahoo’s data centers"

So that makes more sense now, I guess it's possible to spend that much on
"encryption". People and processing power are expensive I suppose.

~~~
Naritai
They really are!

------
at-fates-hands
Its the same issue since Grog tried to hide the first rock from Og:

 _The “Paranoids,” the internal name for Yahoo’s security team, often clashed
with other parts of the business over security costs. And their requests were
often overridden because of concerns that the inconvenience of added
protection would make people stop using the company’s products._

Infosec is never easy, and part of making things secure is that you give up
conveniences for peace of mind. It's 2016 and I'm still a but surprised that
people willingly open themselves and their data to hackers in lieu of stock
holders and customer retention. It's unreal when you stop and think about it.

~~~
CaptainZapp
It's the same problem compliance departments at financial institutions face.

Compliance, by definition is bad for business. They will, after all, try to
crack down on doing business with criminals, shady oligarchs or - dodgy
dictators, which can be highly profitable.

In the end, though, the best working banks are those where compliance has real
teeth.

------
tlogan
If Yahoo end up being successful like, for example, Slack or Dropbox these
security issues will not be at all discussed here.

I did not see any outrage when Slack security issues back in May 2105 [1]:
majority of people were saying "but it is great software". Dropbox the same.

So Yahoo did a similar bet as all other companies (focus on features - we will
fix security latter) but they lost that bet.

[1] [http://www.makeuseof.com/tag/slack-hack-need-know-
collaborat...](http://www.makeuseof.com/tag/slack-hack-need-know-
collaboration-tool-security-breach/)

------
droopybuns
Perlroth is gawker themed Krebs wannabe. Everyone involved in this story
should be ashamed for helping fuel an unsourced article that is purely CYA for
the former security team. Shame.

------
__jal
The iron cliches:

\- Security is a process, not a product.

\- You _always_ pay for security. Up front, after the compromise, or both, if
you're unlucky or bad at your job.

------
robertelder
If Yahoo had indeed positively identified the breach to have originated from a
'state-sponsored actor', it is possible that their thinking was something
along the lines of "Resetting the passwords wouldn't help us much anyway
against someone with so many resources."

Of course, I'm just speculating based on what I see in news reports. Perhaps
the 'state-sponsored' actor was just PR spin to save face? I really just don't
know what to think.

------
rdiddly
"Mr. Bonforte said he resisted the request because it would have hurt Yahoo’s
ability to index and search message data to provide new user services. 'I’m
not particularly thrilled with building an apartment building which has the
biggest bars on every window,' he said."

How about an apartment building where everybody's shit keeps gets stolen then?
Everybody tries like hell to move out, and the only tenants left are those
with no place else to go. Which on the internet is nobody.

------
natch
If this is true it taints Marissa and any business that hires her in the
future, because it's hard to think of a more stark example of putting the
interests of the user last.

~~~
MrPatan
Did she put the interests of the _customers_ last or first, though?

------
tracker1
I stopped using Yahoo the first time I setup an account for a friend and they
were already on Yahoo's spam email reseller list faster than I could disable
the opt-out setting. There was spam waiting in the inbox on an email account
less than 5 minutes old.

I appreciate some of what they've done for the larger community, but decisions
like that which make users take such a distant backseat to the bottom line
make me not want to be a yahoo user ever again.

------
amelius
See also:
[https://news.ycombinator.com/item?id=12563798](https://news.ycombinator.com/item?id=12563798)

Quote:

> Whenever mega-hacks like the Yahoo! fiasco hit the news, inevitably the
> question gets asked as to why the IT security systems weren't good enough.
> The answer could be that it's not in a company's financial interest to be
> secure.

------
erikb
Reality check: Security (especially IT security) takes a back seat in 90% of
businesses. The only exceptions are corps who gain significant power over govs
and users by being secure (I think Google and Facebook here), and when
regulations require a corp to do some kind of security fundamentals then these
are applied as necessary to avoid fines.

------
mathattack
_Google hired hundreds of security engineers with six-figure signing bonuses,
invested hundreds of millions of dollars in security infrastructure and
adopted a new internal motto, “Never again,” to signal that it would never
again allow anyone — be they spies or criminals — to hack into Google
customers’ accounts._

Wow! Security starts at the top!

------
topspin
That begs the question; what was in the front seat? Has Yahoo achieved
anything of note since the 90's?

------
sidcool
Things are not going so well for Marissa. She's a technical person and should
have known better.

~~~
geodel
I think she is financial wellness person and she is doing rather well for
herself.

~~~
sidcool
I think that beyond a certain level of financial success, it's the reputation
and power that matters more. Marissa will never have to worry about money in
her life. But power and reputation are ephemeral. Doesn't take much to ruin
them.

~~~
SilasX
Concretely, though, what consequence does this have in that respect? Is there
a marginal party she's not going to get invited to because of the loss of rep
from this breach? Is there an introduction she's not going to get?

Is there someone who matters to her who will deprive her personally (i.e. not
Yahoo) of something due to having remembered this event?

------
wiremine
I'm not a big fan of regulation, but it feels like there is very little
_internal_ motivation for a place like Yahoo to take security seriously.

Not sure what the solution is, but unless there is a financial reason to
create, I don't think we'll see much change.

------
bwb
Keep in mind shit like this is what happens when people get fired and blame
management, rumors & shit get started like this. We don't know the real story.

------
intrasight
Companies should have a Chief Risk Officer who have a big component of their
bonus based upon the success of their risk management strategies.

------
dumbfounder
Newsflash: struggling company doesn't spend time and effort on things that
don't directly make them money.

------
utefan001
I just returned from DerbyCon. An amazing security conference that covers both
attack and defense. All talks are on youtube. Here is a good summary of the
powershell talks. Really good stuff.

[https://blogs.msdn.microsoft.com/powershell/2016/09/27/power...](https://blogs.msdn.microsoft.com/powershell/2016/09/27/powershell-
security-at-derbycon/)

------
b1gtuna
Google hired hundreds of security experts with 6 figure bonus... Is this kind
of bonus norm in the Valley?

------
crudbug
Interesting how internal politics works. Breach news coming right after
aquisition. Blame the new owner.

------
chenster
How did they know it's the "Chinese military hackers" who's behind the attack?

~~~
wglb
Reading between the lines, it is the same attack that hit google way back
when. Google disclosed it, and disclosed that 20 other companies were
affected, and none of the other companies came forward.

~~~
rudolf0
Quite plausible, but what evidence is there to support that?

~~~
wglb
I don't have any other than there were 20 other companies who were not named
at the time, Google was hacked by chinese military, now Yahoo claims china.

Deduction on my part, but no further evidence than that.

~~~
rudolf0
Sure, but China has also (allegedly) been responsible for many other breaches
unrelated to the Google incident.

------
tn13
The way Yahoo! has been running I think their front seat is completely empty.

------
arviewer
Marketing by deception... It's postponing the inevitable.

------
omouse
Cannot wait for the class-action suit, I wonder if everyone across the world
can join if it's based in the U.S. or if they would need to create class-
action suits in their home country.

------
gist
> Google hired hundreds of security engineers with six-figure signing bonuses

Who left jobs working at other places and in theory left them more vulnerable
and drove up the costs for hiring as well.

------
CrankyBear
I never would have guessed!

------
informatimago
Can anybody cite a single one good decision Marissa Mayer took? Honestly?!

~~~
aikah
Good depends on the perspective. She made a bunch of people fairly rich by
buying their apps, even useless ones.

------
aswanson
Does Twitter seem to take security seriously?

------
jrochkind1
I haven't even read the article, but based on headline I'd say "AND 99% of ALL
companies EVERYWHERE." The headline makes it seem like this is an unusual
thing.

Nothing is secure, anywhere. A few companies actually prioritize it. Very few.

And I truly think the economy could not bear the cost if everyone actually
tried to prioritize security above all else.

------
drzaiusapelord
This is incredible. Hiding a breach like this is bad enough but to not even
forcing password resets should be criminal. I think we're living in an age
where information security is still in the cowboy stage of things. I think
we're due for some tough regulations here. Clearly businesses do not have our
interests in mind and in these cases our interest will conflict with theirs.

~~~
OedipusRex
Regulations would be impossible to enforce. You can write up as many laws as
you want but trying to enforce security constraints on new websites alone is
an impossible task.

~~~
bkjsbkjdnf
> trying to enforce security constraints on new websites alone is an
> impossible task.

Why 'new websites alone'? Enforcing security standards on new/small websites
would be harder than on large ones, and the large ones are more important.
There are already many cyber-security related laws that are in place and more-
or-less enforced. Having these laws makes large companies invest money into at
least attempting to follow them for fear of legal repercussions.

~~~
jsmthrowaway
> Enforcing security standards on new/small websites would be harder than on
> large ones

Wait, you think the larger a property gets, the easier it gets to secure it?

~~~
Naritai
I'm guessing s/he means that there are fewer total such sites, so any
regulatory body wouldn't have as many individual properties to monitor.

That is to say, it's a reference to the effort of the regulatory body, not of
the property managers themselves.

