
Facebook stored millions of Instagram passwords in plain text - mattkevan
https://www.theverge.com/2019/4/18/18485599/facebook-instagram-passwords-plain-text-millions-users
======
eridius
Literally every single time Facebook announces something like this, they
always announce low numbers and then repeatedly revise them upwards.

Is there no penalty to them for basically lying to everyone in their initial
announcement? Doing it once can be explained as they sincerely thought their
original number was correct, but they have a history of doing this and there's
no chance I believe now that they aren't doing this on purpose.

~~~
londons_explore
I think revising upwards is them looking for other bugs of the same type, and
finding lots.

I can totally imagine someone on the instagram photo pinch-to-zoom team saying
"Whoa - if someone re-logs in while a photo is partially zoomed, our metrics
capture the password. Thats bad!". They find thats super-rare, and only
impacts 10k users.

Later, other logfiles are grepped for passwords (possibly even by doing a
fulltext search across all fields and blobs for common passwords), and more
and more are found.

Then someone says "but some of our data is in fields base64 encoded - did you
try grepping for base64 encoded plaintext", and even more are found, etc.

~~~
eridius
You're giving them far too much credit.

~~~
antris
Benefit of the doubt.

~~~
eridius
Benefit of the doubt is something you give to a company the very first time it
does something like this. Not something you give the umpteenth time they do
something like this, and especially not when they try and hide their "sorry it
was waaaay worse" update amid other high-profile news.

------
hprotagonist
New rule: whenever facebook announces a breach and scale of effect, just
square the scale of effect right off the bat. You'll probably still be a
little low, but you'll be a lot closer to the number in the second (or third
or fourth) press release.

~~~
mgraczyk
1e6^2 = 1e12, trillions of passwords? ...

~~~
zephyrnh
"The security lapse was first reported last month, but at the time, Facebook
said it only happened to “tens of thousands of Instagram users,” whereas the
number is now being revised up to “millions.” The issue also affected
“hundreds of millions of Facebook Lite users” and “tens of millions of other
Facebook users.”"

~~~
sreyaNotfilc
I wonder how much of this is the responsibility on how Instagram was built.
Like Instagram may have been built so fast that those things never were
accounted for.

Not that I'm giving Facebook excuses, but maybe something was overlooked
before acquisition. Then again, Facebook is breaching security protocols left
and right.

------
hmate9
"...various errors seem to have caused Facebook’s systems to log some
passwords in plain text since as early as 2012."

1) As a software engineer I can't imagine how such errors could possibly have
entered production code accidentally, especially after code review. If precise
details of these errors are released I am open to have my mind changed.

2) Even if it did, I can't see how it would take facebook's tens of thousands
of engineers 7 years to find this "bug".

~~~
kevingadd
During a job interview at one mid-size startup one of the interview questions
involved handing me a 2 page excerpt of server logs and asking me to identify
bugs/issues from the messages and tracebacks in the logs. (neat idea!)

The logs contained user credentials, and they hadn't noticed. I pointed it out
to the CTO.

~~~
stevenwliao
Wow, their interview question (in the hands of the wrong interviewee) could
have caused a data breach.

~~~
londons_explore
If it's a plaintext password, the minute it hits _any_ human eyes, it is IMO
compromised and the user should be required to change it. Employee,
Interviewee, Hacker - doesn't matter.

~~~
itslennysfault
I'd go a step further. IMO the second it hits a hard drive it's compromised so
just the act of having that in the log would count.

------
hunter23
Obviously you shouldn't log passwords, but what kind of security mechanism
have been effective to catch this error?

One thing I can think of is have some production test accounts that are
regularly used and have a unique password. Then have an automated task that
periodically greps the production logs for the password to see if you have a
log leak.

Any other approaches?

~~~
human20190310
You could grep production logs and other data stores for commonly used
passwords like "password123".

~~~
kevin_thibedeau
More dependable would be tests that inject unique privileged information and
then check for that appearing anywhere it shouldn't.

~~~
londons_explore
You need to do both.

Real users will find routes through your app that automated tests never
consider. For example, what if the users 1-year login cookie expires as
they're on step 3 of a 4 step "change your avatar" process. That's highly
unlikely, and probably not even tested. Yet it might well work (due to good
modular design), but also log inappropriate data.

------
sreyaNotfilc
OK, this is getting ridiculous. There is no best practices. You think a
company worth Billions would have a standard of ethics.

Afterall, that's what they interview on. Best Practices. Ethics.

Doesn't Facebook have blogs on things like efficient servers, algorithms and
such. They are obviously at the forefront of technology (or know about it).
Why aren't they implementing these things?

It seems to me that the more your company is worth, the less you get to bend
rules, all to chase more money. HR an lawyers will take care of the rest.

------
jordache
Maybe instead of quizzing obscure sort algorithms at interviews, they should
have simple yes/no questions such as:

Should you store plaintext password on the server side?

------
basetop
Earlier post of the exact same story.

[https://news.ycombinator.com/item?id=19693233](https://news.ycombinator.com/item?id=19693233)

------
anarazel
And they announced it while they thought the attention was elsewhere (due to
the Muller report release), and in an easy to miss manner (as an update to a
blog post). Very trust inspiring.

~~~
bhelkey
Edit: The following is correct but misleading. The Muller report was reported
to be released before the Facebook security update.

Facebook announced this before the Muller report was released.

The Facebook security post was updated April 18, 2019 at 7AM PT [1].

The Muller report was released "Thursday morning, shortly after 11 am [EDT]"
[2]

[1] [https://newsroom.fb.com/news/2019/03/keeping-passwords-
secur...](https://newsroom.fb.com/news/2019/03/keeping-passwords-secure/)

[2] [https://www.vox.com/2019/4/18/18411966/mueller-report-
releas...](https://www.vox.com/2019/4/18/18411966/mueller-report-release-time-
trump-russia-redactions)

Disclosure: I work for a big tech company but not Facebook. All opinions are
my own.

~~~
k_sh
Sure, but the Mueller report was widely reported[0][1][2] to be released
Thursday morning days ago.

[0]: [https://deadline.com/2019/04/redacted-robert-mueller-
report-...](https://deadline.com/2019/04/redacted-robert-mueller-report-
coming-thursday-morning-donald-trump-bill-barr-russia-1202596040/)

[1]: [https://www.washingtonpost.com/world/national-
security/muell...](https://www.washingtonpost.com/world/national-
security/mueller-reports-release-is-expected-thursday-justice-dept-
says/2019/04/15/dd44eb02-5f91-11e9-9412-daf3d2e67c6d_story.html)

[2]: [https://www.cnbc.com/2019/04/15/mueller-report-expected-
to-b...](https://www.cnbc.com/2019/04/15/mueller-report-expected-to-be-
released-thursday-morning.html)

~~~
bhelkey
I wasn't aware of this. I updated my comment to reflect this.

------
wickrom
How does Facebook, hiring some of the _best_ engineers in the industry with
the biggest bucks ends up in such a mess

~~~
happppy
maybe they need passwords to login as user?

~~~
jordache
even if they want to emulate a user, they can do so without their password. Or
if doing so in a simplistic way, they can login with said user's hashed
password, instead of the original plaintext.

------
johnsimer
I feel like this could be avoided by hashing the password on the client side
as well before sending it to the server, no?

~~~
blattimwind
Yes, but while "storing passwords in plaintext is bad, mkay?" is accepted in
the industry, "transmitting passwords to the server is bad, mkay?" is, in
fact, not accepted. It is usually considered "best practice" to transmit
passwords to the server, which is obviously wrong.

~~~
itslennysfault
That doesn't work. If you hash on the client then the hash IS the "password"
and is thus sensitive info that you'd need to hash again on the server and
shouldn't be stored anywhere in plain text.

~~~
kalleboo
While I agree with what you say, one could argue there is the additional
element of protecting against user password reuse between sites.

If the client-side hash is strong enough it means that your plaintext password
leak becoming a hashed password leaks at least protects the user from reuse on
other sites.

------
xfitm3
If you do layer 7 NSM/DPI then you're most likely logging passwords. The good
news is this data typically has significantly stronger controls over access
logs as it's collected for secops.

------
EastSmith
I am wondering why sending onetime login link to an email is still not a
thing?

