
Facebook Stored Hundreds of Millions of User Passwords in Plain Text for Years - snaky
https://krebsonsecurity.com/2019/03/facebook-stored-hundreds-of-millions-of-user-passwords-in-plain-text-for-years/
======
pizza
> My Facebook insider said access logs showed some 2,000 engineers or
> developers made approximately nine million internal queries for data
> elements that contained plain text user passwords.

This imo is the truly alarming takeaway. FB employees were _retrieving_ user
passwords? Around _two thousand FB employees?_ How in God's name is Zuckerberg
going to perform his usual performative contrition about that one?

I'm just trying to imagine the data structures that were being retrieved from
databases. Either they stored something like a big user account data type that
contained their password in plaintext, which imo is a really weird design
choice, or logs for other services were being mixed in with logs leaking the
user/pass combos.

Surely one of the engineers could have noticed and said 'wait a minute...
those are logins' over the course of the years? We hear all the time that
people want FB to follow a responsible social practices (the debate on what
those are rages on, which is great imo), but can't FB at least wrangle its own
code base?

On the other hand, we shouldn't take the stance that heads should roll, imo -
it would just create a chilling effect that would deter other companies from
ever going public about their own security mishaps.

edit: I should probably tone it down in this comment but I'll leave it for
posterity

~~~
IshKebab
No you're misrepresenting what (probably) happened. They had some system that
logged API requests (fine and normal). Some API requests include plaintext
passwords (also fine and normal).

The issue is presumably that they had no exclusion to the logging for
sensitive information like passwords, which is honestly very easy to overlook.

So two thousand Facebook employees were not "retrieving passwords". They were
looking at the API logs, which is a normal thing to do.

~~~
planb
This. It's a bit baffling to me that even the HN audience doesn't come to this
conclusion immediately - maybe people want to assume the worst because it's
Facebook?

~~~
IshKebab
Definitely. This always happens when Facebook comes up here. People forget to
use their brains and just rant about how evil Facebook is.

------
the_duke
(Edit: reworded a bit to make it clear I don't think this is acceptable)

Sounds a lot like some service was logging the full body of a signup/login
request, which then was readable for anyone with access to the logging/tracing
infrastructure.

Dumb mistake but it's not hard to imagine this happening, considering that FB
probably has a bunch of services involved in the login/signup flow to prevent
bots/spam, abuse, etc.

Not to imply this is acceptable, especially at a IT company like FB with vast
resources and know how. Raw passwords are an especially big screw-up. There
are a lot of failures here, from actually logging something so sensitive over
giving access to so many employees to not noticing this for years. (Assuming
this was actually log data).

BUT if we are honest, anonymizing log data is rarely a priority. Even if it
is, leaking sensitive data can happen easily in a lot of different points in
the infrastructure. In actual application code, client + server exception
tracing (just imagine a deserialization exception which contains part of the
input) , web server, load balancer, proxies, service mesh... There is a lot of
interesting stuff hiding in the logs at pretty much every company.

This is a good time to look in the mirror and audit your logging and tracing
data. Unless you are in a highly regulated field like finance/healthcare or
there is a strong company-wide culture for security/privacy with regular
audits already, I can almost guarantee you will find at least one data point
that should not be where it is.

Protecting sensitive data needs to be a big consideration for every dev, ops
and especially management, which has to allocate enough time for security
reviews and audits.

~~~
CaptainZapp
On a scale of Facebook with literally billions of users this is pretty much
unexcusable.

This community usually scoffs at entities, which store passwords in plain
text. Why do you give Facebook a pass?

~~~
EnFinlay
Because there's two types of storing passwords in plain text. There's the
"your password is stored in plaintext in the database" way which everyone
agrees is 10 different kinds of stupid, and then there's the "we accidentally
logged the body of all requests that went through this system, and it turns
out login requests came through here" kind. One is a bad security decision
(because you must decide how to store passwords in the DB), and the other is a
very easy mistake to make which can go unnoticed for a long time (because you
can be attempting to log things completely unrelated to logins).

Same same but different.

~~~
EpicEng
I've worked in healthcare / biotech for more than a decade and I can promise
you that the FDA would see no difference between the two types of gaffes. As
custodians of sensitive information it was our responsibility to ensure that
said information didn't leak, period.

I don't know anything about FB's infrastructure, but when I was lead I would
have viewed leaks in logs as far worse than something in the DB because our
DB's were harder to gain access to.

I get what you're saying, but it's irrelevant. Easy to screw up, hard to screw
up, doesn't matter; just don't screw up because the result is the same. This
stuff is security 101. If you're logging requests then you need to ensure they
don't contain sensitive info.

~~~
pnutjam
This is exactly how Zuck was able to steal login information in his stories
about founding facebook.

I find it hard to believe this is accidental.

~~~
kerng
Do you have a reference? I was wondering the same but have no reference.

~~~
freeflight
[https://www.businessinsider.com/henry-blodget-okay-but-
youve...](https://www.businessinsider.com/henry-blodget-okay-but-youve-got-to-
admit-the-way-mark-zuckerberg-hacked-into-those-email-accounts-was-pretty-
darn-cool-2010-3?IR=T)

------
terracatta
I think the technical security problem here (properly whitelisting parameters
in logs) is just a symptom and not the core underlying concern (as the article
mentions Twitter and Github just dealt with similar issues)

To me, it seems very likely someone before 2019 laid eyes on these logs and
either:

    
    
      a) Decided not to report it (implies serious security culture issue)
     
      b) Reported it and no action was taken (implies serious security process issue)
    
      c) Didn't even acknowledge it was inappropriate (implies a serious security training failure)
    

If you've already become complicit towards regularly violating the privacy of
your end-users, one can easily understand an employee devaluing the
seriousness of clear-text passwords in a log.

Are FB employees so regularly exposed to sensitive data that they have become
desensitized to the seriousness of clear-text passwords in an internally
accessible log?

~~~
Bhilai
This is very basic stuff, all user-identifying information should be tokenized
before being logged. Controlling access to production login logs or exposing
them on only a need to know basis is another basic security principle. Sounds
like FB is actually a wild wild west internally.

~~~
sydd
What is now a common practice was nonexistent a few years ago. 20 years ago
you could bypass Windows login with a few clicks. 15 years ago CORS was
nonexistent. 15 years ago it was common to send sensitive data
unencrypted..and so on.

Ex Facebook people told me that until around 2010 FB management turned a blind
eye on its employees digging around their databases. Then they released a
warning that people should stop and started locking prod data down. A few
months later those who still peeked around prod data were let go.

------
madrox
As the article mentions, Twitter disclosed a similar mistake a year ago...and
GitHub before them. These are just the organizations responsible enough to say
something.

Can we take a moment to acknowledge that this is an easy mistake to make? A
logger doesn't care if it's a password or not. Strings are strings. As long as
the answer is "humans should be more careful" we'll be seeing these kinds of
disclosures regularly across the industry.

My best attempt to address this in my teams has been to use different data
types for different data classifications. Naked strings must be loaded in one
of these data types after input sanitization. That makes it easier to catch
accidental inappropriate use. This is useful for managing PII as well.

~~~
gwbas1c
That's why modern languages need something like a typedef!

At least C# has a concept of a SecureString for things like passwords.

~~~
saagarjha
How does SecureString actually work? Presumably it's possible to coerce one
into a "normal" string, and I'm sure that someone will end up doing this in
the codebase because it's more convenient to work with.

~~~
gwbas1c
The point of SecureString is to ensure that sensitive information gets removed
from RAM in a timely manner. It's just a wrapper around some native memory
that gets zeroed when the SecureString object is disposed.

Otherwise, when you use standard C# strings, you don't know when the garbage
collector will collect them.

(Granted, you could do similar things with pinning a char[] to a char* and
zeroing it before you end the pin, but who wants to jump through those hoops?)

~~~
saagarjha
But what I want to pass a SecureString to someone? Doesn't a copy have to take
place in that case?

------
Ajedi32
Yet another example of why its important to use _unique_ passwords for every
site you have an account on. Even if the site you're using does password
storage properly, that's no guarantee plaintext credentials couldn't leak
through other means, such as, in this case, improperly configured logging
systems.

In the future, WebAuthn may be able to solve this problem for good, as sites
will only have access to a unique public key rather than a plaintext password.
Until then, a password manager is your best defense against this type of
issue.

~~~
xtracto
This is so prevalent in technology companies that it is funny to read this
thread with everybody throwing mud at Facebook without considering that most
probably the company they are working on has had (or maybe even currently has)
the same issue.

~~~
nabakin
That's a fallacy my dude. You can throw shade at your own company and Facebook
for the same reason. It doesn't and shouldn't minimize what Facebook has done.

------
krisrm
Posted this on the other thread from Facebook, but at what point do we start
imposing strict fines on companies that are found to have done this?

Granted, I guess we wouldn't be hearing about this instance at all if there
was to be some sort of fine attached - it would have just been swept under the
rug - so maybe that's not a good idea. I'm just tired of the "oops we stored
your passwords in plaintext lol" from companies with engineers that should
clearly know better.

~~~
segmondy
When we can start fining you for your mistakes, when developers can start
getting fired for any mistakes immediately.

I don't care about Facebook, but storing user passwords is the plain is not
"privacy violation" without the password they still have access to all your
data. storing user passwords by logging it is stupid amateur security mistake.

I can understand if Facebook stored the password and used it to access your
other accounts with permissions to invite users then sure fine em. But
mistakes are mistakes, they owned up to it.

~~~
enraged_camel
>>When we can start fining you for your mistakes, when developers can start
getting fired for any mistakes immediately.

Sounds good to me. Plenty of people in other industries get fired all the time
for violating best practices, regulations, and so on. Why should software be
any different? Are we special?

~~~
chillacy
Most tech companies have a “blame the process, learn, and fix the process”
approach. I’m not sure what industries you’re talking about but manufacturing
and aviation seem like they have a similar process.

------
Bhilai
So they logged plaintext passwords, what a dumb mistake. To top it all it
sounds like 20K employees had access to these infra components. I have had my
doubts on FB's internal access control model for a while now. Specially after
one particular story where an employee claimed to have internal access to
private information and used that to stalk people on Tinder -
[https://www.wsj.com/articles/facebook-fires-employee-who-
bra...](https://www.wsj.com/articles/facebook-fires-employee-who-bragged-on-
tinder-about-his-access-to-user-data-1525294488)

------
dimtion
This seems like the passwords were inadvertently written into some log files.
While this is a very bad security issue for a company like Facebook, I am
pretty sure that this type of bug is much more prevalent in the industry than
we would like to assume.

~~~
EnFinlay
It's just such an easy mistake to make. Some new ops guys logs all the traffic
through a subsystem and boom - plaintext passwords.

~~~
wy35
An easy mistake with serious potential consequences. I'd be interested in
hearing how Facebook will prevent this from happening again in the future

~~~
sethammons
A lot of folks are angry about this, but I'm not seeing much in they way of
"this is how that should be handled" aside from "be vigilant." The closest is
"encrypt logs," but those logs exist to be seen, else they wouldn't exist.

~~~
qpiox
Hash client side. Send hash.

This is how it is done for 20 years (by those who truly care about the users).

~~~
crazygringo
What?! No!!!

As many have said before, transmitting the hash simply turns the hash into the
password itself. Anyone who has your hash, has your password.

The biggest reason for hashing is that, even if an attacker gets access to the
hashed passwords, they still can't use them to log in. Client-side hashing
completely and utterly undoes that benefit.

Passwords in transit should be protected by SSL. Not a hash.

~~~
jeltz
You should check out SCRAM. It solves this problem without sending the hash or
the plain text.

~~~
crazygringo
Fascinating. Yes that does solve the problem, and most importantly doesn't
send the hash (unlike what the commenter I was responding to was suggesting,
which continues to horrify me).

Most interestingly, it solves the problem where the _server_ may not be
trusted. I see how this would have protected against the Facebook problem.

I'm curious, have any major websites you're aware of adopted SCRAM as a best
security practice? It feels kind of like overkill, since generally the server
is considered to be trusted... but at the same time, it would definitely
prevent accidental logging.

~~~
jeltz
I have not heard of any website using SCRAM, but recent versions of PostgreSQL
use it for password authentication.

------
runesoerensen
Facebook's own announcement is titled _" Keeping Passwords Secure"_[0]

Facebook comms: You can't really use the word _" keep"_ when the entire post
is about how you have failed to keep passwords secure, by storing it in
plaintext.

Or at least, you probably _shouldn 't_ use that word. It signals that you
intend to keep doing what you're doing, which wasn't just keeping passwords
insecure up until sometime in January.

It's up until today when you effectively disclosed that you haven't notify
hundreds of millions of users that their passwords were compromised _for
months_ after discovering it. I know they _" were never visible to anyone
outside of Facebook"_, but that group still includes some +25K people.

[0] [https://newsroom.fb.com/news/2019/03/keeping-passwords-
secur...](https://newsroom.fb.com/news/2019/03/keeping-passwords-secure/)

~~~
CobrastanJorji
Facebook comms knows that. That's why Facebook comms used the word "keep."
It's also why they said "some" passwords and not "hundreds of millions" of
passwords.

Facebook comms also knows that "there is nothing more important to us than
protecting people’s information" is a bald-faced lie, since Facebook is
literally in the business of selling people's information to companies like
Cambridge Analytica, but sometimes it's okay to lie as long as it sounds like
a meaningless platitude so that nobody would be expected to seriously believe
it.

------
arduinomancer
To me it’s no surprise these things happen considering these companys only
care about hiring leetcode experts.

Who knew data structure and algorithms doesn’t qualify you to build a secure
real world web application?

Knowledge of practical things like security has zero value to these companies
when hiring engineers.

~~~
tschellenbach
Well if you hire enough people, and you're not 100% successful at selecting
the good ones... Also even good people make mistakes at times. Almost
impossible to prevent. The most actionable thing you can do is build less
things, and focus on using of the shelve micro services (either in-house or
third party) for everything

~~~
therealdrag0
But as someone else said, even though it maybe a common mistake (I've seen
similar mistakes at my work), how is it that a number of those thousands of
engineers didn't raise the alarm and get traction sooner on securing it?
Companies need to incentivize engineers to not have a "not my problem"
attitude.

------
hammock
Reminder that Facebook also stores(stored?) 3 separate hashes for your
password: passwoRd, PASSWOrD, and PasswoRd. They weren't super transparent
about this fact.

Source: [https://www.zdnet.com/article/facebook-passwords-are-not-
cas...](https://www.zdnet.com/article/facebook-passwords-are-not-case-
sensitive-update/)

~~~
g45y45
You don't need to store 3 hashes, infact, you would need many hashes for all
possible cases. It is easier to drop all cases pre hashing, and only store a
lower case hashed value.

~~~
dymk
Hashing the the lowercase version of the password string is not the same as
what they’re doing. What you’re suggesting greatly reduces the security of the
password, what they’re doing only divides the search space by 3 (which is
nothing).

------
rhegart
My Instagram was compromised a couple of years ago. My password was very
secure and completely unique to Instagram so I found this extremely odd.
Someone had logged in from the Bay Area. I believe it was compromised twice
within 24 hours and then my Facebook was too with a completely unique password
as well. Facebook emailed me to change all passwords. I have perhaps over 50
acquaintances working at Facebook. Could someone internally have accessed it
as a prank? I don’t remember all the details but I did change my bank account
immediately after as well just in case.

------
__ralston3
I find it pretty shocking that other commenters are looking at this as
excusable. I mean, is that OK/excusable at your company? Logging
payloads/bodies of sensitive requests in plain text - 0 obfuscation. That's
ok? Wow. Other commenters are saying "it's logging so it's a forgivable
mistake". Is it though? Obviously the world won't end because of these
decisions, but holy hell I can't believe this wasn't caught/brought up in some
type of code review. This seems pretty 101-ish

~~~
city41
This kind of thing can often be hard to catch in a code review, because often
it's the combination of several systems that cause this to happen. Tracing the
user's password from submission form all the way to logger would probably
require jumping through several layers, most of which are just handed black
box blobs that they hand to the next system.

------
sebringj
Personally as a developer I'm not thinking Facebook is bad but rather that in
general we are bad and we pass over plain text passwords to endpoints (through
SSL) and just trust the data will not be abused (logging) but would be cool to
have client-side encryption prior to sending to any 3rd party so figuring out
a way to do that cleanly on different devices where a password, although the
same everywhere possibly but is always encrypted uniquely for a particular
company prior to going over the wire so even if compromised internally would
only be for that particular service. This is probably just a legacy problem in
the end where a new approach would simply make this problem obsolete like a
yubikey or built in protocol at some point.

~~~
sebringj
All the answers I see on stack say there is no gain to do hash or encrypt
prior to sending. I'm not sure I can agree with this. Let's say you hash a
password and some attacker gains this. They can now log into that website BUT
NOT everyone else for your username if you happen to use the same password.
Jeez am I wrong here?

~~~
johnnyRose
I have looked into this in the past and came to the same conclusion.
Essentially, the "password" sent to the website is the client-side
salted/hashed version of the user's actual password. The server could then
salt/hash the "password" another time before storing it. This could result in
the same issue if the "password" is logged, but it protects the user's true
password from being discovered. Maybe a security expert can weight in on this,
because I don't understand why this wouldn't be the standard.

~~~
lixtra
Ideally the server doesn’t need to remember the salt it sent to the client, so
it should be signed together with a timestamp to avoid reuse.

While you’re at it you can also add some hash puzzle to be solved by the
client increasing difficulty with failed logins.

------
AJRF
Facebook posted a response with the title "How we secure passwords".

The euphemism of company press releases always make me laugh.

------
prophesi
What irks me is that they're not going to issue a mandatory password reset,
since they only do so "in cases where there’s definitely been signs of abuse."

Even if it's an internal server, having passwords stored in plain-text always
merits a password reset. Storing it on a file-server unencrypted somewhere is
just as bad (and arguably worse), than storing them in plaintext in the
database.

------
ravenstine
_We 're sorry._

~~~
jayess
_We take security very seriously._

~~~
snaky
> There is nothing more important to us than protecting people’s information,
> and we will continue making improvements as part of our ongoing security
> efforts at Facebook," Pedro Canahuati, the company's vice president for
> engineering security and privacy, wrote in the post.

------
duncans
Twitter found they did the same thing last year
[https://news.ycombinator.com/item?id=16989534](https://news.ycombinator.com/item?id=16989534)

~~~
tqi
Same for Github:
[https://news.ycombinator.com/item?id=16974851](https://news.ycombinator.com/item?id=16974851)

1 comment, "Crazy level of transparency, they noticed, fixed it and notified
the users even though this was an internal leak. I bet most businesses would
just sweep it under the rug."

------
softdevfromerth
Some number of Facebook employees ABSOLUTELY knew this was going on, quietly
talked amongst themselves, maybe brought it up, but then it was knocked back
down. 100% I guarantee it. One company I worked for had plain-text passwords
and it was quietly discussed (or not discussed) until we EVENTUALLY fixed it
(years). Fuck facebook with a rusty pitchfork. This is the tip of the iceberg
with their data-handling practices. I bet their systems are fucked throughout.

------
craftoman
I remember when I was in Greek college and we were building a login app with
PHP and MySQL. We actually were storing every password in plain text and I
asked my professor why shouldn't we hashed them for better security and he was
like "that's not necessary cause we overload the server hashing all those
passwords". For a moment I thought I was paranoid but then I start thinking
how vulnerable some systems may be.

------
scriptkiddy
How do companies keep messing this up. The cardinal rule of web application
security is to NEVER store plain text passwords anywhere. The only time your
application should have access to plain text passwords is when it is hashing
the password or verifying the password against a hash.

If you need to log all request data for some reason, strip the passwords out.

It really isn't difficult.

~~~
qpiox
In properly built software, the clear text password never leaves the client
(browser in this case). There is no real need to have the password sent over
the net.

------
hetspookjee
Meanwhile the stock of facebook has risen with 0.5%. How does this even work?
This is enormous lack of oversight and proper processes.

In addition the original post of facebook is rich in media speak:

First they speak of _some_ users and _readable format_ and next paragraph they
speak of _hundreds of millions_

We estimate that we will notify _hundreds of millions_ of Facebook Lite users,
tens of millions of other Facebook users, and tens of thousands of Instagram
users.

As part of a routine security review in January, we found that _some_ user
passwords were being stored in a _readable format_ within our internal data
storage systems.

With this technique, we can validate that a person is logging in with the
correct password without actually having to store the password in _plain
text_.

------
crankylinuxuser
Better question:

As a system administrator, how does one find issues like "improper logging
that leads to credential capture"? And how do es one find credentials for
internal services crammed in and hidden in old crufty source tomes?

Obviously, when you find it, you nuke the creds, change them, and fix it. But
you can't automate, because that requires the raw data. And if you hash the
passwords, you end up having to hash everything in order to perhaps find the
bad hashes. And when dealing with MBs/s of log, you can't offset-hash
everything to scan... and you're back to raw text-matching passwords..

How do others do this, so we can do the right thing?

~~~
prophesi
> Obviously, when you find it, you nuke the creds, change them, and fix it.

The status quo is to toggle a boolean on their account. If they sign in and
this boolean is true, then don't actually sign them in. Tell them they need to
change their password first.

Then you'd send out an email to the account holder to change their password.
If they actually own the email, and aren't an imposter, then you're good to
go.

So in terms of automation, it's literally just a boolean check during log ins,
and make sure you're actually doing password resets correctly.

~~~
crankylinuxuser
You've reduced the scope of my original question, to that of compromised
accounts. My question is a magnitude larger than that.

For example, someone checks in code and left a config file semi-populated
which included a live login credential. Obviously the answer is "Dont do
that!" but we all are human. People accidentally post apikeys, credentials and
other secure things. It happens.

But how do you automatically _find_ issues like this? If PII is radioactive,
credentials are gamma-emitters.

We solved this at a job long ago with a list of plaintext passwords on a
machine you couldn't log into except local (butt-in-seat), and it took your
SVN commit, scanned it, and gave a pass/fail. That methodology at least
stopped our internal shared service accounts from being accidentally
compromised.

~~~
prophesi
Ah, sorry about that. I got hung up on the second half of your post. Can't say
I can think of a good automated solution for bad security hygiene among
employees. :/

Though if you guys primarily use Git, this does wonders:
[https://github.com/zricethezav/gitleaks](https://github.com/zricethezav/gitleaks)

The only thing I can think of that would help prevent credential compromises
is to either implement a company password manager (akin to your butt-in-seat
solution) with an ACL, and only accessible on the local network. That
shouldn't be too much friction for employees to actually utilize it.

Next is having a secure channel to transmit secrets. That + Password manager
has personally helped stop my coworkers in the past from sending passwords in
emails, slack messages, post-it notes, text files on their computer, a
committed file in a repo, etc.

~~~
crankylinuxuser
That's fine, it happens :)

Yeah we've seen that git module. We use something similar to that.

I was also thinking inadvertent log capture as well. Capturing creds in an ELK
stack is just as bad. And I'm guessing this FB issue is some variation of
complete logging snarfing creds.

I'm just trying to think of coherent ways we can scan, detect, and act on
creds that work across a backend. Seems to be a rather large hole.

------
mmcclellan
Facebook's press release [https://newsroom.fb.com/news/2019/03/keeping-
passwords-secur...](https://newsroom.fb.com/news/2019/03/keeping-passwords-
secure/) on the matter is pretty nonchalant:

>Keeping Passwords Secure >As part of a routine security review in January, we
found that >some user passwords were being stored in a readable format >within
our internal data storage systems.

Nothing to see here folks.

------
Zenst
Sadly many companies did in their early days from that period (even when it
was known to be bad). Blackberry had plaintext passwords stored in a postgres
dB up until 2.0 of BES (circa 2007). More so as no constraints (yes many many
people had a password of "pass","god" and the like).

So let us not forget legacy systems we have all encountered or read nostalgic
stories about how old XYZ kit is still in use today.

Or to put it another way - the amount of work that went into Y2k, just to fix
a date field, fixing a plaintext password is more work and we have had no
password2k like event and any event is local with some company falling foul
and by that - getting hacked and data shared publicly.

So even today, you can guarantee that their are still systems out there using
plaintext password storage. Some will be legacy hangover reasons, but not all.

But it sure does make you think, how many legacy systems are still in use
today and by legacy, they don't even have to be that old in some fields or
work. Sure does make you wonder about satellite security given the age of some
of those still operational and in orbit.

------
thefifthsetpin
Why is it normal to not hash passwords in the browser before sending them over
the wire? Seems like it's opening the door to issues such as this one, or even
more serious ones like when cloudflare leaked web server memory into other
sites.

I understand that you'd still need another hashing layer before inserting it
into the db, but wouldn't some hashing client-side be useful?

~~~
mic47
This does not help at all. If you hash client side,then the hash will became
your password. If you accidentally log this hash, anyone who have access to
that hash can log in as you easily by constructing logun request.

------
OrgNet
Somewhat unrelated but if you have two Facebook accounts with very similar
login names/email address, and different passwords, it doesn't matter which
login name that you use, it will login into the account based on the password
only. I contacted Facebook about it and they say that it is an intentional
"feature".

~~~
devilmoon
I can actually log into Facebook with a password I used on there roughly two
years ago, which to me seems like a VERY bad security issue. I'm hoping they
are at least fingerprinting my devices so that if someone else tries to access
my account with an old password it won't let them, but still...

~~~
julienreszka
What do you mean by "here"

------
MagicPropmaker
With 20,000 employees who potentially had access, there's no way of knowing if
any password was abused. They should message their users to change their
password, especially if they use it on other sites, just to be sure. And
because many sites use Facebook OAuth, there's a lot of room for abuse.

~~~
dexterdog
With 20,000 employees who potentially had access, we can rest assured that
passwords were abused.

~~~
MagicPropmaker
Exactly! It's nearly certain that at least one of them had an enemy, ex, etc,
that he/she wanted to exploit.

~~~
dexterdog
Or just a person saving the data for a rainy day

------
Mugwort
I hope Amazon, Apple and Google are doing better. How can we know for sure?
It's not unreasonable to ask because I would have thought it completely NUTS
to think FB used clear text but apparently they did (or still do). What about
everyone else? Does anyone know how to find out?

~~~
javagram
> Does anyone know how to find out?

Join the ops team at each of those companies, work your way to a position
where you can analyze log files, then start analyzing them and see if you can
find data that should have been obscured.

I don’t see any way to find out otherwise, we are talking about querying TBs
of internal, company specific private logs to see if someone made a mistake.
The best anyone could tell you is “I work for company X and I don’t think
we’ve had this problem.”

Edit: just using a password manager with unique password for every site will
solve most of the problems from a customer perspective. My Facebook password
is unique to Facebook and I have 2FA so even an engineer with my password from
a log couldn’t login to my account.

~~~
puzzle
At Google you don't get to read random log files, sometimes even from your own
project. There are entire pipelines for raw and sanitized logs, with
expiration dates and corresponding access controls. Unless you're on the
security team, crawling randomly through logs for no good reason is a quick
way to get in trouble.

~~~
lstodd
> At Google you don't get to read random log files

> crawling randomly through logs for no good reason is a quick way to get in
> trouble.

So do you get to or do you not?

~~~
puzzle
I meant crawling your own logs

------
tschellenbach
If it happens to a company with the best engineers, focus on security and
almost endless budget...

~~~
qpiox
No, this did not happen by mistake. It is plain and simple that they don't
have focus on security.

There is no true need for the cleartext password to leave the users browser.

~~~
dymk
So if it's not by mistake, do you think they're intentionally trying to reduce
security? Are you saying that every website that hashes on the server (almost
100% of the internet) isn't just negligent, but consciously sabotaging user
security?

------
lkrubner
Possibly this is the biggest deal of all:

> and searchable by thousands of Facebook employees

Any time we use a big Web service we understand that there must some employee
at that service who could see our activity, but we expect the number of
employees who could see our data must be strictly limited.

Thousands of Facebook employees could see user passwords? If true, that is an
amazing number.

I know that when I use Gmail, there must be some employee at Google who could
potentially see what I've written. But I've also read that Google has strict
controls about who could see my email, and the activity of their employee is
logged.

What Facebook allowed sounds serious, if only because they were so
lackadaisical about enforcing strict policies around data access.

------
nedwards72756
What I find more in more in life is that those that get ahead are doing so by
not doing things the right way. We have such a large population now that it
seems those at the top have got there because they have done something wrong
in order t get there.

------
misiti3780
This is really inexcusable, but so much of what facebook has done or does is
at this point I'm not surprised. I blocked all their IPs and do not have an
account.

If i had to guess, it was probably legacy code that causes this. Mark Z.
started this with raw PHP, if he had access to or had use an framework, they
would have provided salted passwords for free. (it is basically impossible to
do this in django, rails, etc). That does not make it Ok since they have 1000s
of engineers are qualified to fix it, but they probably have/had higher
priorities, clearly security is not one of them

------
bookmarkacc
Feels like this falls ln the infosec team. Our team has a scanner that parses
all* logs for passwords and PII.

We know that the infosec team at Facebook was a second class citizen. So, its
no surprise that this got through.

*sampling on large sets

~~~
bogomipz
What's PII here? I'm not familiar with this acronym.

~~~
creatornator
Personally Identifying Information

------
duchenne
Let's say that this problem occurred because they logged the body of post
requests (including the passwords).

Could this problem be mitigated by hashing the password on the front-end (and
a second time on the back-end)?

Like this, the server would never see the plain text password.

Also, if a facebook employee intercepts the hashed password, he could only use
it to login to a facebook account. But, I guess he could do that anyway if he
has access to the db.

However, this employee would have no way to know the real password and use it
on another website.

------
diogenescynic
Facebook seems to be the biggest cyber-threat and insecurity to the entire
United States. When does Facebook get shut down for being a national security
threat?

------
dontbenebby
I wonder if this is related to the fact Facebook passwords are (were?) not
case sensitive?[1]

(Ex: some script checks for hashes of the password, but not hashes of the
permutations that are also accepted?)

[1] [https://www.zdnet.com/article/facebook-passwords-are-not-
cas...](https://www.zdnet.com/article/facebook-passwords-are-not-case-
sensitive-update/)

~~~
dymk
Facebook passwords are case sensitive. The only case swapping that happens is
to correct leaving your caplocks key on, or if your keyboard auto-capitalizes
the first letter (like some mobile keyboards). This reduces the search space
by a factor of 3 (aka nothing), not by all permutations of capitalization.

~~~
dontbenebby
>Facebook passwords are case sensitive. The only case swapping that happens is
to correct leaving your caplocks key on

That's not the definition of "case sensitive" I've seen used by non-FB
engineers.

------
redsavagefiero
I've seen this so many times in self-rolled auditing kludges that it has
become kind of a running joke. I'm now surprised if anyone does anything more
sophisticated than sym encryption of master credentials used to guard secrets
without using the same secret for both as a first pass. Embedded master
passwords abound in most legacy OSS configurations.

------
DBYCZ
To facebook's credit: I created a brand new email address and unique password
to be used exclusively for my Facebook account, and 6 years later (with no
password changes), I have received no spam and had no discernable security
issues.

Maybe I was just fortunate, but everyone should use a burner email if they
choose to use Facebook.

------
Shivetya
Oh the fun that is scanning source repositories, pretty much a guarantee at
any large company you are going to find plain text passwords, as in hard
coded.

as for more fun in plain text passwords it is no worse than any system which
permits an admin to extract the user's current password and those systems
still exist

------
taneq
Move fast and break things, am I right?

------
tareqak
Facebook's newsroom post on the story:
[https://newsroom.fb.com/news/2019/03/keeping-passwords-
secur...](https://newsroom.fb.com/news/2019/03/keeping-passwords-secure/) .

------
turc1656
One of the many, many reasons to not ever use the "login with your Facebook
account" (or Google or whatever) on any site. And that's _before_ you even
take into consideration what sites like Facebook and Google are doing with
that information.

------
gfody
> an ongoing investigation has so far found no indication that employees have
> abused access to this data

what sort of indication would they be looking for? presumably it wouldn't be
hard for an employee to have made a copy for themselves without leaving a
trace of evidence.

~~~
ashelmire
I wonder this every time I see it in a report. It’s not like every file access
is recorded for 10 years. Or at all. If you’re lucky you know who accessed a
machine since the last time you rotated logs. But let’s say the data was
mounted and accessible to all internal machines; literally anyone could have
looked at it and done whatever they wanted without anyone knowing.

------
lern_too_spel
Is anybody surprised? This is the same company that served its login form over
unencrypted http for many years. Prior to using a password manager, I used my
password for "likely incompetent" companies for my Facebook account.

------
morningmoon
Another reason to use passwordless login. Just send an one-time login code via
email.

That’s what my companies do and there’s no way we can leak passwords... there
aren’t any!

I honestly don’t understand the point of using passwords when a website allows
reset by email.

------
coldtea
> _In this situation what we’ve found is these passwords were inadvertently
> logged but that there was no actual risk that’s come from this._

We, because of all those 2000 devs who had access to them, nobody could never
had stolen them...

------
0xmohit
Did someone say that it happened because Facebook believes in transparency?

------
Mugwort
Is it even possible for this to be the result of negligence when Facebook has
some of the best programming talent in the world? Maybe, but I don't think so.
I think this was INTENTIONAL. The question at least is why? IDK. Some FB
employees probably didn't mind because it gave them unbelievable spying
powers. Another thing to consider is how the NSA had a strong relationship
with FB for years and the odds of them not knowing about FB's clear text
practices is zero. They knew. People within law enforcement probably knew as
well. Given how nothing happens at FB without Zuck's apporoval there's no
doubt he knew as well.

------
packet_nerd
It's a shame we're still using passwords, even more so that they are sent _to
the server_. FIDO UAF or U2F is the way this should be done.

------
caseymarquis
I'm surprised more sites don't hash passwords with a per user salt and
hash/match with a per session salt before transmitting them.

------
known
Criminal negligence that can invite class action suit

------
jordache
What the Fck FB?

Why would you ever store plaintext password?

------
fcantournet
How the hell are plaintext passwords ever touching a FB server in 2019 ? oO

You should always hash the pwd client-side first... wtf ?

~~~
javagram
> You should always hash the pwd client-side first... wtf ?

This is very rare. What production systems do you know of that use SRP or
similar mechanisms? As far as I know the vast majority of companies and
applications use server side hashing.

------
uvu
How about private posts? Every time when you type something on Facebook, it's
all logging to logs?

------
residentfoam
This is just not acceptable. I deleted my Facebook account over a year ago and
I am doing just fine. I don't miss a bit of it.

At a company at the scale of Facebook mistakes like this are not acceptable.
It looks like engineers there don't have too much experience in the field, but
they are probably very good at flipping binary trees on a white board.

------
DGAP
Not a great look for Stamos.

------
laacz
I do wonder if there was a devops feature built on top of this.

------
C14L
Wasn't it up to 6% of worldwide annual revenue for each individual case,
according to the GDPR?

In reality though, I would be very surprised if this resulted in any fine at
all.

~~~
devilmoon
4% for each European user, theoretically. Realistically, nothing is gonna
happen and they will keep happily earning a fuckton of money from unaware
users.

------
lateralux
Orwellian Cyberpunk Idiocracy

------
xwvvvvwx
this is why it’s a good idea to delete your old logs

------
julienreszka
I'm shocked.

------
pansinghkoder
but can you code the DAG in 30 minutes?

------
jdlyga
My facebook feed has really become stale in the past few years. It used to be
a good way to see what your friends are up to, and see pictures. But now it's
ads, news stories, posts from my old university, bad memes, and that's about
it. Even instagram has become full of influencers, models, brands, etc. I just
want a platform to stay in touch with friends and that's it.

------
polote
Why is that an issue ? devs will also likely have access to the hash of the
password of a user, and then access his account.

We dont care if passwords are leaking, this is your issue to not use the same
password on all websites. Password are not really a private thing for
developers inside a company

~~~
EnFinlay
It's an issue for the same reason storing passwords in plaintext in the
database is an issue. Reading the hash != access to the account. Passwords
should be private to everyone except the account owner. If devs have access to
all user passwords then you have a fucking issue.

~~~
icedchai
On many systems, "real passwords" are just a tcpdump away (most sites aren't
doing end-to-end TLS. It's TLS terminated at the load balancer / proxy level,
everything else is in the clear.)

~~~
jeltz
That is a much smaller issues than passwords stored in logs, because the
people who have access to running tcpdump usually also have the ability to
intercept traffic in other ways (e.g. by updating the application to log all
traffic or by inspecting RAM). On the other hands access to logs is usually
handed out to a much bigger group of people.

