
Notifying administrators about unhashed password storage - ingve
https://cloud.google.com/blog/products/g-suite/notifying-administrators-about-unhashed-password-storage
======
Wowfunhappy
This happens to include a nice two-paragraph summary of how password hashing
works, that I think a non-technical person could easily understand. It's a
shame they used the term "string", but it reads fine in context. I should save
for future use...

\-----

> If you have a Google account, Google’s core sign-in system is designed not
> to know your password. How, then, can we verify your password when you sign
> in to your Google account again? The answer lies in a bit of cryptography:
> when you set your password, instead of remembering the exact characters of
> the password, we scramble it with a “hash function”, so it becomes something
> like “72i32hedgqw23328”, and that’s what we store with your username. Both
> are then also encrypted before being saved to disk. The next time you try to
> sign in, we again scramble your password the same way. If it matches the
> stored string then you must have typed the correct password, so your sign-in
> can proceed.

> The effectiveness of the hash function lies in its one-way nature: it is
> simple to scramble your password, but nearly impossible to unscramble it.
> So, if someone should obtain the scrambled password, they won’t be able to
> recover your real password. The downside of password hashing is that if you
> forget your password, we cannot show you what it was; there’s nothing we can
> do other than reset it to a temporary password (valid one time only) and
> then require you to pick a new one.

~~~
hammock
Doesn't really explain that the "scrambling" is deterministic which is an
important feature. In colloquial use "scramble" means to randomize, for
example "scramble the code to your gym locker."

When you scramble your locker combo, is really is random. When you scramble
your password with a hash function, it may look random, but it's really not.

Also just for fun, here's a creative stab at my own explanation.

As every dog knows, each dog has a unique smell. If you have smelled the dog
before, then you can use its smell to recognize who it is, but without that
past experience, nothing in the smell itself tells you who it might be.

The hash function creates the smell from your password, which is the dog. A
website does't save a photo of your dog, just the smell. So if a hacker breaks
in, they might smell your password but they can't see it. And when you login
to the website, it makes sure your password smells right before it lets you
in.

~~~
Wowfunhappy
> Also just for fun, here's a creative stab at my own explanation.

I know you said "just for fun", but I really think we need to do better than
these types of explanations.

I remember many years ago, I was wondering what the heck hyper-threading was,
and I came across a video that compared it to a bowl of cereal. Even though
you only have one mouth, you'll be able to eat the cereal more quickly if you
use both your hands, instead of just one.

I watched this video, and felt gratified, and closed the browser window... and
then suddenly realized that I _still_ had no idea what hyper-threading was,
only that I should use both my hands if I'm ever in a cereal-eating
competition.

A better explanation would have gone something like, "CPUs solve math problems
more quickly than we can give them problems to solve. Hyper-threading queues
up additional sets of problems that the CPU can switch to when it would
otherwise be doing nothing."

Analogies are useful tools, but usually only after we have some understanding
of the real version.

~~~
Fnoord
Analogies are great to _memorize_ the content. They don't explain the content
because they're not 1:1 translatable.

~~~
afiori
Does that mean that analogies themselves are an analogy of hash functions?

(not serious; hope it is not against the guidelines)

More seriously analogies are better for a bigger picture of a complex system,
as they effectively describe mode of interactions. The dog/smell analogy was
not bad but you have to also add a role for the other actor in the system
(here I would say: the third party, and the database)

------
tptacek
The ordinary response to notifications like these is, to put it bluntly and
accurately, a rage-mob. But here you have a corrective, a case where that
doesn't make sense. Because Google operates what is considered, by an
overwhelming majority of expert opinion, one of the 3 best security teams in
the industry, likely exceeding in so me ways the elite of some major world
governments. And they can't reliably promise, at least not in 2019, never to
accidentally durably log passwords. If they can't, who else can? What are we
to do with this new data point?

The issue here is meaningful, and it's useful to have a reminder that
accidentally retaining plaintext passwords is a hazard of building customer
identity features.

But I think it's at least equally useful to get the level set on what
engineering at scale can reasonably promise today.

 _correction: in the world - > in the industry_

~~~
xSHAme
For one of the world's best security teams, "rumor" has it that their password
hashing scheme is very far from best practices...

~~~
cheeze
What do you mean?

------
discreditable
My domain of 800+ users had one unhashed user. It was an account I created for
testing something then suspended over two years ago. It never completed the
first logon. I have many newer users that have not completed a first logon,
yet their passwords are (apparently) hashed.

They said users created through admin console are affected. The might explain
my nearly zero count. I've used GAM to automate creation for a very long time.
We are a high school school so we churn about 1/4 of our user count every
year.

~~~
whalesalad
I had the same thing happen, only one email. An account for my sister,
actually. Turns out she has never logged-in either.

~~~
chmars
Same here: One user!

------
judge2020
the email I got said it was primarily accounts with one-time use and generated
passwords were affected. I'm really confused as to why this isn't in the blog.

> due to legacy functionality that enabled customer Domain Admins to view
> passwords, some of your users’ passwords were stored in our encrypted
> systems in an unhashed format. This primarily impacted system generated or
> admin generated passwords intended for one-time use.

Sure enough, the accounts listed in the email as affected were never signed in
and still needed a password reset.

The email:
[https://i.judge.sh/Twilight/iRcF0e7D.png](https://i.judge.sh/Twilight/iRcF0e7D.png)

~~~
BIair
Same, the address in the notification email had never been signed in.

------
regnerba
They sure liked repeating "secure encrypted infrastructure".

I was a little confused about the timeline of this as they mentioned
implementing some functionality back in 2005. If I understand this right the
specific issue of storing some (unsure as to what percentage or quantity)
users passwords unhashed:

    
    
      - started in January of this year
      - is now fixed
      - were stored for up to 14 days in this state
      - would only effect new years signing up (based on it being part of the signup flow)
    

Do I understand that right?

~~~
owaislone
Looks like two issues were discovered.

1\. All passwords set by admins for other accounts since 2005 were impacted.
2\. Also all passwords starting Jan 2019 were also stored un-hashed for 14
days.

~~~
jkaplowitz
I have no inside info on this incident and haven't worked for Google in over 4
years, but the description of the 14-day 2019-only issue sounds like it would
have affected only a subset of users whose data they included in a sample for
troubleshooting purposes. The sample rate they use is probably far below 100%.

(Once again: this is speculation based on general firsthand knowledge about
how Google operates, but I know nothing specific to this incident.)

~~~
owaislone
Yes, of course. Not all but a different subset of passwords.

------
bredren
Just had a thread yesterday where people raged at the very idea companies and
organizations still use poor hashing algos.

Here we have google storing in plaintext by accident.

~~~
Godel_unicode
It can simultaneously be true that nobody should do this and that even the
best people can make mistakes.

~~~
asdfasgasdgasdg
I am reminded of the discourse on hypocrisy in The Diamond Age. That we fail
does not mean we are insincere in our aspirations to do good.

------
miohtama
I doubt Google would store passwords unhashed in a database - after all you
check against the hashed record. This sounds like good old "accidentally
logged sensitive data" issue.

~~~
eropple
Their control panel allowed domain admins to view these passwords. That
suggests that they're telling the truth.

~~~
j16sdiz
Those user are "password resetted by admin"

------
tomglynch
I've always been intrigued as to why passwords are not hashed and salted
client side before sending (and then hashed again on the server side). The
usual argument against it is that SSL is better - but I don't see why it
shouldn't be standard to do both. It would prevent issues like this (Malicious
admins or compromised server) from ever occurring.

Assuming the salt is unique to the website it also solves the problem of users
using the same password on many different websites - this would no longer be
an issue.

It would also mean as an end user we can have a look at our login POST request
and confirm our password is not sent and therefore cannot be stored in
plaintext.

EDIT: I see the other discussion and understand the hash now becomes the
password so if a hacker obtains the hash they now can login. I'm advocating
for it to be hashed server side also, protecting the user from cases just like
this one.

~~~
olalonde
This is essentially the same as using a password manager since your browser
(and all your devices) would have to know about your username/salt pairs.

~~~
tomglynch
Yeah very good comparison. And password managers are by all means the best way
to go to ensure privacy. So I don't see why people argue against client side
hashing.

~~~
GoblinSlayer
I think they only say it's not perfect.

------
U3OSB
Strange... Didn't realize G Suite and Cloud Platform were well coupled, but it
appears they must be.

I recently opened two of what I though were new 'consumer' Google accounts to
be dedicated to managing Google Cloud Platform resources. On both accounts I'v
also setup Identity & Access Management with an organization and additional
sub-user-accounts.

Both of those parent account received this email warning about five hours
ago...

Subject: [ACTION REQUIRED] Important Information about your G Suite Account

Google Customer Alert Dear G Suite Administrator, We are writing to inform you
that between January 13, 2019 and May 9, 2019, an internal system that logged
account signup information for diagnostic purposes, inadvertently stored one
of your user account passwords in our encrypted systems in an unhashed format.
This impacted the user account password provided during the initial account
signup process. The log information was retained for 14 days following the
signup process, and then was deleted according to our normal retention
policies. We have reviewed the login information for the account and have
found no evidence that the unhashed password was misused. The following is the
user account impacted in your domain(s): [ __ _@_ __] Google Planned Action:
for your security, starting tomorrow Wednesday, May 22, 2019 PT we will force
a password change unless it has already been changed prior to that time. Our
password update methodology is as follows: • We will terminate the impacted
user’s session and prompt the user to change their password at their next
login. • In addition, starting Wednesday, May 29, 2019 PT we will reset the
password for the user if they have not yet selected a new password or have not
had a password reset. This user will need to follow your organization’s
password recovery process. However, Super Admins will not be impacted. For
information on password recovery options please refer to the following [Help
Center Article]. For further questions please contact [Google Support] and
reference issue number ###. Sincerely, The G Suite Team

------
segmondy
In addition, as we were troubleshooting new G Suite customer sign-up flows, we
discovered that starting in January 2019 we had inadvertently stored a subset
of unhashed passwords in our secure encrypted infrastructure. These passwords
were stored for a maximum of 14 days.

Is this another way of saying they stored cleartext passwords? Interesting
they choose to use "uncashed" instead of "cleartext"

~~~
maxton
They are probably not saying "cleartext" because the data was still encrypted,
just not hashed.

------
rodmena
Thank you Google (AKA f __k you Facebook).

------
solotronics
would this violate GDPR?

~~~
superfrank
I'm not a GDPR expert, but I've done a fair bit of work making old systems
GDPR compliant. With my understanding of GDPR, yes, this is a violation since
part of GDPR requires you keep customer data secure using "appropriate
technical and organizational measures". I would argue that this violates the
"appropriate technical measure" part.

Again, I'm not a lawyer or expert in this domain, so I could be wrong.

~~~
kitotik
They seem to be hammering home the point that the passwords were always stored
on ‘secure encrypted infrastructure’ but the data was not hashed, so anyone
with access to googles ‘secure encrypted infrastructure’ could read the data.

The encryption is most likely enough to be within GDPR compliance.

~~~
pbhjpbhj
>The encryption is most likely enough to be within GDPR compliance. //

Why do you think that, allowing staff to read plaintext passwords is contrary
to standard security practice; companies are expected to make reasonable
effort to secure PII and allowing staff to read your password doesn't appear
to be "reasonable effort" by even the casualist of readings.

I don't think the EU courts are that stupid.

FWIW I don't think there is a case here particularly, as it appears to be a
genuine error and being fixed.

------
awinter-py
I mean hedgqw is my mother's maiden name and 23328 was my first pet, this
isn't a very strong hash

------
topmonk
It's a shame that password hashing isn't standardized.

There should be a standard password box in html in which you can just call a
function to pull out a salted hash of a password. The attributes of the box
could include options for password restrictions such as a minimum of 8 chars,
etc.

The fact that developers still need to write custom hashing algorithms in
javascript is kind of ridiculous .

edit: nevermind, I just realized this wouldn't work

~~~
comboy
There is no client-side security. If you hash on the client side then that
hash becomes the password and you have plain text password storage.

~~~
odensc
You could just hash the hash that the client sends, no? Then you never know
their password and it's still stored securely.

~~~
munk-a
If the client always hashes the password in the same way (no nonce or usage of
encryption) then the password essentially becomes the hashed string with some
extra clientside code you're burdened with forever. And since that same
hashing will need to be done whenever they log in the hashing algorithm needs
to be in the client side code sent to everyone... meaning an attacker would
have a bit of obscurity in their way and just need to regenerate a rainbow
table based off of your hashing algorithm to test common/weak passwords.

~~~
odensc
It's not about the security, it's about preventing side-channel leaks of the
plain password (e.g. incorrectly sanitized logs or buffer overflows). Having a
data breach of your site-specific hash is much better than leaking the user's
password that an attacker could use on any website.

~~~
munk-a
Your site specific hash, if leaked into logs and generated using a fixed
hashing algorithm, would just be a step of obscurity, a rainbow table
generated according to that hashing algorithm would be able to extract the
original typed password pretty reliably for use in compromising other accounts
held by the individual on other sites.

There are ways to actually add security above SSL but they require a per-
session nonce. The most secure versions of these operate on some sort of out-
of-band time-scrambling algorithm that's issued to users on an individual
basis (usually a dongle) which is basically a nonce that's been pre-arranged a
few years in advance, these can (if done correctly) provide all the security
of on-demand generated nonces and physical keys combined.

Fixed hashing algorithms add such a negligible amount of security that I'd
prefer to avoid them due to the tech debt their existence incurs outweighing
their advantage.

~~~
odensc
Their password would have to already be in a rainbow table, and at that point
nothing can help them. But I agree it's kind of pointless, I was just arguing
that there is a benefit even if negligible. If you want real security,
WebAuthn is the future.

