
Google Infrastructure Security Design Overview - emilburzo
https://cloud.google.com/security/security-design/
======
contingencies
Many of these solutions are unavailable below a certain scale, and there is
currently little commercial utility or pressure in offering these features in
a wholly-owned-and-operated fashion to small businesses or individuals. The
new deal (eg. DDOS resistance) is to rent an implementation, or go without.
Basically, the gap between everyone else and the Googles of the world is large
and growing.

On the other hand, I wonder how useful some of them are. Boot-level security
sounds fantastic but the cost of engineering and at the rate they probably
cycle hardware, with decent service-level signatures this probably largely
wasted money (eg. unexpected behavior like comms from service X to service Y
is default-denied at multiple levels, logged, triggers hard shutdown/reset of
system). While performance is cited as a concern, you'd save a lot of money
removing the design/deployment/maintenance of all that complexity and could
afford a little extra (more standard) hardware.

~~~
puzzle
Without boot level security, it's easy for the NSAs of the world to slip in a
hard drive or two with extra "surprise" software on it, later engaging in
active/passive surveillance or credential theft. Always assume that any one
single employee could be compromised.

~~~
notriddle
The NSA's of the world don't need to hack Google's infrastructure. They can
just ask.

This is protection from rogue employees acting independently, assuming it's
not just marketing and ego-stroking for the engineers.

~~~
toomuchtodo
Why don't you google "NSA google smiley face".

~~~
leesalminen
I don't get it....search results returned only your comment.

~~~
toomuchtodo
First result for that search:
[http://www.slate.com/blogs/future_tense/2013/10/30/nsa_smile...](http://www.slate.com/blogs/future_tense/2013/10/30/nsa_smiley_face_muscular_spying_on_google_yahoo_speaks_volumes_about_agency.html)

OPs comment:

> The NSA's of the world don't need to hack Google's infrastructure. They can
> just ask.

NSA doesn't just ask; they found ways to MITM Google.

------
zbjornson
> We have started rolling out automatic encryption for the WAN traversal hop
> of customer VM to VM traffic. ... all control plane WAN traffic within the
> infrastructure is already encrypted. In the future we plan to ... also
> encrypt inter-VM LAN traffic within the data center.

It would be nice if this was more explicit. For example, is traffic that is
TLS-terminated at their LB reencrypted all the way to the back end VM? At what
point is it decrypted again? Are those keys unique to us or are they used for
whatever traffic happens to traverse the same network paths? (I assume shared
but with software-defined networking maybe it's practical for them to be
unique.) What does the "control plane" encompass?

In any case, I'm curious what people think about trusting the service provider
for inter-service and inter-VM encryption. Do you use the LB's TLS
termination? Do you still enable encryption for your DB connections even if it
is (or will soon be) redundant with their network encryption?

~~~
mentat
Anyone with access to the hypervisor at the service provider will have access
to plaintext. TLS protects you from service provider network compromise within
whatever scopes that covers. If you're in the cloud, you do have to have some
basic trust in your service provider as compute is always in plaintext
(barring homomorphic encryption).

~~~
xyzzyz
_Anyone with access to the hypervisor at the service provider will have access
to plaintext._

This is mostly true with today's state of the industry, but with upcoming
technologies like Intel SGX[1], the hypervisor will not be able to access the
plaintext anymore.

[1] - [https://software.intel.com/en-
us/blogs/2013/09/26/protecting...](https://software.intel.com/en-
us/blogs/2013/09/26/protecting-application-secrets-with-intel-sgx)

------
AJRF
In the CIO summary they mention every service uses KeyCZAR.

First line on KeyCZAR repo:

"Important note: KeyCzar has some known security issues which may influence
your decision to use it."

[https://github.com/google/keyczar#known-security-
issues](https://github.com/google/keyczar#known-security-issues)

~~~
nealmueller
I work at Google. The final bullet in the CIO Summary on Keyczar was a
typographical error, taken from our paper on encryption at rest
([https://goo.gl/hSordh](https://goo.gl/hSordh)). It's since been removed from
this Security Design Overview. The encryption at rest paper goes into
additional detail and includes the important clarification that while a very
old version of Keyczar was open-sourced, the open-sourced version has not been
updated to reflect internal developments.

~~~
AJRF
Thanks for the reply and follow on information. Wondering why those internal
changes didnt get rolled into the public release, especially if they were
security focused updates? Lack of adoption of the library maybe?

------
dwheeler
This is great to see. For those who don't know, this is an "assurance case"
(definition: "a body of evidence organized into an argument demonstrating that
some claim about a system holds, i.e., is assured") - [https://www.us-
cert.gov/bsi/articles/knowledge/assurance-cas...](https://www.us-
cert.gov/bsi/articles/knowledge/assurance-cases/arguing-security-creating-
security-assurance-cases)

I'm glad to see more assurance cases. You can't just do one thing and have a
secure system. And if you want people to trust you, you need to give them a
reason to trust.

The CII best practices badge (
[https://bestpractices.coreinfrastructure.org](https://bestpractices.coreinfrastructure.org)
) also has an assurance case; details at
[https://github.com/linuxfoundation/cii-best-practices-
badge/...](https://github.com/linuxfoundation/cii-best-practices-
badge/blob/master/doc/security.md) . If you want to help us make that better,
let us know!

------
petters
> ... and laser-based intrusion detection systems

Huh? I thought that was exclusive to movies like Entrapment and Mission
Impossible.

~~~
scrollaway
It's a fancy term for motion detectors. If my neighbour can afford one for his
yard, it's not that crazy to put some in a datacenter :)

Edit: I obviously wasn't implying they're using the same ones. Come on, now
>.>

~~~
foxylad
Motion detectors are usually passive infra-red (PIR) sensors - no lasers
involved. Unless you can cite a consumer-grade laser-based motion detector, I
think this means Google's data centers are protected by slightly higher level
gear.

~~~
dalore
[https://www.amazon.com/Homesafe-Safety-Motion-Detector-
Senso...](https://www.amazon.com/Homesafe-Safety-Motion-Detector-
Sensor/dp/B00YWDDCOW)

Homesafe Safety Beam Laser Motion Detector Sensor & Alert

Only $39.99!

~~~
runeks
> This high tech device creates an invisible infrared beam up to 60 feet long
> and sounds a loud alarm, pleasant chime, or mutiple chimes when the beam is
> crossed.

------
mnm1
I wonder what their data deletion policies really are for something like
Photos. I deleted all my old photos weeks ago but when I pull down the archive
of my Google data, they're still there. With such a policy, I could see that
data sitting around for years while Google claims that it's in the process of
deletion, something that is not actual deletion. Then again, I doubt they
actually ever delete anything.

~~~
londons_explore
In which case, you likley didn't delete them correctly.

If you delete them from your device, it doesn't delete the cloud copy.

If you delete from an album, it removes the image from the album, but not from
your account.

Google's privacy policy says has limits to delete user data, and I can assure
you they are very strict about that. (Lots of data is deleted within hours,
but the multiple days is to ensure all backups of it are gone too)

See [http://blog.tech-and-law.com/2010/11/google-data-
retention-p...](http://blog.tech-and-law.com/2010/11/google-data-retention-
periods-for.html?m=1)

~~~
petters
This is correct. A deletion should be effective/visible immediately, but it
can take some time before all backups are guaranteed to be gone.

~~~
drieddust
No it isn't google takes your data to tapes as well as offline long term
storage.

[1][https://www.youtube.com/watch?v=eNliOm9NtCM](https://www.youtube.com/watch?v=eNliOm9NtCM)

~~~
mkj
They could be deleting encryption keys to the tapes? All speculation.

~~~
puzzle
There was a talk about the backup infrastructure. The speaker talked about the
issue of keys, but didn't provide specific details:

[http://highscalability.com/blog/2014/2/3/how-google-backs-
up...](http://highscalability.com/blog/2014/2/3/how-google-backs-up-the-
internet-along-with-exabytes-of-othe.html)

------
fowl2
so they've reinvented kerberos, presumably in a way that works. interesting.

(and there are many other things)

~~~
puzzle
Yeah, it's called LOAS:

[https://mobile.twitter.com/jbeda/status/715373975182807040](https://mobile.twitter.com/jbeda/status/715373975182807040)

------
bogomipz
I have a question about Step 5 in the post, it states:

Is "Step 5: Add '1' to the end"

Is this a delimiter for beginning of the padding or does it server some other
purpose?

~~~
timdierks
Did you mean for this to be somewhere else?

~~~
bogomipz
Yeah I did, thanks. That's what I get for multitasking. Unfortunately its too
late to delete :(

