
Lessons learned while protecting Gmail - hektik
http://googleresearch.blogspot.com/2016/03/lessons-learned-while-protecting-gmail.html
======
JacobJans
I am a victim of these "lessons."

I run several email newsletters. One of them consistently gets marked as a
"phishing" attempt by Gmail. They do not get marked as phishing if the email
includes no links – but if I include a link, it immediately gets marked as
phishing. This disables all of the links in the emails. I regularly get emails
from subscribers complaining that the links do not work.

I once sent an email to all of the Gmail subscribers, asking them to
explicitly mark my emails as trusted. Many of them did. But, it did not help –
they still get marked as phishing attempts. I've implemented SPF and DKIM. It
still didn't help.

The problem: Once an email is flagged as "phishing", the links are not
clickable. On mobile, there is no way to make them clickable. Most users don't
know how to make them clickable, even when it is possible.

All of the emails come from the same domain and the links are always to the
same domain. All of them have the same basic format. Other email from this IP
address gets through no problem. Google's Postmaster tools give the IP address
a 100% reputation. I tried switching to a new sending service, to no avail.
Nothing seems to work – no matter what I do, Gmail marks my emails as
phishing.

I've contacted Google's postmaster, to no avail.

Basically, Gmail has made it very difficult for people to read the content I
send them. There seems to be no way to convince Gmail that my emails are not
phishing.

~~~
solotronics
there are plenty of examples and at this point it seems that Google has a
vested interest in making life difficult for independent or self hosted email
providers

~~~
techsupporter
This has been my experience. I run a very small email setup for a few friends
and a couple of tiny businesses. (Yes, we should just pack it in and move to
Fastmail but I am hard-headed and have been around the Internet long enough to
still be idealistic about these things.)

Every couple of months, Gmail starts depositing any email from my users into
spam. No warning, no rejection, just a vague yellow banner on the recipient's
view that says "this message has the characteristics of spam." The message
could be in plain text with no links, still the same. I have SPF, DKIM, and
rDNS all configured. I send from a single IPv4 and single IPv6 address with
matching records. No RBL entries. Even the headers on a "spam" message say
that everything passes and is fine.

Oh, and it isn't a domain reputation problem: only one of the domains I host
even has a web site and all of them are at least five years old. Two of my
domains predate Google itself. :P

The real kicker? I can't use their postmaster site, either. Why? We don't
generate enough email to rank a report!

Meanwhile, no problems at all from any other receiving hosts. I have to log
into several Gmail accounts and click "this is not spam" on some test messages
and then it is fine for another 50 days. After that, back to the bit bucket.

Grr.

------
Meekro
It sounds like the lion's share of their effort is going towards blocking
attacks that prey on users' lack of knowledge. Infected email attachments and
phishing are both examples of this that they brought up in the talk[1].

It worries me, though, that they're willing to accommodate novices to such a
degree that they open up advanced users to targeted attacks. For example,
Gmail bugs you quite a bit to set a "recovery phone number," but doesn't make
it clear that this isn't like 2-factor auth. The phone number is actually a
_single factor_ that can be used to reset your password. It even works if you
have "traditional" 2-factor enabled.

Thus, the attack looks like this:

1\. Look up target's social security number. This is easy with certain online
services that were meant for private investigators, but actually let anyone
get an account.

2\. Contact their cell phone provider. If you don't know which one, guess.
There are only a handful of common providers and you'll hit on it eventually.
Impersonate the target, say you're going on vacation and need your calls and
texts forwarded, and give them the SSN from step 1 to verify.

3\. Go to Gmail and say you forgot your password. Opt for the phone based
reset, and wait for the text with a reset token to come in. Ideally, do this
while the target is asleep to give yourself time to work.

High profile individuals have actually been hit this way, and I think Gmail
should offer greater protection to sophisticated users who do everything
right, don't fall for phishing, and would never forget their password, but can
fall victim to highly targeted attacks.

[1]
[https://www.youtube.com/watch?v=nkV9kOsTyJU](https://www.youtube.com/watch?v=nkV9kOsTyJU)

~~~
wstrange
Would a cell phone provider forward to any number on a verbal request over the
phone? That seems pretty sketchy to me.

~~~
bigiain
Check this out:

[https://youtu.be/bjYhmX_OUQQ?t=2m30s](https://youtu.be/bjYhmX_OUQQ?t=2m30s)

A phone provider's key goal is to continue to provide charged-for services to
their customers, not to secure your bank account or dns registrar account...
They'll do whatever's needed to allow paying customers to pay or pay more.

~~~
sametmax
It makes sense : their business it not security. Companies use them for that,
and never asked for permission, they don't have to honor it.

Alhough as a customer, I'd like my phones not to be redirect to anybody asking
for it, but I understand they don't have the security measures a bank should
have.

------
elbigbad
Some of these are pretty good, but mostly just intuitive stuff. Really wish it
was to more than just an infographic style png giving broad strokes.

Things like "You are only strong as your weakest link," "There is no silver
bullet," and "Never stop improving" are essentially meaningless platitudes. I
would, however, love to see data on headings like "Attacks come in bursts."

Is there a link to something that delves more into each topic?

edit: Saw the links to the slides and video talk. Looks much more
comprehensive!

~~~
Shorel
"Make it hard for attackers to understand your defenses"

------
amelius
> "Users are your best allies. Empower users to action through meaningful
> feedback."

Interesting how that works only one way. If you are in need for some support
from Google, then good luck trying to contact them.

~~~
vacri
If you're on a paid account, then you get to contact them. It's not the best
support ever, but you do get support.

------
jlg23
actual talk:
[https://www.youtube.com/watch?v=nkV9kOsTyJU](https://www.youtube.com/watch?v=nkV9kOsTyJU)

slides: [http://www.slideshare.net/elie-bursztein/lessons-learned-
whi...](http://www.slideshare.net/elie-bursztein/lessons-learned-while-
protecting-gmail)

------
nickpsecurity
I'd like to learn more about their DDOS, network-level, and cross-datacenter
protections. They have to deal with so much volume and so many integrations
that there's probably some lessons they learned there on dealing with issues.
I'd also be interested in how their filesystem or data-processing tech with
built-in integrity and availability mechanisms factor into it. They might
leverage it in interesting ways like they do with F1 RDBMS.

------
seeing
GMail cost me over $3K in rent.

I'm looking for a job, and after sending out resumes (and often a URL to my
resume) for over a month I realized practically no one was getting my emails.
I was being flagged as a spammer.

The only way around this was to signup for a GMail account.

I won't feel very protected losing the roof over my head, GMail. Please fix
this.

~~~
rimantas
Why not $100k instead of $3k, for the job you did not get because of GMail?

------
sushisource
Sweet infographic for ants - clearly it was done by a design pro, why upload
an image with such tiny fonts?

~~~
simoncion
Looks fine to me at ~3 feet on my 96 DPI screens and at ~2 feet on my ~200 DPI
Nexus S screen. If I put on my _dramatically_ too-old glasses, it still looks
good from substantially further away.

 _shrug_

------
devy
This blog post has tons of good stuff. For one, George Hotz's intro to his
timeless debugger QIRA and his 4 mins pitch to his autonomous driving company
comma.ai was pretty awesome! Well worth that 20 minute than NSA TAO Chief's PR
talk for half an hour.

~~~
nickpsecurity
Probably better. I wouldn't knock the TAO chief's talk as he basically gave a
lot of good advice. He could've bullshited about just stacking a few security
products like I've seen with RSA conference types. Instead, it was a thorough
look at many ways they compromise systems and _most_ of what needs to be
considered. It was one of few positives I give to NSA as anyone listening
might improve their stuff.

------
vacri
One of the comments points out that the URL shortener link at the bottom of
the infographic is broken - the actual link is mixed-case, but the infographic
is in trendy all caps. A lesson here for both devs and designers - don't make
case-dependent urls, and don't force all caps on content that is case-
sensitive...

~~~
simoncion
> ...don't make case-dependent urls...

Ehh. If you're running a URL shortener, case-sensitive URLs let your URLs stay
short longer. Ditto for things like YouTube's video IDs.

~~~
hueving
Don't make case-dependent URLs that humans have to type by hand.

~~~
simoncion
Soft disagree. There's generally nothing wrong with case-sensitive URLs when
they're rendered in a font that makes commonly confused characters easily
distinguishable. The only problem I can see is when you have to _speak_ such
URLs.

~~~
Ensorceled
eh cap bee cap cee dee eee eff gee is aBCdefg

~~~
simoncion
Sure. It's not an _insurmountable_ problem. ;)

------
AdmiralAsshat
The icon for "Encrypt Everything" says "In transit & at rest" with a picture
of the Gmail icon.

Does this mean they encrypt customer e-mails at rest on their server?

~~~
lazaroclapp
I would think it common-sense to do that. If nothing else, it makes hard-drive
disposal much simpler, and accessing disk storage is slow enough that I can't
imagine the overhead of (hardware-assisted) encryption is a problem. Most
places I worked at require all desktops and laptops to use storage encryption
too. Encryption at rest is very different from end-to-end encryption, though.

~~~
bostik
The overhead of at-rest encryption is negligible.

If you think about it, the files Gmail backends need to access are fairly
small: even the upper limit is just couple of tens of megabytes. This is true
regardless of whether you get to use hardware acceleration or not. (And in
case of modern Xeon servers, you certainly do!)

Servers will spend more time waiting for disk seeks to complete than they do
decrypting the data once it's read.

You can even test it yourself: just run "openssl speed aes". My puny laptop
does 85MB/s at the most unoptimal settings (AES-256 with 16B blocks), and
92MB/s with conservative settings at same security level (AES-256 with 1kB
blocks).

A decent server system can do multiples of that. And once you add hardware
acceleration, we're talking about crypto throughput of several hundreds of
MB/s. Google servers are connected to top-of-rack switches, and I can make an
educated guess that the per-server bandwidth is 1Gb, or roughly ~120MB/s.

For hilarious comparison, even my RPi 2 can do 16MB/s.

So: if we're talking about on-disk storage, crypto will never be your
performance bottleneck.

~~~
wolf550e
protip: run `openssl speed -evp AES256` and be amazed by the performance of
AESNI!

------
ikeboy
>only as strong as weakest link

>diverse team

Adding diversity makes the weakest link weaker (and the strongest link
stronger). The point of diversity is to increase variance in multiple areas so
that a team's "strongest" member in any area is strong. Does not make sense as
a solution to weak links.

~~~
AnimalMuppet
I think the "only as strong as weakest link" was referring to the
_technological_ measures, not to the _team_. It's not the team that's under
attack (normally - if it's a phishing attack against the Gmail team as a way
of attacking Gmail, then yes, diversity on the team may be a weakness).

~~~
ikeboy
If each member is responsible for a different part of security, then you'd
still want each to be as strong as possible.

It just occurred to me that diverse could mean "skilled in different forms of
security", and then it's just saying to hire domain experts in as many
security domains as you can. That would make diversity a direct solution to
the weakest link problem.

------
lucb1e
Expected an article about new kinds of attacks or unexpected attacks, instead
got an infographic with a couple of common proverbs ('only as secure as the
weakest link!', 'layered defenses!' aka defense in depth, etc.)

------
tbabb
Has anyone found a link to a higher res version of that infographic? The
orange bubble text is infuriatingly unreadable.

------
ernsheong
Anyone have any idea on the Gmail stack? I think it's GWT for the front end?

