Hacker News new | past | comments | ask | show | jobs | submit login
How a Google Headhunter’s E-Mail Unraveled a Massive Net Security Hole (wired.com)
477 points by trendspotter on Oct 24, 2012 | hide | past | web | favorite | 101 comments

DKIM is an anti-spam mechanism. It does not authenticate the sender of an email message; to do that, use something like PGP. This is an interesting story, but it's not a story about a "massive net security hole". Mail on the Internet has always been spoofable.

Gmail (possibly Hotmail) put a little lock icon next to DKIM authenticated email from some senders, such as eBay & PayPal and outright reject unauthenticated emails from such domains. They've flaunted this feature in the past

So if an authenticated PayPal email pops up in your Gmail inbox saying you must do this and that to unlock your account, you may be more likely to do so due to the legitimacy of DKIM.

The reality is that people will act on "Paypal" mail that comes from "Payapal.ng". Let's not pretend that DKIM has much to do with that decision. I agree, though, that the little lock in the Gmail UI is misleading.

That UI affordance is actually incredibly frustrating for us, because we get that question a lot. Tons of people get confused by it.

Just register serverX-paypal.com (where x is a number) ftw. People in general are stupid. When asked what browser they use, the overwhelming majority respond by saying "Google". That says all that needs to be said about the general public.

Quite a sad view of humanity. I don't think people are stupid, I think they just don't care and shouldn't care about the browser. It's a tool used to get access to the information they need.

I am reading HN on chrome, but unless I go looking for what browser I use, I wouldn't know.

Decoupling the concept of "you don't intimately know what I have spent my entire life playing with" from "stupidity" seems to be really difficult for the tech crowd.

Always sad. People willing to discount countless hours of expertise and knowledge because a user doesn't know what the name of their browser is. As if that means anything.

For example, I really don't give a shit if my neurosurgeon is aware of what his browser is named. Nor would I dream of calling him stupid if he didn't. Chances are he knows leaps and bounds more about me on most topics, just not casual desktop computing.

Likewise, discounting someone entirely because they're uncomfortable with or uninterested in computers is one of the most ridiculous, ignorant, and self-absorbed things you can do.

This is about maturity and empathy, and it's certainly not exclusive to the tech crowd. [Insert half-baked pseudo-psychiatric idea about the link between tech people and empathy and Asperger's and so forth]

This is a useful consideration. It Appears to be an affliction of many (if not all "experts") of various stripes.

I'm immediately tempted to apply it to politics. I won't, or at least certainly not here, but it's interesting to think about.

Stupid is a harsh word. But it's short and simple and easy to understand.

"Assume the user is stupid" isn't really being mean to users. It's just shorthand for "make everything as easy as possible. Sane defaults; great design; remove ambiguity; correct documentation; and so on."

Why does not being intimately familiar with the structure of a URL make someone stupid exactly?

Same reason handing your wallet to a stranger who calls himself Dr. Bankersmith is stupid.

How in the hell is that even somewhat similar?

"People" includes major websites like Amazon and banks that send official email from phishy domain names.

I have mistakenly reported several Amazon security emails to their phishing team.

Does that Padlock really have anything to do with DKIM or SPF? I thought it was just some magically hard coded thing for eBay and PayPal messages. My own DKIM signed messages certainly don't get it.

DKIM/SPF is used. However I believe both gmail and hotmail use a whitelist for the showing the padlock.

For GMail, it's an optional feature that AFAIK is not on by default, and it is hardcoded for eBay/PayPal only.

Are you saying that 1) anti-spam is not about about security and that 2) DKIM, even if properly implemented, doesn't make spoofing harder and security better?

I'm saying that DKIM is an anti-spam mechanism, and that there are real ways to authenticate the actual sender of a message, like PGP or S/MIME. Calling a cracked DKIM key a "massive Internet security hole" is like calling a bug in SpamAssassin a "massive Internet security hole".

You didn't answer the questions. That's because you well know that anti-spam is security. Just because DKIM isn't foolproof, doesn't make it useless as a layer in anti-spam and security.

You asked a question that had nothing to do with my comment, so I've declined to engage with you on that point. I'm not interested in litigating whether DKIM is "useless" or not.

But I'm happy to continue backing up why this particular Wired headline is silly. DKIM is a cryptosystem backed by the insecure DNS. Mail has been spoofable since before RFC822. The whole idea behind phishing attacks is that you can't trust email. Nobody credible has ever suggested that DKIM resolves that problem; you will find no credible Internet security advice anywhere suggesting that a DKIM signature on a piece of email from Paypal or Chase means you should click on a link in that email and log into something.

Saying that DKIM doesn't adequately verify an email is genuine ignores the point of DKIM which is to weed out emails that clearly aren't genuine.

So Gmail can simply deadpool hundreds of fake paypal phishing emails. That doesn't mean the occasional one that gets through by fooling DKIM is authentic - but the security benefits exist.

Unfortunately, in quibbling over the headline, which you are free to do, you argued that anti-spam has nothing to do with security.

Simply put keeping most of the spam that purports to be from paypal.com out of an inbox is a security issue, even if a determined spammer can thwart dkim via dns shenanigans.

US CERT seems to agree: http://www.kb.cert.org/vuls/id/268267

What seems to be happening here is that you don't like the fact that I criticized the Wired headline, so you're moving the goalposts from "Massive Net Security Issue" to "Thing That Might Make It Harder For Google To Reject Spam". Then, when I point out that you're moving the goalposts, you respond by simply repeating the claim --- and then suggest that pointing out what DKIM actually is amounts to a "quibble".

DKIM is one of many, many anti-spam mechanisms in place at Google (and presumably Yahoo). But of course, far fewer people would read and forward this Wired story if you had simply written "Mathematician Finds Weakness In One Google Anti-Spam Measure During Recruitment Attempt".

I know you are the resident curmudgeonly voice of security truthfulness on Hacker News, but it's really odd that you keep referring to anti-phishing measures as anti-spam, as if they have no relation whatsoever to security.

DKIM plays a key role in keeping phishing e-mails out of the inboxes of hundreds of millions of people who have no idea what PGP or an e-mail header is. It's a standard, not just a Google feature.

And if you'd read the story, you'd see that a number of companies fixed their weak crypto thanks to his efforts.

But, of course, far fewer people would upvote your comments if you didn't diss everything with a tone of condescension.

Ryan, I wrote:

DKIM is an anti-spam mechanism. It does not authenticate the sender of an email message; to do that, use something like PGP. This is an interesting story, but it's not a story about a "massive net security hole". Mail on the Internet has always been spoofable.

What else do you want me to say? Mail on the Internet is spoofable, with or without DKIM. I literally don't know what I can do to placate you at this point.

Framing it as a mere anti-spam weakness rather than anti-phishing is kinda disingenuous.

Here I'm just going to refer you to "Three Myths About DKIM", linked from the front page of the DKIM site:


None of those myths talk about spoofing the sender domain.

DKIM doesn't validate the From line. But where is the weakness in trusting that someone with Google's private key approved the entire content of a message that is signed with Google's key, and so, my trust in Google should extend to the content of that email?

I already provided one specific example upthread: most domains people care about host applications with bugs that allow attackers to generate arbitrary mail from those domains. This is, please note, a very different problem than "being able to generate arbitrary mail under a specific signature". It is considerably easier to forge mail from BANKOFAMERICA.COM than it is to forge mail under the S/MIME signature of a specific Bank of America employee.

Domains simply are not a meaningful security boundary. At no point in the life of the commercial Internet have they ever been. Yes, there are security mechanisms on the Internet that are simultaneously (a) important and (b) misguided enough to ignore this fact. Fortunately, the most important of them are in practice difficult to reliably and scalably exploit.

This whole story is a tempest in a teacup. As Matthew Green said on Twitter: a 512 bit DKIM key says more about how little Google cares about DKIM than it does about any laxity on Google's part. Google is not lax about security.

Since this thread is also dead, we'll have to wait until someone else tries to get to the top of the front page with a DKIM story to continue arguing about DKIM.

It authenticates the domain of the sender, right?

If you want to impute that much authority to it, sure, but the actual signature verification depends on the insecure DNS anyways. Do not make security decisions based on DKIM. It's an anti-spam mechanism and that's all.

My personal opinion is that all email should be considered from a random person.

I tend to agree, and further suggest that DKIM shouldn't really change that calculation.

Curious - does DNSSEC add much to the amount of trust one can assign to DKIM verification? Or are there better solutions (both for trusted DNS and DKIM-alike)?

The limitations of DKIM are bigger than the fact that it relies on DNS. The bigger challenge is that domain names are not a particularly valuable trust boundary. Applications on the Internet are full of bugs that allow attackers to source messages from other domains.

DNSSEC is a boondoggle, and so far as I know no popular application on the entire Internet relies on it for security. But we don't need to hash out DNSSEC vs. DNS to see why a cracked DKIM key isn't a major Internet security hole.

How does that signature verification work? Something like a callback to paypal.com to get the verification? (Analogous to downloading an SSL cert from a website without using a trusted side channel like a handwritten note?)

Seriously old news... I attacked Facebook's 512 bit DKIM key back in 2010: http://blog.jgc.org/2010/06/facebooks-dkim-rsa-key-should-be...

I was just writing that!

It was on HN too, of course: http://news.ycombinator.com/item?id=1442385

I was surprised then that anyone would be using a 512-bit RSA key in the wild, let alone now.

I'm curious to hear what you think about DKIM in general.

At one time DKIM and SPF were good indicators that an email was spam because spammers adopted it so that their mail looked more legit.

I never look at DKIM or SPF. If I really care about who a message comes from I use PGP. It's a handy input for learning spam filters that use it as one piece of information, but it has about a 10% failure rate (legit mail where the signature fails to verify because of message manipulation in transit). It's most useful if the domain it's matching against is also trusted in some way. For example, a good signature against google.com is likely to mean the mail is good; a good signature against frohfuwehfwo.biz is not very helpful unless we know that that domain always sends spam.

Also, the most important piece of mail I ever received (from the Prime Minister's office) came without SPF or DKIM and I authenticated it by calling the office. In general, external authentication like that tends to reassure me.

So this is one key, used for all a domain's email for a long time, right? How lazy must you be to use an under specced key?

Plenty of crappy DNS web admin "tools" limit the maximum length of a TXT record. Sometimes it's harder than just typing "1024" instead of "512"...

> Harris thought there was no way Google would be so careless, so he concluded it must be a sly recruiting test to see if job applicants would spot the vulnerability. Perhaps the recruiter was in on the game; or perhaps it was set up by Google’s tech team behind the scenes, with recruiters as unwitting accomplices.

Ha! That's optimistic.

Or paranoid. Or naive. Or narcissistic.

It might be somewhat feasible if they wanted him to be security engineer, not a devop. Still, he expected they have set up what essentially is an elaborate prank just to send a cold-call email to just one of probably numerous potential candidates.

How likely this is? What would be the risk-to-reward ratio for doing that, considering that many of unsolicited recruiting mails are not even read? Isn't it more feasible for it to be a genuine mistake on their part? Google's not infallible, omnipotent being after all.

Back when I believed the hype, I made the same mistake with materials from Google's recruiters. They gave me driving instructions from SJC which left me in the wrong part of the valley on a Friday night during rush hour (this was before smartphone navigation). I figured it was some kind of test. It wasn't.

I called it my "cleverness attribution error" and wrote about it this summer: http://rachelbythebay.com/w/2012/06/19/attrib/

I've run into it in a few other places, too.

The chess champion Capablanca said that he was protected from losing games due to minor blunders because his opponents assumed he was so brilliant that he saw something they didn't, so they played safe and avoided taking advantage.

I think I read an article recently (probably highlighted on HN) that talked about how Deep Blue did exactly this versus Kasparov. A bug caused it to make a sub-optimal move, and it's quite likely Garry misinterpreted it as genius and psyched himself out going forward.

Yeah I'd read that. Kasparov was pretty freaked out by Deep Blue. He wrote this:

"I got my first glimpse of artificial intelligence on Feb. 10, 1996, at 4:45 p.m. EST, when in the first game of my match with Deep Blue, the computer nudged a pawn forward to a square where it could easily be captured. It was a wonderful and extremely human move. If I had been playing White, I might have offered this pawn sacrifice. It fractured Black's pawn structure and opened up the board. Although there did not appear to be a forced line of play that would allow recovery of the pawn, my instincts told me that with so many "loose" Black pawns and a somewhat exposed Black king, White could probably recover the material, with a better overall position to boot. "

about a move most computers of the time would find pretty quickly, and most decent human players would intuitively have thought OK at first glance.


My theory is that a significant part of his game was based around human psychology, so he found it hard to grasp computers. He played computer-friendly risky openings as if to taunt the machine, and heavily talked up the influence of the programmers on Deep Blue to an almost paranoid extent.

The result was that he lost against Deep Blue when he should have won fairly easily if he'd been more disciplined.

"a _few_ other places"

I think you are being very diplomatic. :)

But seriously, this cleverness attribution error might be a big problem with large players like Google. Forum posters, blogs and the digital versions of mainstream media all seem convinced Google is somehow special. That they know what they are doing, at every turn. Assuming some silly mistakes is a "test". It's a potentially harmful meme: people assuming ideas like "genius" or "utmost comptetence" without requiring any proof.

Blind faith followers take note, because here we have _proof_ that Google makes mistakes too. Silly ones at that.

They could send a large number of recruitment e-mails crafted like that at negligible cost, not specifically for that one guy, and see who catches it. A little far-fetched but within the realm of possibility with a company known for geeky stuff.

Google has done this kind of stunt in the past. Remember {first 10-digit prime found in consecutive digits of e}.com?

There was also this very elaborate easter egg in a promotional video of the first chromebook that someone cracked which got him a chromebook, I can't find it right now though

Sure, but that's not exactly subtle.

>But the government of Iran probably could, or a large group with sufficient computing resources could pull it off.

Yes, I can see it now: Iran endures crushing sanctions in order to pursue spam email program.

The rule of thumb that I have heard, and seen played out, is that there are basically 3 tiers of who can factor what. governments or government sized companies (upper millions, or billions sized budgets), modestly sized companies / research projects (millions), or hobbyists. Each can handle about 256 bits more than the next, and the time for one to reach the capabilities of the other is somewhere between 5 to 10 years.

Hobbyists have been factoring 512bit keys on a whim for a few years now, so....

SPAM is not the only attack that can be carried out with this kind of exploit.

I find it cute how clueless mathematicians and physicists are about how clueless we (programmers) are. Weak crypto? Assume it is a puzzle!

I hope this guy's inbox is full of job offers. That's a heck of a find.


Wow, the guy's a monster. Fluent in classical (and Levantine) Arabic, Chinese, Greek; Top Putnam score (twice), teacher, Christian missionary. Sounds like he's got drive.

Nit-picky corrections:

1. Top Putnam score in Colorado. There's a pretty big difference between that, and say, top Putnam score in Massachusetts (which is more likely the same as top overall due to many Putnam Fellows coming from Harvard or MIT).

2. Elementary proficiency in Classical and Leventine Arabic, Mandarin Chinese, and Koine Greek

It's not like there aren't plenty of smart people in Colorado.

http://www.colorado.edu/news/series/cu-boulder-nobel-laureat... (add David Wineland to that list).

Can you beat any of it?

Sure, I may have actually done better on the Putnam than he has. I've gotten top 100 and top 200 before... but that really wasn't the point I was trying to make. I was just making the correction because the original claim is, at least to me, much more impressive than what is actually stated on his resume.

Wow, #1 sets off my tryhard alarm. Especially at the college level where a huge portion of high Putnam scorers migrate to locations like Cambridge and California.

top 50 or even 200 overall or whatever is far more impressive than #1 in a state that has no reputation for high scores.

Maybe he loves the place he lives in, he's quite happy doing the kind of work he does, and thus want to stay instead of moving to another place.

The rat race is not for everyone.

“A 384-bit key I can factor on my laptop in 24 hours,” he says. “The 512-bit keys I can factor in about 72 hours using Amazon Web Services for $75. And I did do a number of those. Then there are the 768-bit keys. Those are not factorable by a normal person like me with my resources alone. But the government of Iran probably could, or a large group with sufficient computing resources could pull it off.”

"But the government of Iran probably could"...At this point I stopped reading, as this article became propaganda.

Did you know this month is National Cyber Security Awareness Month, as advertized by the DHS?


> At this point I stopped reading, as this article became propaganda.

Even if that was true (it's not), how could you know it without reading further?

What's not true? (what's 'it' that you talk about)

The article did not become propaganda.

The article up to that point was great.

However, that sentence "But the government of Iran probably could" made the preceding paragraphs appear to be a vehicle to deliver a meme (like a shaggy-dog story). The rest of the article could have been great, I just stopped reading.

The journalist could have made a neutral statement about what entities have the resources to crack a 768-bit key. But they or their editor chose not to.

Instead, everyone that reads the article will go away with the meme "Iran, if they wanted to, could crack 768-bit keys". Which is, by common definition, propaganda.

It might be unintentional, i.e. the journalist is riding a wave of popular opinion, which they should not do; or it might be an attempt to load the article with link bait.

I don't understand. That statement was part of a quote during the interview. A single, continuous quote. Do you consider reporting what someone said to be propaganda. Should the journalist have left out that part of the quote?

Good observation. I stand corrected. I wonder how Zachary Harris would defend the lack of neutrality of that quote, if he was asked to do so.

Hasn't (hackers in) Iran been behind hacking registrars and intercepting social networks etc? I thought it was a nod to that.

Unintentional still feeds into the problem. Journos should be extra careful, the have a microphone.

That's your choice if you don't want to read information if it contains certain sentences. No need to tell the rest of the world about it.

The same could be said about your comment. Touché.

> "But the government of Iran probably could"...At this point I stopped reading, as this article became propaganda.

How is that propaganda? You don't think most countries have that kind of computing power?

> How is that propaganda?

defn: "Information, esp. of a biased or misleading nature, used to promote or publicize a particular political cause or point of view"


2: the spreading of ideas, information, or rumor for the purpose of helping or injuring an institution, a cause, or a person

3 : ideas, facts, or allegations spread deliberately to further one's cause or to damage an opposing cause; also : a public action having such an effect

> You don't think most countries have that kind of computing power?

The quote itself states that: "...or a large group with sufficient computing resources could pull it off."

I think you may have misunderstood my point of view. The point is, in his quote, the interviewee (Zachary Harris) singled out Iran. As it stands it is an out-of-the-blue assertion.

How is it misleading? Iran has shown willingness to use fake certificates and the US has been cyber-attacking them. It's a reasonable example for a country. The point here is to make the threat less abstract by using a scenario, not to rally political sentiment.

Dangerous move, other companies have would set the police on him for that stunt.

There are a lot of stories of people getting jobs by disclosing security holes like this.

What a happy ending, no threats of jail or lawsuits.

It's still bit of a let down that they didn't contacted him in any way. They could at least track him and since he is a well meaning guy they could send him something funny.

Seems to be some conflation in this thread. Are DKIM and authentication (PGP) really comparable in practice?

Here's my take: DKIM is an attempt by _third parties_ (i.e. "email providers", not the author or the recipient of the message) to control who can send email (but guess what? anyone can send email, go figure). On the other hand, authentication (PGP) is an attempt to allow senders to sign messages and receivers to verify signatures (no third parties needed).

Bob printed his PGP public key on a card and gave it to Alice when they had lunch. He then signed an email message the following week using PGP and sent it to Alice. But Bob's "email provider" decided to block Bob's message because Bob didn't pay money to someone for the use of a "domain name" and Bob's "email provider" thought his email was "spam" because he hadn't been "authorized" (by paying money for use of a domain name) to send email.

Sloppy work by affected companies since RFC was unambiguous. But why didn't RFC keep it consistent by requiring verifiers to only work with the same minimum key length?

RFC 4871 (sorry for formatting but ipad issue) " signers MUST use RSA keys of at least 1024 bits for long-lived keys. Verifiers MUST be able to validate signatures with keys ranging from 512 bits to 2048 bits, and they MAY be able to validate signatures with larger keys. Verifier policies may use the length of the signing key as one metric for determining whether a signature is acceptable.

   Factors that should influence the key size choice include the

   o  The practical constraint that large (e.g., 4096 bit) keys may not
      fit within a 512-byte DNS UDP response packet

   o  The security constraint that keys smaller than 1024 bits are subjec to offline attacks..."

Ok, this got me scared, checked with Sendgrid support,of course they use 1024, back to breathing again.

I got one of those emails too once. I still can't figure out why. I did post to LKML a couple times in the past, but I haven't done anything kernel in over a decade. And a random Google recruiter emails me to congratulate me on my experience and offer me an unspecified position as a SRE. Not only do I have zero experience or interest in sysadmin and large server type stuff, they don't even have any facilities within 400km of me. What the fuck, Google?

In my experience (ex-Googler here), recruiters tend to be contractors who work off a script and get bonuses proportionate to how many people end up in the hiring pipeline. There are checks and balances to prevent outright spam, but the motivations are aligned in such a way that a pretty wide net is cast.

Also, until you have interviewed, all positions are "unspecified". Many positions need to be filled and they don't pick one for you until they know what you can do.

Geography is not really considered to be an issue. Once SRE finds someone they really want they will help with relocation.

Also an ex-Googler here, they cast the net far and wide trying to pick up SRE. When I was there they were offering significant bounties for submitting recommendations that only had to make it pass the resume screen. Want a brand new shiny PS3? Less than 10* that made it past the resume screen, and it was yours.

I has additionally heard rumors that recruiters were so silo'ed that they would actually just throw away a resume rather than route it. Reason being that they were in a competition with all recruiters, and worst performers (based strictly on a numbers game) didn't get their contracts renewed.

*May have been as low as less than 5, its been a few years, and I never really took to memorize what was posted on the wall while I was at the urinal.

DKIM is not the only tool for catching spoofed emails; to my knowledge SPF is more widely used because it is much easier to set up. I'd be shocked if the little Larry/Sergei joke email made it to their inbox since it would fail the SPF lookup.

Yes lots of sites have SPF, but what I've seen most sites set it to soft-fail mode.

I think this is because SPF is still sometimes broken in practice. For example, it can fail when there are misconfigured e-mail forwarding (e.g. mail aliases) at _other_ peoples servers[1]. Or with web forms that set the envelope sender to the "From" field in web page...


What is the Direct Link to this guy's website?

Props to Wired for disclosing that their silly phony photo setups are phony. I found that comforting.

Props to Google for fixing the problem instantly.

Weird that he thought the email was phony based on content. Who wouldn't want a computer savvy math genius on their team? Google has lots.

Google very rarely hires pure mathematicians AFAIK.

Well, afaik key length isn't the problem. Weak algo is. I assume they use RSA, they should use ECC. 512 bits is more than enough.


You can't use ECC for DKIM.

That doesn't make much sense. There are key lengths for which RSA is strong, and key lengths for which ECC is weak.

ECC keys may be stronger at shorter lengths, but that hardly means that key length isn't the problem. After all, using a longer key would fix this problem.

Using ECC would solve this particular problem, because in both DKIM and DNSSEC usage of key lengths of 512 to 768 bits and not more are motivated by what can fit into one UDP DNS reply packet. Also for RSA signature size is equal to key length, and not some small multiple of security parameter and you don't want to expand messages by including large signature that might well be larger than actual payload.

Right. Either expanding the key length or using ECC would fix it. So saying the problem isn't one of those, but is only the other, doesn't make much sense to me.

ECC may even be a better solution, as you say, but that doesn't mean that the problem isn't also one of an insufficiently long key.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact