
Is HTTP Public Key Pinning Dead? - okket
https://blog.qualys.com/ssllabs/2016/09/06/is-http-public-key-pinning-dead
======
eganist
I figured interesting words would probably start to trickle out about HPKP
after the last month or so of talks. Opinions are split on how much damage
RansomPKP and other hostile pinning attacks can do. I'm inclined to believe
that it's a big enough risk to warrant some action (such as more diligence by
free certificate authorities -- the crux of the talk was rekeying at up to 20x
per week enabling the attack). Others, notably Mozilla, are taking the wait-
and-see approach. Google and Let's Encrypt were the most resistant to changing
anything at all.

Anyway, Scott Helme's writeup did a quick and solid job of summarizing a lot
of it. It's linked from the Qualys piece, but here it is if you'd rather get
to it directly: [https://scotthelme.co.uk/using-security-features-to-do-
bad-t...](https://scotthelme.co.uk/using-security-features-to-do-bad-things/)

Full Disclosure: I and buu700
([https://news.ycombinator.com/user?id=buu700](https://news.ycombinator.com/user?id=buu700))
did the AppSec Glory talk at Black Hat / DEF CON and developed the RansomPKP
attack pattern.

Edit for Fuller Disclosure plus added thoughts: I'd rather not see HPKP die
because we've done some cool stuff with it over at Cyph. I'd rather see HPKP
be refined with extra requirements, possibly some sort of tie-in with CAA
(that little-known standard for enforcing which CAs can issue certs for your
domain).

Speaking of CAA, we did a show of hands in both the BlackHat and DEF CON
versions of the AppSec Glory talk. Five hands across thousands of attendees.
That's it.

~~~
jb613
> CAA (that little-known standard for enforcing which CAs can issue certs for
> your domain)

I'm sure the CA's absolutely _LOVE_ this - customers are signaling that they
will stay with CA 'X'.

Given 'vendor lockin' or 'guaranteed recurring revenue', cert prices would
drop thru the floor for 1st year only for renewals to increase.

~~~
pfg
CAA is just a DNS record that can be changed at any time, so there's no vendor
lock-in.

CAs that implement CAA request that record and check whether they're permitted
to issue the certificate; if they're not, they should refuse issuance.

~~~
jb613
1) CAA tries to address the threat of attacker getting a cert issued for your
domain - to carry out such an attack inherently requires taking over your DNS
record or your domain account - if they can do this then they can also
reconfigure your CAA record.

2) CAA requires the CA to perform additional operation (retrieve and check
that DNS record) - not all of the 300-600+ CA's will do this - all it takes is
1 CA and the attacker will get his fraudulent cert from that CA.

~~~
notatoad
>CAA requires the CA to perform additional operation (retrieve and check that
DNS record) - not all of the 300-600+ CA's will do this - all it takes is 1 CA
and the attacker will get his fraudulent cert from that CA.

CAA seems like another good step that the browsers can demand after a CA fucks
up - like google has mandated that some CAs publish Certificate Transparency
logs after they've mis-issued a cert, they should also be requiring CAA on
threat of being revoked from the browser's trust lists.

~~~
jb613
Yeah - in theory, but in reality the browsers are reluctant because of all the
other sites that a CA has issued certs for - no browser wants to be singled
out for blocking some other non-related sites.

------
tptacek
This is going to sound glib, but here goes:

1\. The fact that HPKP can create this kind of gigantic foot-gun is a pretty
good sign that it actually works. So much of security technology turns out to
be cosmetic. The stuff that isn't often is hazardous.

2\. Don't HPKP your blog. Don't HPKP your cat-sharing starting. Key pinning is
a good thing, but it doesn't need to be universally deployed.

3\. In order to hijack your site with HPKP, an attacker needs to take control
of your server; they probably need RCE. In other words: if you can't safely
HPKP your secure messaging system (cough), you probably aren't yet qualified
to be running it: something worse happened to your users than HPKP
misconfiguration.

If there's a capability we're missing right now, it's probably the ability for
sites with low security requirements to opt out of HPKP. Remember, the threat
model for HPKP isn't fraudsters or trolls; it's state-level adversaries.

~~~
jb613
> 2\. Don't HPKP your blog. Don't HPKP your cat-sharing starting. Key pinning
> is a good thing, but it doesn't need to be universally deployed.

Just to be clear, this is equivalent to saying don't bother with SSL for your
blog or cat-sharing site (because without pinning someone else can get a cert
for your blog/cat-sharing site).

> the ability for sites with low security requirements to opt out of HPKP.

opt out?! It's off by default.

> Remember, the threat model for HPKP isn't fraudsters or trolls; it's state-
> level adversaries.

Again, by saying HPKP is only for high security sites (now you're saying only
for those threatened by state-level adversaries) you are essentially saying
SSL is useless for everything less.

~~~
pcl
> > the ability for sites with low security requirements to opt out of HPKP.

> opt out?! It's off by default.

I think tptacek meant that a site owner should be able to explicitly declare
that nobody should be able to pin a key, so that nobody can hijack the site in
the future.

~~~
nsgi
An opt-in would make more sense, since people using HPKP are those that know
the most about it. Just like an opt-out, it would have to be via a different
channel than HTTP headers.

------
zeveb
HPKP is a workaround, not a solution, to the problem of trusting every CA in
the world to certify every site in the world — it's a symptom of the fact that
the economics of CAs are completely, totally and fundamentally out-of-whack:
relying parties have no relationship with the CAs they trust.

There are two things a CA can do: it can a) certify that a key is associated
with some real-world identity or b) certify that a key is associated with some
DNS name. The first is actually pretty desirable, but … there is no global
namespace of real-world identities, so there's nothing a CA CertCorp can
certify other than 'public key 0x1234ABCD0987FEDB is associated with CertCorp
identity X,' which is … not nearly as desirable, or useful.

The second possibility is actually pretty useful, but also imperfect. To a
huge extent, Amazon really is amazon.com; Google really is google.com. But, of
course, .com could always decide to reallocate hostnames at will, regardless
of its contracts (there may be contractual, civil or criminal remedies, of
course).

Where does HPKP come into all this? It really doesn't: it's a half-assed way
for a site to say, 'you visited me once, and trusted CA X to verify my
identity then; in the future, please trust CAs X & Y' — but it does nothing at
all to actually ensure that CA X was reliable in the first place! After all,
CA X could have issued a certificate to some malfactor, who served the HPKP
instructions to only trust X & Y.

It's smoke and mirrors, all the way down.

~~~
pfg
There's nothing stopping you from pinning exclusively to keys under your
control. A pin counts as satisfied if the hash matches at least one of the
keys in the certificate chain, including the end-entity certificate itself. In
other words, if you do not trust _any_ CA, HPKP can still help you.

For most sites, pinning to a small set of CAs is sufficient and a vast
improvement, but if you chose to trust no one, that's fine too.

------
mieko
Here's an issue not mentioned in the article, which gives me the half-baked
feeling about HPKP:

Thought experiment: (just using companies I think HN will be familiar with,
I'm not involved with these.)

1\. You're Tumblr or Basecamp or Slack with thousands/millions of
user.example.com user subdomains

2\. You want to protect all of them with a root HPKP policy with
includeSubdomains, because users will be bouncing between them, and with that
many trust-on-first-use gaps, it'd be pointless otherwise.

3\. And then you want status.example.com hosted at StatusPage,
blog.example.com hosted at Medium, and someapp.example.com hosted via GitHub
Pages with a CNAME. Assume there are long-lived links to all of these
subdomain URLs.

You can even assume that the third parties implement HPKP, and that they'll
never have a hiccup.

There's no way to make this work without either:

A) Removing includeSubdomains, and no longer catch a large amount of cases as
users traverse your site, barely solving a problem anymore

B) Serving a concatenation of every third party pin, opening up gaps huge
enough where suddenly every CA is now authorized again, not solving a problem
at all.

C) Noticing something I haven't.

And note that there are security (not merely vanity) reasons for using
subdomains instead of paths for hosting user-generated content. I don't think
this is too crazy of a scenario.

~~~
mseebach
That's fairly easy to fix by putting your users on user.fooapp.com and your
corporate pages on *.foo.com, just like you're already (for a different
reason) putting static resources on foo-static.com.

~~~
mieko
I think a plumbing-level security implementation detail should be a little
less intrusive than having to move either the corporate domain or millions of
user's addresses. I can't imagine Tumblr rolling out HPKP now with this
workaround (though github did do the .com -> .io thing, so who knows).

The spec should've just included excludeSubDomains=

~~~
mseebach
For security, especially at the plumbing-level, simplicity is a virtue.

------
niftich
_EDIT: I was mistaken about something fundamental in how TLS operates.
Unchanged question included below:_

I have an amateurish question. Please correct me with counterpoints if I'm
mistaken.

In an attempt to 'explain like I'm five', HPKP adds an additional checksum, so
that the server can say, 'if you see any certs whose Subject-Public-Key-Info
doesn't match this checksum, _those aren 't me_, because I generated this
checksum off of my cert's Subject-Public-Key-Info.

But since the Subject-Public-Key-Info is (by definiton) public, a CA _could_
issue an owner-unsolicited cert to the same site, containing the same Subject-
Public-Key-Info, and which will pass pin validation and allow the site to be
impersonated [1]. This may happen if the CA is rogue, or if the CA makes a
mistake and issues a certificate to some bad actor other than the actual site
operator.

So if HPKP is designed to protect against CA errors (malicious or accidental),
how can it do that when it can't protect against CA errors, malicious or
accidental?

[1]
[https://tools.ietf.org/html/rfc7469#section-4.5](https://tools.ietf.org/html/rfc7469#section-4.5)

~~~
yuubi
If the attacker doesn't have the private key that matches the Subject-Public-
Key-Info, then he won't be able to impersonate your server given such a cert.
(If he has, then there's no need to produce a replacement cert; the official
one is good enough).

------
DyslexicAtheist
the issue is that HPKP is as user friendly as GPG but without the same
benefits. (learning-curve vs ROI) Also IMO the underlying problem is choosing
a CA's in the first place. None of them have 'Skin in the Game'[0]. How do you
actually decide on which CA is trustworthy?? Trust is measured by how long
they have been in the business and how many trust related incidents[0] they
had. And how is that confidence measured/created? (hint it isn't Math but
whoever has the best marketing or lowest price - not the most trustworthy
certs).

CAs are a business model impatiently waiting to be replaced (with something
decentralized). It's a self-serving quasi-monopoly that all other vendors that
want to implement trust must use.

[0]
[http://www.fooledbyrandomness.com/SITG.html](http://www.fooledbyrandomness.com/SITG.html)

[1] [https://www.schrauger.com/the-story-of-how-wosign-gave-me-
an...](https://www.schrauger.com/the-story-of-how-wosign-gave-me-an-ssl-
certificate-for-github-com)

~~~
DyslexicAtheist
afterthought: I wouldn't actually call HPKP dead. It's a prerequisite if you
want to do any PKI related functions on your site. E.g. <keygen> support was
dropped by most browsers and the industry is heading towards native clients.
Nevertheless WebCrypto[0] is supposed to offer the same type of quality and
you need both HSTS/HPKP if you want to secure your implementation in any
meaningful way.

[0] [https://www.w3.org/TR/WebCryptoAPI/](https://www.w3.org/TR/WebCryptoAPI/)

------
groue
May I ask a stupid question? Last time I heard about HPKP was as a solution to
prevent an attacker from installing a fake root CA certificate in the _client
system_ (a mobile device), so that it could observe the behavior of a mobile
app, by posing as its API server (for any purpose of reverse-engineering). Is
such an attack possible, and HPKP a reasonable solution to it?

~~~
vbezhenar
Locally trusted CA certificates circumvent HPKP protection, because many
organisations use TLS MITM attack to spy on their staff and browsers must
support it in order to work properly in those environments. But it makes sense
for browser. If you are making your own application, you can implement it in
any way. Simplest way would be to embed public key in the application and
check it without HPKP. Though, probably, if your app is popular, you'll face
that in some networks your connections are MITMed and you (or user) can't do
anything about it, so it's choice about dropping security or not to work at
all.

~~~
groue
> Simplest way would be to embed public key in the application

Yes, but one eventually faces the need to update the certificate on the
server, and still support installed applications (which fails when apps have
an obsolete certificate pinned). Hence the need for smooth certificate
updates, and the good smell of HPKP. But it addresses another need, hence my
confusion.

------
rocqua
DANE certificate pinning seems to like it solves most issues HPKP does,
without the imminent danger.

It won't stop a dedicated attacker from MITMing a connection if they can block
DNSSEC. However, if someone uses false BGP route to hijack your IP, they can't
reach your DNS queries.

You'd still be fucked if you connect to a local wifi connection though.

~~~
tptacek
The downsides of DANE are substantial:

* DANE is effectively a key escrow system. COM's keys would be held by the USG.

* In the same vein: people with cosmetic TLDs like .LY would be forced to concede their TLS policy to governments like Libya. Similarly: the Internet would be conceding that protocols should work differently and be less trustworthy depending on what country you're in.

* The "last mile" on DANE is spoofable, because DNSSEC doesn't protect the link between browsers and DNS servers; it's a "server-to-server" protocol. In particular: your ISP could still override pins!

* DANE doesn't actually work in practice, because of all the networks that block odd-looking DNS lookups; this is Adam Langley's point from "Why Not DANE In Browsers".

* Stale or misconfigured DNSSEC signatures knock whole sites off the Internet with no diagnostics. Unlike a broken HPKP pin, there's no UX saying something's wrong with the site; it just doesn't exist anymore. This is what happened to HBO Now on Comcast, and why Comcast users occasionally can't reach .GOV sites (which mandate DNSSEC).

* DNSSEC is far, far more expensive to deploy than HPKP (HPKP is just a header you add to your web server configuration).

* The cryptography behind DANE is 1990s-grade. The DNSSEC roots are still RSA-1024! (Don't worry! They've got the upgrade to RSA-2048 on the calendar!)

~~~
garaetjjte
> * DANE is effectively a key escrow system. COM's keys would be held by the
> USG.

They can already temporarily change DNS records and get certificate from any
legitimate CA.

> * In the same vein: people with cosmetic TLDs like .LY would be forced to
> concede their TLS policy to governments like Libya.

They already are doing this, see first point.

> * The "last mile" on DANE is spoofable, because DNSSEC doesn't protect the
> link between browsers and DNS servers; it's a "server-to-server" protocol.
> In particular: your ISP could still override pins!

Run a local resolver. If you have trusted server some you can use DNS over
TLS.

* DANE doesn't actually work in practice, because of all the networks that block odd-looking DNS lookups; this is Adam Langley's point from "Why Not DANE In Browsers".

Then these networks are bad. For what reason network should be interested with
more than UDP/TCP layer in DNS packet? Again, you can use DNS over TLS.

For other readers, look at discussion in
[https://news.ycombinator.com/item?id=12382752](https://news.ycombinator.com/item?id=12382752)

~~~
tptacek
Regarding your first point: no, they can't. See what happens if they try to
exploit the DNS to get a Google certificate. (Notice the last several years of
Google punishing CAs).

It is very important to DNSSEC advocates to convince people that our security
can't be any better than the security of the .COM zone, which is controlled by
the US Government through Verisign. But nobody accepts that level of security
today. Arguments that we can escrow all our security with Verisign are DOA,
post Snowden. Move on.

Regarding the rest: you can't persuasively rebut arguments by conceding
they're correct.

~~~
zeveb
> Regarding your first point: no, they can't. See what happens if they try to
> exploit the DNS to get a Google certificate. (Notice the last several years
> of Google punishing CAs).

If the U.S. government, or the Russian government, or the Luxembourgian
government orders a CA subject to it to produce a certificate for a given key
and hostname, it will be given a certificate for that key and hostname.

If the U.S. government, or the Russian government, or the Luxembourgian
government orders a DNS provider subject to it to resolve all requests for a
domain to a different IP than it would have otherwise, those requests will be
so resolved.

If the U.S. government, or the Russian government, or the Luxembourgian
government orders a network subject to it to route all traffic for one IP to a
different IP it controls, that traffic will be so routed.

> It is very important to DNSSEC advocates to convince people that our
> security can't be any better than the security of the .COM zone, which is
> controlled by the US Government through Verisign. But nobody accepts that
> level of security today.

Everybody accepts that level of security today, because that's the level of
security we have today.

Actually, it's _better than_ the level of security we have today, because at
least a DNS-level CA system would mean that no-one _other than_ someone who
can compel .com can take over a .com hostname. Right now any government in the
world can do so, at the drop of a hat.

Restricting that to the government a company or person is subject to greatly
reduces the attack surface.

~~~
tptacek
If the US government orders a CA to mint a bogus certificate for Google Mail,
they will get the certificate, and then the CA will get put out of business,
one way or the other.

There are consequences to suborning the TLS CA system. DNSSEC advocates
pretend there aren't, because they have to, or their arguments become
untenable, because there cannot plausibly be similar consequences for
suborning the DNS: we know for a fact, with repeated existence proofs, that
governments will manipulate the DNS (like DOJ did with file sharing sites),
and we know .COM isn't going anywhere.

DNSSEC advocates will grudgingly acknowledge one half of the modern CA system
security model: key continuity (pinning).

What they seem unable to acknowledge is the other half. To use Moxie
Marlinspike's terminology, which he coined in an argument against DNSSEC:
trust agility.

~~~
rocqua
> If the US government orders a CA to mint a bogus certificate for Google
> Mail, they will get the certificate, and then the CA will get put out of
> business, one way or the other.

Probably, but in the year (or more considering that startSSL is still trusted)
until the CA's trust is removed, that certificate still works.

Besides, if you accept that governments manipulate DNS, why do you deny they
manipulate DNS to get domain validated certificates? Cause if not, we are back
at trusting verisign for all websites that aren't statically pinned. (with
that added 'bonus' that those statically pinned sites can be wrecked by their
pin at any time)

------
vbezhenar
I won't use or recommend HPKP until there'll be a way to recover domain with
lost keys. I think of something like bit in the certificate which will
override HPKP and this bit should be very thoroughly verified by issuer,
something like EV. Risk of losing domain is unacceptable and overweight
security gains.

~~~
ivanr
Well, there's an idea: require a special certificate (via an extension) to
enable pinning.

------
jnky
I recently had an idea to make HTTPS more secure for the average user: What if
Google were to check the certificates when crawling a site, ensuring that its
keys did not change in a malicious way.

Now, sites could present a different cert to Google IPs and to users. To solve
this problem, the html anchor element could be augmented with the certificates
fingerprint. The users browser would make note of the fingerprint and alert
the user if something is off.

I think embedding information about a certs finger print or its root CA in
links makes a lot of sense, as it would allow a sort of "web of trust" between
linked sites.

~~~
blowski
I get the idea, but it sounds like it would make the problem of link-rot 100x
worse, and the number of broken links would quickly desensitise users to the
warning.

~~~
lallysingh
Obviously it'd be optional, and it'd only make sense for dynamically-serving
systems like a web search engine.

 _And_ a multi-domain'ed serving system. Like a blog host. It could serve
links to other blogs it hosts (on completely different domains) with cert
hashes, and suddenly make it very hard to MITM any of the served sites without
MITM'ing all of them.

------
pfg
The article makes some good points, but I don't believe the risk of malicious
pins has any effect on adoption. It doesn't really matter whether your site
currently uses HPKP or not - the attack works either way, as long as the user
agent supports HPKP. Once an attacker has compromised your server in a way
that allows them to add arbitrary HTTP headers, you're doomed.

That's not to say that things like RansomPKP aren't problems worth thinking
about, but when it comes to adoption, time would probably better be spent
improving tooling and documentation.

------
ejcx
Like tptacek said, "Don't HPKP your blog".

There's only a handful of sites that need HPKP and that's okay. That doesn't
mean there is something wrong with HPKP. It's a useful tool for people with a
specific threat model. If you are one of these sites then deploy HPKP and
handle the operational responsibility that it comes with.

If you aren't one of these handful of sites then do not deploy HPKP. HPKP is a
huge footgun and the benefits are not super large on an average case.

~~~
ivanr
I think high profile sites are better served by static pinning, which seems
simpler to maintain and also comes with the benefit of preloading. Thus, if
you have those sites at one end, and lots of small/no-large-risk sites who
don't need any pinning on the other end, you're left with no one for HPKP to
serve in the middle. But, if HPKP were easier to deploy, perhaps more sites
would use it.

------
wyldfire
> Or, if you’re lucky, they seek ransom, giving you a chance to get the backup
> pinning key from them and keep your business.

That's a rough choice -- let the business stumble/fail by moving to a new
domain or accept the keypair from an untrusted party who's shown themselves to
be untrustworthy. What's to stop them from sharing the private key with other
ne'er-do-wells?

~~~
vbezhenar
You can gradually migrate to another non-compromised key. Generate new key,
add HPKP headers for both compromised and non-compromised key, keep it long
enough so all users who got bad HPKP headers will get new ones, then replace
certificate and set HPKP header to include only new keys.

------
kelnage
The problem is if you dull HPKP, you potentially weaken the benefit that the
really high-profile targets gain and therefore make it less useful for them.

Perhaps the answer is only a small number of websites actually need/want this
functionality - and that's okay? Simply dulling it to broaden take-up may have
a net negative affect.

------
mkagenius
The article did mention this but I will re-iterate that mobile apps can
benefit a lot from key pinning. That's a huge chunk of total internet
activity. So, not that dead if you ask me.

The scenario of attacker having root(?) access to the machine warrants much
bigger (or equal) concern than bricking your website in my opinion.

~~~
ivanr
Just to clarify: in my article I discuss mostly HPKP as defined in RFC 7469. I
think static pinning works great (and also has the benefit of being
preloaded). Mobile and other types of pinning where you control both sides of
a conversation also work well and are easier to get right.

------
dorfsmay
Wouldn't a new record in DNS a safe dynamic way to publish the certs
signatures?

~~~
KayEss
It's too easy for the same attacker to MITM both the certificate and the DNS.

------
Tergmap
He says "bricking your website" while he means "your website will appear as if
it has an invalid certificate".

Bricking means to crash something beyond repair.

~~~
ivanr
No, when HPKP breaks your site no longer works, period. The error page doesn't
allow clicking through. Some browsers (e.g., Chrome) support manual editing of
the HPKP configuration so some users might be able to get around the problem,
but that's unlikely to work for many.

Try it here: [https://pinning-test.badssl.com](https://pinning-
test.badssl.com)

