
Issue with TLS-SNI-01 and Shared Hosting Infrastructure - jlgaddis
https://community.letsencrypt.org/t/2018-01-09-issue-with-tls-sni-01-and-shared-hosting-infrastructure/49996
======
jlgaddis
> _At this time, we believe that the issue can be addressed by having certain
> services providers implement stronger controls for domains hosted on their
> infrastructure._

Requiring unrelated third-parties to take specific action(s) to prevent Let's
Encrypt from incorrectly issuing certificates doesn't sound like a valid
solution to this issue.

What about the shared hosting providers that are "affected" that decide to do
nothing to "fix" this?

~~~
tialaramex
In specific answer to your second question, Let's Encrypt intends to blacklist
IPs for those providers from using tls-sni-01 proof of control.

Now, requiring unrelated third-parties to take action is (perhaps
unfortunately) nothing new for DV. Back in the day lots of the web mail
providers got bitten because some CA would decide that, say, receipt of email
to ssl-certificate-officer@example.com is proof of control of example.com, so
you just sign up for web mail and say you want the email address ssl-
certificate-officer and bingo you get a certificate for a few bucks. This was
exploited over and over.

Even today, the Ten Blessed Methods let a CA choose any email address you
listed in WHOIS (fair enough, don't list anybody else's address in WHOIS) _or_
any of 'admin', 'administrator', 'webmaster', 'hostmaster', or 'postmaster'.

So when you build a web mail system you need to explicitly blacklist those
five email addresses even if you would never need an account named "webmaster"
or "admin", if you fail to take this specific action anybody can get
certificates for your domain by talking to any of the commercial CAs that use
a method complying with 3.2.2.4.4.

Let's Encrypt already disabled a feature of http-01 validation before going
live to avoid needing bulk web hosts and CDNs to make a sensible configuration
choice, because surveys suggested that lots of them had left insecure default
choices enabled.

Or another email example (again thus not affecting Let's Encrypt): Lots of
companies purchased anti-phishing systems that scan all incoming email to
check for dodgy links. These systems typically dereference all the links,
check what they find and quarantine suspicious emails. Several CAs had systems
which sent an email saying "Just click here to authorize issuance". The anti-
phishing system follows the link, now it's clicked, anyone could get a
certificate for these domains because the anti-phishing system would
"authorize" anything they requested...

Now, all these are historical rather than current examples, but they do
illustrate that "requiring unrelated parties to take specific actions" won't
be new or unique to Let's Encrypt.

~~~
ademarre
> _" Let's Encrypt already disabled a feature of http-01 validation before
> going live"_

I'm curious what this was. Can you elaborate or point me somewhere I can read
about that?

~~~
tialaramex
Sure, http-01 as defined originally has a flag for whether to do the
validation with plaintext HTTP on port 80 (as today) or using HTTPS on port
443.

Now, suppose [http://no-ssl.customer1.example/](http://no-
ssl.customer1.example/) is a site Customer 1 is running on a bulk host, and it
doesn't have SSL. We (the attacker) ask Let's Encrypt for an HTTPS version of
the http-01 validation for this name no-ssl.customer1.example

A DNS lookup of no-ssl.customer1.example gives Let's Encrypt the IP address of
the bulk hosting server. They connect to port 443 of that address, and they do
SNI and say they expect to talk to no-ssl.customer1.example, then they do an
HTTP request with Host: no-ssl.customer1.example and a GET for the relevant
.well-known/acme-challenge/ file - what happens?

A common default configuration of web servers used by bulk hosters says if
anything doesn't match, let's use a "default" such as the first (often
alphabetically first) named virtual host to serve the answers. No error, just
assume that must be what the request was really for.

Since there is no SSL version of no-ssl.customer1.example, the server drops
through to this default, and the web server offers the certificate and content
for aaaaa.attacker.example which we, the attacker have prepared. This includes
the ACME answer file, Let's Encrypt is satisfied we have "control" over no-
ssl.customer1.example and grants us a certificate we clearly shouldn't have.

Apache and nginx both do this by default. Lots of bulk hosts are set up to use
them without a configuration that prevents this happening, and they didn't
seem to be in any hurry to fix it.

So, Let's Encrypt disabled this feature for their production release.

~~~
ademarre
Thanks for expanding. That's interesting. So they disabled http-01 over TLS
[0]. Two questions:

1\. Isn't the default virtual host practice still a problem even without
TLS/SNI? Apache chooses a default virtual host on port 80 in the same way. Is
that "acceptable" simply because it's more common for the default port 80
virtual host to be controlled by the hosting provider? That's not as robust as
one would hope.

2\. Wouldn't http-01 over TLS be made safe if the certificate was required to
be valid for the domain being challenged? Granted, there's a TLS chicken and
egg problem there, but that's beside the point.

[0] [https://tools.ietf.org/html/draft-ietf-acme-
acme-09#section-...](https://tools.ietf.org/html/draft-ietf-acme-
acme-09#section-8.3)

------
rgbrenner
_Over the next 48 hours we will be building a list of vulnerable providers and
their associated IP addresses. Our tentative plan, once the list is completed,
is to re-enable the TLS-SNI-01 challenge type with vulnerable providers
blocked from using it._

This is a terrible solution.

You're assuming you can scour the internet and find every shared hosting
provider affected, and add them to a list. And then you're also going to keep
that list updated. That's crazy.. an impossible task.

So after this is done, Let's Encrypt will continue issuing certificates
improperly on any shared hosting providers that didn't get added to this list.
Attackers just need to find a provider and check a list now.

I'm actually kind of appalled that this is what you've come up with.

~~~
DanielDent
I think this issue can be described as a form of confused deputy problem.

Is not the real solution to have the confused deputies stop acting in a
confused manner?

If this issue were confined to a handful of small hosting companies almost no
one uses, then I don't think this would be described as a problem with Let's
Encrypt - it would be described as a problem with those hosting companies.

The only party that can truly fix the issue of the incorrect user being
granted access (directly or indirectly) to a domain pointed at shared
infrastructure is the operator of that infrastructure.

~~~
e_d_e_v
I came here to say this. What's more, the spec was agreed upon, in relatively
public forums, with a voice from the community. Crappy shared hosting
providers are going to mostly ignore their customers and perpetuate insecure
scenarios while they continue to bill exorbitant rates that exploit the
customers' ignorance or inertia. That has been the case for some time, and
will continue to be the case, this is just another symptom.

~~~
rgbrenner
_What 's more, the spec was agreed upon, in relatively public forums, with a
voice from the community._

It was agreed upon and no one caught this issue. Now we know the issue.

There's nothing wrong with using a protocol you think is correct. There is
something wrong with using a protocol you know is incorrect, but continue to
use it anyway.

The entire internet should not be required from now to forever to workaround
LE's mistake. LE should fix their protocol.

And worse, this protocol isn't even needed for LE. They could remove it, and
everyone could use one of the two others that are secure, and LE would be just
fine, and everyone -- even those crappy shared hosting providers -- would be
perfectly secure.

LE created this issue all by itself, and is capable of fixing it all by
itself. LE should do that.

~~~
DanielDent
LE "created" this issue in the sense that they were the first to formalize an
implementable specification for automated verification of authorized domain
name use.

In comparison to the prior relatively unspecified approach to verification,
it's still an improvement.

"The last person who touched needs to take ownership over anything anyone can
blame on the change" is the management style which leads to enterprise IT
being unable to get anything done. Because at that point, the safest thing is
to never change anything - regardless of how bad things currently are.

Sometimes the world changes, and other parts of the technology ecosystem need
to adapt.

[Editing to add since I can't reply to you]: The fundamental "flaw" here is
that it's possible for people to get self-signed certificates served for
domains where they haven't validated ownership of the domain.

Hosting companies can't be simply adjusting their routing tables for anyone
who asks. If you are pointing a domain name at an IP address which will accept
routes from any untrusted party, that's simply not a secure situation.

A _signed_ certificate might be good evidence of some authority for a domain,
but a self-signed certificate used in a challenge process most assuredly is
not.

------
paralelogram
I can create "773c7d.13445a.acme.invalid" in almost all shared hosting control
panels I have access to.

When I send an HTTP request with Host: 773c7d.13445a.acme.invalid, the server
responds with a file from ~/domains/773c7d.13445a.acme.invalid/public_html or
a similar directory available through my FTP account.

When I connect using openssl s_client ... -servername
773c7d.13445a.acme.invalid, the server sends a certificate configured for
773c7d.13445a.acme.invalid in my control panel.

Is this a problem for Let's Encrypt? Doesn't Let's Encrypt's verification
require creating files with random names in [http://example.com/.well-
known/acme-challenge](http://example.com/.well-known/acme-challenge) where
example.com is the certificate's common name?

~~~
pfg
> Is this a problem for Let's Encrypt? Doesn't Let's Encrypt's verification
> require creating files with random names in [http://example.com/.well-
> known/acme-challenge](http://example.com/.well-known/acme-challenge) where
> example.com is the certificate's common name?

That applies to the http-01 challenge. The tls-sni-01 challenge works solely
based on the returned certificate. If the SAN value in the certificate matches
the SNI value sent by the validation server, the challenge succeeds.

Would you mind sharing which control panel you tested this with?

~~~
paralelogram
DirectAdmin, the most popular webhosting control panel in my country.

In my opinion this is not a bug because when I need to test a website, I often
create an invalid hostname on the server and add the server's IP address to my
computer's /etc/hosts. When I need HTTPS, I upload a certificate for the test
hostname signed by my private CA.

~~~
pfg
Thanks. I signed up for the first shared web hosting provider I could find
that uses DirectAdmin and was able to reproduce this. I'll bring this up in
the relevant thread on mozilla.dev.security.policy, this is definitely
concerning.

------
BillinghamJ
Ultimately, if you're attempting to prove hostname control, it would seem
fundamentally incorrect to make a request with a Host header or SNI hostname
set to anything other than the hostname requested for issuance.

I don't really understand why this didn't come up while considering the
protocol initially.

Why aren't we using ALPN for this?

(ALPN is used to negotiate HTTP/2 without prior knowledge of the server's
capability - [https://en.wikipedia.org/wiki/Application-
Layer_Protocol_Neg...](https://en.wikipedia.org/wiki/Application-
Layer_Protocol_Negotiation))

~~~
tonyarkles
> it would seem fundamentally incorrect to make a request with a Host header
> or SNI hostname set to anything other than the hostname requested for
> issuance.

Pondering why they'd do it the way it is right now, my best guess would be
that it allows you to get a LetsEncrypt certificate while still having a valid
certificate from another CA. For example, example.com currently has a Symantec
certificate that is soon going to be invalidated, and I want to swap over to
an LE cert with 0 downtime.

~~~
BillinghamJ
You could still do that! Just accept LE’s requests with the existing Symantec
cert and respond as needed.

Or use the DNS challenge.

Or use something like ALPN instead.

~~~
tonyarkles
How would LE do SNI validation using the Symantec cert though? I can't add
anything to it without invalidating it. My understanding is that that's why
they do the .acme.invalid SNI responses; so that the server can respond with a
self-signed cert that's not going to interfere with anything else.

DNS and ALPN are both great solution, but here's why neither of them work all
that well in my specific circumstances, but TLS-SNI works awesome.

\- I can provision VMs at will, but I don't personally have access to DNS. To
get a DNS record added, I need to submit a change request. I haven't done the
DNS challenge, and I'm not sure if a new TXT record gets generated when you
renew a cert or if it's static after initial creation, but either way, DNS
very much adds a painful manual step to cert provisioning here.

\- ALPN would be great, if it were baked into the default Centos Apache :).
That, I suspect, would be a long time coming...

Right now I generally just use HTTP validation, because I've worked out a
pattern that generally works pretty good. I set up an Apache rule that auto-
redirects all traffic from http to https anyway; I just add a condition to it
so that /.well-known/ gets served off the filesystem before redirecting
everything else.

~~~
BillinghamJ
I don't see why you couldn't do the HTTP validation, but over the existing
HTTPS cert you have, without having to accept non-TLS connections.

To be honest, I just don't think the SNI challenge has any place here.
Although I totally recognise it is an easier option where you're more
restricted, the fundamental security concerns are more important.

~~~
tonyarkles
Your sibling comment suggests that the LetsEncrypt verifier would potentially
follow a redirect HTTPS, which I'm going to have to experiment with today
because it'd keep things simpler here.

Best I know, currently HTTP validation is required to happen on port 80. And
that works awesome for bootstrapping; on a new install, you don't need to
provision a fake self-signed cert to be able to request a real one. If it can
happen over HTTP or HTTPS, that would be amazing.

~~~
BillinghamJ
It seems... very strange that it doesn't allow the use of port 443/HTTPS with
a self-signed cert. It's exactly the same thing.

------
jcassee
[Copying my comment from
[https://news.ycombinator.com/item?id=16112237](https://news.ycombinator.com/item?id=16112237)]

Unfortunately, this means that Traefik's default Let's Encrypt integration
(without setting a DNS provider) does not work anymore. Although the logs now
say "could not find solver for: http-01", they actually use tls-sni-01.

~~~
BillinghamJ
The entire Golang autocert functionality relies on this - which presumably
Traefik is using

~~~
fapjacks
Through a twist of fate, I was looking into this over the weekend, and you're
correct about Traefik.

------
tialaramex
My _guess_ at this point is that there's a bunch of scary corner cases with
the behaviour that makes this possible so that if your bulk host or CDN does
this you need to be _very_ careful even if Let's Encrypt abolishes tls-sni
family proofs of control. If I'm right, fixing this is pretty urgent for
anyone with untrusted users sharing an IP address.

It looks very much as if affected hosts/CDNs are letting any customer
impersonate any other customer with the only caveats being that you've got to
get in first and you need the victim customer to make the config change that
causes this to be useful.

Suppose I am victim.example, I have a contract with Funky Jim's Hosting, and I
have configured *.clowns.victim.example to go to a server at Funky Jim's but
I've only every bothered setting up www.clowns.victim.example,
images.clown.victim.example and js.clowns.victim.example in Funky Jim's config
system. I use HTTPS, so the valuable Clown Login tokens available on
www.clowns.victim.example are protected - aren't they?

Some loser, let's call her Sad Carol buys a basic account from Funky Jim, and
tells Funky Jim they want to add a new site, busted.clowns.victim.example.
Funky Jim OKs it, because that's "harmless" right? No need for Carol to prove
to Jim that she owns it, if not why would it matter? But wait, actually
busted.clowns.victim.example matches my wildcard DNS entry, it goes to the
right server and serves up Carol's site - now Sad Carol can get all my bloody
Clown Login cookies sent to "her" web site just by injecting "her" site into
an ad inventory or whatever.

~~~
mdhardeman
An excellent example of a plausible attack scenario.

Having said that, I think what we're all dying to know is what was it,
initially, that made Carol so sad?

And, indeed, was it that same sadness which stagnated, fermented, and evolved
into the obvious hatred and antisocial behaviors that she exhibits here?

------
jwilk
See also:
[https://news.ycombinator.com/item?id=16112237](https://news.ycombinator.com/item?id=16112237)

------
benmmurphy
does this mean you can collide legitimate certificates on these hosting
providers? like someone is hosting www.foo.com can I upload a www.foo.com
certificate then it will choose which cert to use based on some random
factors. or maybe it only works on names that haven't been uploaded so i can
block someone from uploading certificates for a future domain they might want
to use.

~~~
jo909
Maybe, but that is a different issue.

If as a hosting provider I have accounts for foo.com and bar.com, I certainly
need to make sure to never reply to requests for foo.com with data from the
account bar.com.

But I might still allow the account bar.com to upload a certificate that
includes bar.com www.bar.com bar.io getbar.com or any other domain name even
if those are not registered in my platform.

So what if bar.io is actually not owned by the same person as bar.com? The DNS
for bar.io does not point to my platform, no real user will connect to me.

If an attacker is also able to change the DNS for bar.io (say by an MITM
attack in a public wifi) it is of no real consequence that he uses my
platform. He could just as well respond with an IP completely of his own and
do everything himself. That is why we have SSL in the first place.

~~~
benmmurphy
so if someone has www.foo.com registered with a provider (but has wildcard DNS
pointing *.foo.com to the provider) would it be possible for someone who has
another domain bar.com to upload a cert that would be used for x.foo.com.
because even though no content would be served from bar.com that is kind of a
weird situation.

like i think the way ACME tls-sni works is broken because it should be using
the DNS hierarchy to make the SNI request [maybe something like:
773c7d.13445a.acme.foo.com] but maybe some providers are still broken even
with this fix because they let people upload certs with names that belong to
other clients.

------
nodesocket
> At this time, we believe that the issue can be addressed by having certain
> services providers implement stronger controls for domains hosted on their
> infrastructure.

Is DigitalOcean one of these service providers that need to implement stronger
controls?

------
mh-cx
Does this basically say, that letsencrypt will no longer work in a virtual
hosting setup?

What about existing certs? Can they at least be renewed when they expire?

~~~
luto
Let's Encrypt works just fine in a virtual hosting setup using the http-01 or
dns-01 challenges. In fact, all (shared/virtual hosting) users in my company
automatically get certificates this way.

~~~
v4tab
Is there a way to tell whether I am using http-01, dns-01 or TLS-SNI-01? I'm
using the letsencrypt nginx module.

~~~
discreditable
Seems to use tls-sni-01. This is my guess by looking at the certbot-nginx
source directory:
[https://github.com/certbot/certbot/tree/b1826d657ffc7c278041...](https://github.com/certbot/certbot/tree/b1826d657ffc7c278041cc0c8e14dc2ba23f95e3/certbot-
nginx/certbot_nginx)

~~~
mnordhoff
That's correct.

Certbot can use different plugins for validating the name and for installing
the certificate.

You can configure HTTP-01 to work and use "certbot -a webroot -i nginx -w
/path/to/whatever -d example.com -d www.example.com".

[https://community.letsencrypt.org/t/solution-client-with-
the...](https://community.letsencrypt.org/t/solution-client-with-the-
currently-selected-authenticator-does-not-support-any-combination-of-
challenges-that-will-satisfy-the-ca/49983)

------
mappu
This is not specifically a vulnerability in TLS-SNI-01 - more like a major
misconfiguration by the shared hosting provider, to share ACME secrets.

I guess TLS-SNI-01 does share part of the blame for making this possible in
the first place.

It would be great if TLS-SNI-01 could be salvaged, for all the binaries out
there - but i don't think a blacklist is a great long-term solution. How about
a strict one IP = one domain lockdown until you can prove that you don't share
challenges? i.e. negative responses.

EDIT: See reply

~~~
rgbrenner
I disagree. It's a fatal flaw in TLS-SNI-01/02\. Here's why:

All of the information required to solve the challenge is in the attackers
control. SAN-A and SAN-B (in v02) are known by the attacker. And no action or
information from the target domain is required to solve the challenge.

Every shared hosting provider and CDN that lets you pick a domain and upload a
custom certificate can be used to solve the challenge.

Your solution of restricting it to ip = one domain just reduces the vulnerable
sites to sites that don't use LE and are on a shared provider. (Also note, if
the provider does not implement LE.. then they are vulnerable to this attack
against all of the customers.. with your solution: 1 customer for each IP
owned by the service)

As long as SNI is part of the TLS standard, it is incorrect to assume IP =
domain ownership.

~~~
mappu
You're right. Thanks for clarifying.

------
rgbrenner
How does TLS-SNI-02 fix this issue?

...

Edit: I've removed most of my post here... This was already acknowledged in
the article. 02 is vulnerable also.

~~~
mdhardeman
I think the only fix is to create TLS-SNI-03 in which the only dnsName
component in the self-signed certificate is a well known child of the domain
label to be validated.

Validating www.abc.com, SNI and dnsName is well-known-acme-pki.www.abc.com and
the certificate should have some other parameter stuffed with a challenge
response that is defined to be a challenge response calculated over the inputs
of a random token provided to the requestor by the CA and the account key of
the requestor.

