
How We Deliver Global SSL with Let's Encrypt - jnunemaker
https://fly.io/articles/how-we-deliver-global-ssl-with-lets-encrypt/
======
tialaramex
Would greatly benefit from explaining why they couldn't do dns-01 challenges
successfully.

This an issue all the way from individuals with crappy or misconfigured DNS
locally to whole TLDs (fortunately no really major ones) with non-working DNS.
But it's often a teachable moment, something others can learn from.

~~~
mrkurt
DNS challenges weren't broken, they technically worked just fine. But they
added an additional step to our setup process, and we were losing customers
during signup because of it. We can still do the DNS challenge it's just not
how we guide people through our setup.

Basically with the DNS challenge, we had to send people away to make DNS
changes, have them come back and confirm they were approved, and then we sent
them away _again_ to make another DNS change. It was kind of brutal.

(edit: we also tweaked the article to not imply "broken"!)

~~~
zAy0LfpBZLC8mAC
Why would you need to send them away twice for DNS changes?!

~~~
matahwoosh
I'm assuming mrkurt meant that you send your users once to create a dns record
(in case you haven't already) - you can add any hostname you want on Fly, it
doesn't have to exist, yet. Then you go and create a DNS challenge for Let's
Encrypt. Obviously, these 2 steps are orthogonal, but this is the reality of
user onboarding.

~~~
zAy0LfpBZLC8mAC
But why would that need to be in two steps?

~~~
mrkurt
Well, it's two DNS entries. You could do them both at the same time, but
people were getting that TXT record wrong pretty frequently, which would have
meant https connections getting an invalid certificate had they changed their
actual hostname at the same time.

Since we control the http response once DNS changes, we don't have that
problem. And it's simpler for people to create CNAME/ALIAS records.

~~~
zAy0LfpBZLC8mAC
Erm ... you had people create TXT records? Why would you do that? Wouldn't
people then have to manually update the TXT record on each certificate renew?!

Why not have them create a DNAME, a delegation, or just two CNAMEs?

~~~
mrkurt
txt records are the only way to do the dns-01 challenge with certbot/Let's
Encrypt.

The http-01 challenge is simpler, we can get people setup with one
CNAME/A-record.

Once we're serving traffic, we can do all renewals with an http challenge and
they don't need to change DNS ever again.

~~~
pfg
This is an often overlooked option, but you can indeed solve the DNS challenge
by having someone create a CNAME record for _acme-challenge.example.com that
points to a domain under your control, and then serve the TXT record needed to
verify the challenge from that domain. acme-dns[1] provides a nice
implementation of this idea, as well as a more detailed explanation.

[1]: [https://github.com/joohoi/acme-dns](https://github.com/joohoi/acme-dns)

~~~
infogulch
Whoa really? I didn't know this option existed. This would make these types of
configurations a lot better.

~~~
zAy0LfpBZLC8mAC
Sorry, not trying to mock you, but I'm really curious how you (or anyone else
on this thread who seems to be surprised by the possibility) managed to not
know.

I mean, assuming you do know that CNAMEs are a thing, this reads to me like
"well, yes, HTTP can do redirects, but I didn't know you could also redirect
requests for images!" ...

I guess what I am wondering is: What is/was your mental model of CNAMEs that
they could redirect DNS resolution, but then it's somehow surprising that they
can redirect DNS resolution!?!

~~~
mrkurt
I don't think we'd considered a CNAME to redirect the specific txt records
that Lets Encrypt requires. It's not intuitively obvious that they support
redirects for those lookups.

~~~
zAy0LfpBZLC8mAC
But how is it not?! I mean, apart from the fact that you never know whether
software is buggy, of course.

CNAMEs are not a feature of application software, but rather a mandatory part
of the DNS specification that any standards-compliant resolver has to
implement. As they claim to be using DNS, that obviously implies the claim
that they implement what the DNS specification defines, and that would
necessarily include following CNAMEs.

~~~
infogulch
Part of it was that I'm not very familiar with the internals of the challenges
and I didn't know that `_acme-challenge` was _the_ record name for the dns-01
challenge. (I probably could have put 2+2 together if I realized this much.)

Another part was that I've never read the dns rfc and honestly I've only ever
spent enough time messing with dns until it worked. And I've never seen a
CNAME for a record that's resolves to anything but A (or AAAA or another
CNAME). So I suppose it was a combination of a failure of imagination, lack of
knowledge, and lack of experience. Or maybe, "I dunno I never thought about
it".

But now I know! CNAMEs work for all records regardless of type, forwarding the
entire lookup elsewhere, which I guess is why they require no other records
exist with that name. Pretty cool.

I suppose I could have learned that with a bit less incredulity but hey, it
got me here in the end. I guess a link to the RFC might be relevant here
[CNAME 1]. I'll probably look through it later, maybe there's some other cool
features I didn't know about.

[1]:
[https://tools.ietf.org/html/rfc1034](https://tools.ietf.org/html/rfc1034)

------
geetfun
Interesting product - fly.io

It’s essentially a reverse proxy as a service. Sort of like Cloudflare but
with what looks like a nicer API, and custom domain name SSL eg. for multi
tenant apps.

It was a bit pricy when I plugged in how many requests we get per month on our
servers, easily in 5-figure per month. Having said that, internally we built
most of their selling points already.

~~~
adrinavarro
I agree with the pricy bit. I've made a calculation, and it would cost me
1200$ a month to run what Cloudflare does for free.

Now, I get that free is not an option. 50/100$ a month is probably OK. But
that's beyond reasonable. And prices change depending on where people visit
you from (!?)

~~~
mrkurt
I'm not sure what kind of app you're running, but our product/pricing are
targeted almost entirely at SaaS apps that need multiple hostnames — which is
$2500/mo min on CF: [https://fly.io/mix/custom-
hostnames](https://fly.io/mix/custom-hostnames)

It's really expensive to do that on top of CloudFlare, and most apps like that
aren't using a tremendous amount of bandwidth.

We're not _yet_ a great option as a general purpose CDN replacement (both
pricing and featureset) but we're getting there, and are pushing prices down
continuously. It's more expensive to buy bandwidth and power in certain
cities, too, hence the different pricing per region.

If you're willing, I'd love to know what your traffic breakdown is and how you
got to that $1200/mo number. You can email me if you'd like!
(kurt.mackey@fly.io)

~~~
geetfun
Agree it is not a trivial task to build out the infrastructure for custom SSL.
We’ve done it ourselves and it’s just now another service component we have to
manage. ie. more time sink for something that isn’t our business core feature

For others who are reading this, the whole process involves reading and
storing private keys using asymmetric ciphers and all that. Definitely more
stuff than we’d like to be responsible for.

Will definitely keep an eye on fly.io for new features! Thanks

------
k__
LE is awesome.

Clients want an SSL server and it lets me set it up in 10 minutes without
additional costs.

~~~
stevenwoo
Yes this is a delight compared to the old way of having to hand over $10 or
whatever to someone for a certificate that one had to remember to renew in a
timely fashion versus just using a cron job now with LE.

~~~
mschuster91
Why use a cron job? I have certbot running in one Docker container, the .well-
known route directing to the one dedicated Docker host running certbot, and
whenever certbot renews the certificate, it writes the cert to the NAS share
that is only mounted on the LBs and then triggers a SIGKILL to the haproxy
docker containers (which causes haproxy to re-read the cert file and
configuration).

Fully automated, no cronjobs to fail. In addition, I have a monitoring on all
LBs that verifies the expiration time - it has never hit so far. That
cronjob/monitoring is something everyone should have anyway because a well
working monitoring system can alert you to so many other possible failure
states...

~~~
vog
_> I have certbot running in one Docker container, the .well-known route
directing to the one dedicated Docker host running certbot, and whenever
certbot renews the certificate, it writes the cert to the NAS share that is
only mounted on the LBs and then triggers a SIGKILL to the haproxy docker
containers (which causes haproxy to re-read the cert file and configuration)._

Is this really more stable than a simple cronjob? It is far more complicated,
for sure, so there should be significant benefits to make up for that.

------
moulidorai
On a side-note, ManageEngine Key Manager Plus can automate certificate
management (request, acquire, deploy, track and renew) for public facing
websites.

Video link:
[https://www.youtube.com/watch?v=oYelZided-E](https://www.youtube.com/watch?v=oYelZided-E)
[https://www.manageengine.com/key-manager/](https://www.manageengine.com/key-
manager/)

Disclaimer: *I work for ManageEngine

------
thiagocsf
It’s a x.509 certificate, not ssl. Also, the ssl protocol has been deprecated
in favour of tls.

I’ll let tls certificate slide but ssl certificate just triggers me too much.

~~~
vog
What's the point of this nitpicking?

The title is perfectly fine for its purpose.

First, thanks to the terminology used by almost all commercial CAs, "SSL
Certificates" is still the most commonly known term for that. Like it or not,
but if you want to reach a wide audience, you need to say SSL in addition to
TLS.

Second, even fewer people know the term X.509, insisting on that is like
insisting on saying RFC-7540 instead of HTTP/2.

Third, in the very first sentence the article sets everything straight:

 _> We've been hard at work making Let's Encrypt TLS certificates as simple
and safe as possible for developers and creators of all kinds_

So ... is this an instance of commenting before actually having had just a
tiny look at the article?

~~~
parasubvert
Because it is important that everyone understand that SSL is dead and TLS is
the replacement. SSL/TLS is forgivable, saying SSL is sort of like writing a
headline “How this company uses VHS” when you actually meant Betamax. It’s
misleading.

~~~
vog
I still don't get it. This is clarified in the very first sentence of TFA. Why
do you think it needs to be stuffed into the title?

~~~
mcny
We wouldn't need to have this conversation if the title said TLS.

~~~
nailer
You're correct, but you're fighting an uphill battle:
[https://certsimple.com/blog/ssl-or-tls](https://certsimple.com/blog/ssl-or-
tls)

~~~
stronglikedan
When talking to management, I try to avoid the acronyms (SSL, TLS, HTTPS,
etc.), and in this case, I've coined a phrase that I find be equally
understood by management, devops, and sysadmins alike. I tell them that we
need to "encrypt server X with a secure certificate" (or "secure cert"
depending who I'm talking to, which is as short as the shortest acronym
syllable-wise). It's vague enough to cover the spectrum of acronyms in a
timeless (for the foreseeable future) way, but unambiguous enough that
everyone understands.

------
nodesocket
Sorry a little off topic, but any idea when Let's Encrypt will (if ever)
support wildcard SSL certs? We provision
([https://commando.io](https://commando.io)) a subdomain for each of our
accounts (thus thousands of subdomains) using a wildcard DNS A record. Willing
to pay, as long as it is less than the $99 a year currently paying through
NameCheap.

~~~
matahwoosh
LE still says Jan '18 - [https://letsencrypt.org/upcoming-
features/](https://letsencrypt.org/upcoming-features/)

~~~
matahwoosh
for the time being, you could try to issue SAN certs with LE (from their
website):

"If you have a lot of subdomains, you may want to combine them into a single
certificate, up to a limit of 100 Names per Certificate. Combined with the
above limit, that means you can issue certificates containing up to 2,000
unique subdomains per week."

------
anubhavmishra
Completely unrelated, what is the blog framework / cms you are using the the
fly.io blog? It is amazing!

~~~
matahwoosh
Thanks! We're using ghost.org and of course all our illustrations are done by
Annie ([https://twitter.com/annieruygt](https://twitter.com/annieruygt)) :)

~~~
matahwoosh
Oh, and we use the basic Ghost theme, Casper, with some design tweaks.

~~~
anubhavmishra
Thanks for the insight! Really appreciate it!

------
hw
Clearalias.com is another service that provides SSL multi-tenant apps and
SaaSes that want to provide secured custom domains for their customers.

There's only one step required by the customer which is to point their DNS to
Clearalias, no API calls or extra set up by the app or the customers.

~~~
matahwoosh
Ohai! Yeah, you can do the same thing with Fly (which is what most of our
customers do - just point DNS at Fly). I cannot stress enough how nice it is
being able to build on top of Let's Encrypt! _bows_

The API is mainly for people who'd like to automate the process, because they
want to provide custom domains for their customers (so more for B2B use-cases)
;)

~~~
eo3x0
The grandparent post means that the customer hostnsmes also work without any
API calls if configured properly.

