
How Etsy Manages HTTPS and SSL Certificates for Custom Domains on Pattern - detaro
https://codeascraft.com/2017/01/31/how-etsy-manages-https-and-ssl-certificates-for-custom-domains-on-pattern/
======
jaas
I'd like to thank Etsy for writing this article. Let's Encrypt's availability
makes large deployments easier, but they're still not actually all that easy
to pull off in many cases. It takes a lot of thought and work to scale
securely.

We at Let's Encrypt talk to a lot of large providers about large deployments
and there's really nothing more helpful than being able to refer engineering
teams to detailed accounts of previous efforts.

These writeups really make a difference because they accelerate other large
deployments. The sooner those other large deployments happen, the sooner many
more people will experience a more secure and privacy-respecting Web.

~~~
brongondwana
This is something that's on our "very soon" roadmap for customer domains at
FastMail as well, so we're likewise very pleased to see a writeup.

------
sideproject
I've googled this subject a number of times over the years and this seems to
be the most thorough answer. Thanks for providing it!

However, for a small startup (like mine), I think it's still a bit too much
work to get this working. I know that there aren't any other "automated"
options out there anyway. Perhaps one day. :)

~~~
bowersbros
I'll be writing a blog post in the coming week and publishing it on HN
regarding how we do it; its a simpler method I think, though we aren't the
scale of Etsy yet, but should be useful for smaller startups to adopt.

~~~
no1youknowz
Thank you for this. Like the GP, I am a start-up and with my current workload,
it too is a bit too much.

Looking forward to seeing your work. Will definitely aim to provide feedback
too!

(upvoted)

------
devwastaken
>To connect all this together, our proxy hosts periodically query CertService
for a list of recently modified custom domains. Each host then 1) fetches the
new certificates from CertService, 2) writes them to disk, 3) regenerates a
config like the one above, and 4) does a graceful restart of Apache. These
restarts are staggered across our proxy pool so all but one of the hosts is
available and receiving requests from the load balancer at any give time
(fingers crossed).

How many pools are there? If you're having to restart apache every time you
renew a cert, are they renewing certs in the background and then only restart
apache when they have around a thousand renewed? If so, wouldn't the current
running certs be revoked in that time? It seems like on-the-fly cert usage is
something that could be made, if not already.

~~~
predakanga
I can't speak for Apache configurations, but on-the-fly cert lookup is
possible in nginx using the lua_nginx module[1] - IIRC that feature was
developed by Cloudflare, to power their SSL termination.

[1]: [https://github.com/openresty/lua-nginx-
module/#ssl_certifica...](https://github.com/openresty/lua-nginx-
module/#ssl_certificate_by_lua_block)

~~~
scurvy
I was also wondering about this. I would have put in the engineering effort to
migrate to nginx (at least as a front-end proxy) from Apache for this. Seems
like a lot of reloads that could be avoided (even doing graceful reloads in
Apache gives me the willies).

~~~
seanp2k2
[http://kb.linuxvirtualserver.org/wiki/IPVS](http://kb.linuxvirtualserver.org/wiki/IPVS)

This stuff is ancient but still works great.

~~~
scurvy
LVS/IPVS is OK, but what does it have to do with SSL termination and
certificate churn?

------
tomschlick
Was just talking with my boss a few weeks ago regarding how we might allow
customers to use a custom domain name in the future. Was dreading that
implementation but now it's looking a little brighter with this writeup.
Thanks Etsy!

------
cniemira
It's a good article, but it's a bit light on some of the technical points. We
tried doing something similar about a decade ago (before the widespread
adoption of SNI) and ran into the classic connected name/host header mismatch
problem. Here, it's not explained that client has to have a way to suggest
which certificate it's going to validate, and that there wasn't always an
answer for that problem.

Also curious to know more about their LB solution and how it scales.
Encrypting everything isn't free if you're doing thousands of ECDH[E]
handshakes per second to chase down that "A+" rating.

~~~
richardwhiuk
You can basically assume SNI support on all clients. Etsy is using it based on
that fragment of Apache config.

------
zhan_eg
Is't only me, or their initial reasoning ('it's pricey' and a screeenshot of a
certificate being $1500+ when you can have it for less than $10) moves the
focus of why Let's Encrypt it important - not that it's free, but that the
ACME protocol gives you the freedom to generate certificates very easy, and
have that setup process even easier for multiple (think all) services
flawless.

Congrats for them for supporting Let's Encrypt as an organization (it's not
mentioned in the article it is in the comments here).

~~~
mnordhoff
You're right, but i'm afraid to imagine how much it costs to get an enterprise
account with one of the major CAs with API access to issue hundreds of
thousands of certificates.

------
didgeoridoo
Where's the part where they donate to Let's Encrypt as a thank-you for making
all this possible?

~~~
jaas
Head of Let's Encrypt here. They did, but did not mention it in the article.

~~~
Wheaties466
Great to hear, I was thinking the same thing.

