
Let's Encrypt certificate issuance was down - romseb
https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/5dd157fd53c977075541890a
======
DarkmSparks
I'm going out on a limb and saying that "emergency response" is a thing of
beauty. timestamped. To the point. Examplar of how it should be done.

------
lifeisstillgood
These look like "canned" responses - one of the terror inducing parts of
managing an emergency response is, in the middle of everything, choosing the
right set of _words_ to use when telling loads of really worried people - and
it is dry easy to put it off and that lack of communication makes things bad

this _smells_ like a tickbox on an incident response webform - the responder
is just clicking "we have identified", "production database" (or network issue
or ...) and the response handler does the rest

would love to hear more

~~~
icebraining
One of the features of status.io is "Create reusable templates to streamline
your incident and maintenance updates": [https://kb.status.io/planned-
maintenance/status-update-templ...](https://kb.status.io/planned-
maintenance/status-update-templates/)

~~~
WrtCdEvrydy
Yeah, we have our 'Shit's on fire' one.... generates a message like 'We have
some unscheduled maintenance of databases to improve your customer
experience'.

~~~
9nGQluzmnq3M
That's borderline insulting. When a service is down, it does _not_ "improve
customer experience".

~~~
WrtCdEvrydy
It's triggered during times when database response time increases beyond 1
second per query.

------
d0ugie
Is there a detour here or is using letsencrypt reliant on these apis?

[https://medium.com/enigma-shards/lets-encrypt-uptime-and-
ope...](https://medium.com/enigma-shards/lets-encrypt-uptime-and-
operation-811cc1e018c0)

~~~
ownagefool
You need the API up; that's how it works.

However, should you fail to renew a cert, your script should just keep
retrying until the API is back and you should allow enough time for it to
fail. I believe most clients renew certs ~10 days in advance.

Obviously if you're relying on getting a new cert to bring up something like a
preview environment or to hand out your own subdomains, this will result in a
downtime/delay in provisioning, but most people would be fine with a single
wildcard and never really experience a problem, as long as their script runs
and they keep it reasonably up-to-date.

~~~
TimWolla
> I believe most clients renew certs ~10 days in advance.

The recommendation and the time certbot uses is 30 days.

~~~
tehbeard
I've heard 10 days before, possibly older documentation?

~~~
Arnavion
Based on the reminder emails they've sent me, they recommend renewing when
one-third of the current cert's validity is remaining. That is, 1 month for
their 3-month certs.

------
gumby
Some of these can (and should) be written ahead of time, at least as
templates.

Not only does doing so save someone from having to write them on the fly when
they're worrying about other things, they can encourage you to communicate
specific factors ("OK, we think we've found the problem, let's publish the
message to that effect now"), _remind_ you to think of certain things ("we
think we'll be up within the hour" "we don't know yet how long it will take to
fix" etc) and for that matter, if you have further failures, help someone look
back to see how the process went.

------
z92
It was down for 5 hours since when they posted the first update. And now it's
fixed.

> November 17, 2019 19:33 UTC > [Resolved] We have resolved the issue that was
> affecting production certificate issuance. Service is restored.

------
j_h_o
Isn't it WAI that the Service Disruption API is unavailable?

~~~
Thorrez
It's quite confusing, but I don't think it's a "Service Disruption API", but
instead a Service Disruption in the API.

------
kissgyorgy
A small outage should not matter at all in case of renewal, because clients
are configured to renew days or weeks before cert expiration.

~~~
GetOutOfBed
Yeah... my own setup for a hobby project of mine renew 30 days in advance but
I don't think the way it is set up will try again tomorrow if it fails today.

~~~
Filligree
I've got a systemd timer that tries to renew it once per day. It just aborts
early if the existing cert has more than 45 days left in it.

~~~
copperx
Isn't that placing a lot of load on a free service?

~~~
shawnz
With certbot, the check to see if certificates are close to expiry is done
offline. So the service is only hit if a renewal is needed. The developers of
certbot actually recommend that you schedule the cron twice a day.

EDIT: Actually I'm wrong. It also checks if the certificate was revoked via
OCSP. However I can't imagine that it consumes much resources.

~~~
schoen
They're also different resources, and cacheable ones.

------
Animats
I stopped using Let's Encrypt when they broke Apache servers not using virtual
hosting.

~~~
wolf550e
1\. The certbot client is not a product of the Let's Encrypt CA and is not
mandatory. You can use another client.

2\. With the certbot client, instead of using a deep integration with your
http server, you can use `certbot certonly --webroot --webroot-path
/var/www/html/.well-known/ -d example.com` (or similar) and provided you told
your web server to serve file from the /var/www/html/.well-known/ filesystem
directory under the /.well-known/ url path prefix on your domain, cert
issuance and renewal will work. This should be easy to do with any web server.
certbot will not know which web server you happen to use and will not care.
You can replace the web server and renewal will keep working.

