
Letsencrypt, the Good, the Bad and the Ugly - jacquesm
https://jacquesmattheij.com/letsencrypt-good-bad-ugly/
======
lub
I think the author is confusing three things:

* Let's Encrypt - the actual CA

* certbot - probably the mentioned python software

* ACME - the protocol, which had a change in the supported challenges

Also there are alternatives for all three points. There are other CAs
implementing ACME, there are other ACME clients and there are still CAs
supporting the old 'send an email and click a link' domain validation.

~~~
jacquesm
If you run something that issues 100 million or more certificates per year
then backwards compatibility is not something that you toss out just because
you can. Forcing that many web properties to upgrade their software
(regardless of which party produced what) is discounting the combined effort
that will take on the part of the users/sysadmins of those systems for
something that could have just as easily been avoided.

You deprecate interfaces like these but you don't just shut them down,
especially not when they are still seeing major use.

Just imagine that tomorrow IPV4 would be shut down because we've all had
enough time to switch by now.

~~~
b3lvedere
>Just imagine that tomorrow IPV4 would be shut down because we've all had
enough time to switch by now.

Honestly? I would absolutely love to watch that shitshow.

~~~
jacquesm
Can you please wait until I'm past my 'best before' date when you pull that
particular plug?

------
ohazi
I can sympathize with the frustration, but I disagree with the criticism.
Certbot is capable of doing dry-runs, and is configured by default to try to
renew with a _month_ to spare. It logs failures. You should read your logs.

If you've built a system where silently failing is a problem, and you haven't
configured any sort of "email me when anything isn't nominal" then you haven't
finished configuring your server.

Software updates happen. Unattended updates are a thing. It's not the ninties
anymore... You can't hook something up to the internet, chuck it in your
closet, forget about it for five years, and expect zero issues. You need to
pay attention to it.

~~~
logicchains
>You can't hook something up to the internet, chuck it in your closet, forget
about it for five years, and expect zero issues. You need to pay attention to
it.

This kind of contempt for users is why I try to avoid using any
libraries/software written in Python. Somebody always thinks their minor
backwards incompatible change is such a precious snowflake that it justifies
wasting collectively thousands of hours of users' time debugging and updating
their stuff to work around it.

~~~
RandoHolmes
That's not a Python specific problem, but otherwise I agree with your point.

~~~
paul_f
No, but when it occurs, it's usually Python

------
rnijveld
I'd start by ignoring Letsencrypt's advice to start with Certbot (which I
think was used here given the description). Instead look for something like
lego[1] or even simpler clients, something like Acme.sh[2] or acme-tiny[3]. At
least for me certbot has way to many moving parts and makes the acme protocol
feel more complicated than it is.

[1]: [https://github.com/go-acme/lego](https://github.com/go-acme/lego) [2]:
[https://github.com/acmesh-official/acme.sh](https://github.com/acmesh-
official/acme.sh) [3]: [https://github.com/diafygi/acme-
tiny](https://github.com/diafygi/acme-tiny)

~~~
corford
Couldn't agree more. For a person that runs/admins their own DNS and
mailserver infra, I'm surprised they didn't immediately gravitate towards one
of the lightweight clients.

Been a while since I last needed to get involved with letsencrypt stuff but
I've always used and liked: [https://github.com/acmesh-
official/acme.sh](https://github.com/acmesh-official/acme.sh)

Ansible's letsencrypt module was also not bad last time I used it:
[https://docs.ansible.com/ansible/latest/modules/acme_certifi...](https://docs.ansible.com/ansible/latest/modules/acme_certificate_module.html#acme-
certificate-module)

~~~
jacquesm
Mailinabox comes with its own pre-installed client that apparently does not
upgrade unless you upgrade the whole machine. The upgrade instructions for
that particular operation start with: take a newly installed Unbuntu machine.
Which isn't always an option.

~~~
gertrunde
In the mailinabox case - I suspect this refers to the jump from v0.3x to
v0.4x? Which was the point that they moved to Ubuntu 18.04, I think somewhere
around the beginning of 2019?

Admittedly I'm somewhat slack about maintenance on my own mailinabox system,
so it took me ~12 months to get around to going through the upgrade process,
which was pretty painless.

It was just a case of backup/re-image/setup/restore on the same machine, took
maybe an hour?

(But yes, it would have been nicer to have a better rollback point than that).

~~~
jacquesm
That change happened about a week after I installed it the first time, just
after migrating many (way too many) imap stores to it. I figured it should be
good for a while at least. My mistake I guess. Or maybe two years counts as
enough? Wonder what the lower border is for an operation like that.

------
jarym
I sympathise with the sentiment of this post and admit things can and should
be better. But there are a few points to be made:

> the situation hasn’t changed and if it worked last time it should still work
> this time

The situation has changed though - who knows what vulnerabilities had to be
addressed in any of the dependencies or the protocol. Just because you forgot
about it doesn’t mean others did.

I’m not a fan of automated software updates and believe they should be
restricted to just essential security patches (where the risk of a
vulnerability is greater than the potential to break a system)

> You would not have known this unless you interacted with Letsencrypt on a
> daily basis, but like most of these things their whole value lies in NOT
> having to interact with them all the time.

There’s plenty of stuff that we interact with on an infrequent enough basis
that we can get caught out when something changes and we realise we didn’t
know and maybe had little way to know.

This is just impossible to avoid in the real world. The only way to address it
is better communications.

I generate my LetsEncrypt certs in docker containers mainly because of all the
dependencies and that it’s clear it’s still evolving. Maybe in 10 years time
it’ll be as mature as a nix shell command like ‘ls’.

~~~
jacquesm
> who knows what vulnerabilities had to be addressed in any of the
> dependencies or the protocol.

Well, if there were unsolved vulnerabilities in the V1 of that protocol I'd
like to know about them, but so far all I have seen are vulnerabilities that
were typically solved within days at worst.

------
raziel2p
> Somewhere someone has decided that Letsencrypt should auto-update even if
> absolutely nothing had been changed on the system that it was running on

Letsencrypt is not a piece of software you install on your systems. You're
probably thinking of certbot, which is just one of many available ACME
(Letsencrypt-compatible) clients that can manage your TLS certificates. I've
used dehydrated and acme-tiny in the past, both work well.

The article is also devoid of any details on what the weird technical issues
you ran into that make you think you need a completely new machine. For all I
know the machine was running a 10 year old version of OpenSSL.

I assume the change in the challenge protocol being referred to is ACME v1 to
ACME v2? Why not just say this outright?

~~~
jacquesm
A two year old install of mail-in-a-box. I am not in the habit of reinstalling
my main mail and DNS server just for kicks, it is a lot of work to put it
mildly, and very easy to mess up.

> I assume the change in the challenge protocol being referred to is ACME v1
> to ACME v2? Why not just say this outright?

Because this has now happened twice, for entirely different reasons.

The first time letsencrypt-auto totally hosed a machine, the second time mail-
in-a-box (which was otherwise humming along just fine) spat out an
incomprehensible error about it not having enough authority to request a new
certificate even though nothing whatsoever had changed on that machine.

~~~
hyder_m29
Without any details, it's uncertain what the real issue was. It could very
well have been attributed to a misconfiguration on your machine.

I manage over 20 client domains over a dozen servers. Have letsencrypt
protecting them all. The only time I needed to manually intervene after the
initial domain + SSL setup was to upgrade to ACME v2. Everything generally
works out of the box with Letsencrypt + Certbot.

~~~
jacquesm
Good, so this hasn't bitten you. I'm sincerely happy for you. But my
experience so far has been pretty rotten.

The real issue was a deprecated protocol, one that - as far as I know - could
have been kept running at zero cost to LE because it already worked just fine.

~~~
PaulKeeble
Likewise I have had issues 3 times. In none of those cases did I really
understand what went wrong, the fix was blow it away and redo it. Something
just gets messed up the answer the community comes to is just start again. Its
hard to answer the question of what went wrong when the error message is often
garbage and gives no clue and just says the certificate can't be updated and
there is no information on what just got broken.

The answer for me each time has been the age old turn it off and on again with
redoing the install (or changing the way I utilised it in docker completely)
and I have just found the whole thing more hassle than SSL certificates ever
were.

~~~
jacquesm
The responses here are an interesting mix between those who have not had it
happen to them (yet) and those who have (or have had it multiple times).

The former can't imagine the latter are anywhere near competent, it could
obviously never happen to them so the person experiencing the problem must
somehow be at fault.

It's the message board equivalent of 'can't reproduce', closed.

~~~
hyder_m29
My comment was not accusation of incompetence, rather the lack of details
provided. It's like receiving a ticket saying "This doesn't work. Fix it now"

------
tialaramex
> The basic idea is that certificates should be free, that the barrier to
> install them should be as low as possible and that updating certificates
> should be automated.

This is the wrong order to think about it, and that's a little bit important
or I wouldn't mention it.

They want to drive automation and _robots don 't have wallets_. The Let's
Encrypt service has to be free at point of use because if it costs say $1 per
certificate then automating it becomes much trickier as you need to give the
automation a supply of money to buy certificates and now you're solving a hard
problem you didn't need to have.

If the first goal was to make it very cheap, you can see how to get to say $1
certificates pretty easily from where we were already, and since name owners
are likely spending a few bucks per year on the domain name that probably
counts as success. But once you set the goal as automation a $1 certificate is
only slightly better than a $100 certificate, because adding some sort of
payment mechanism for an automated service is a big problem.

Now, being free at the point of use is also good for other reasons, but I
think it'd be a harder sell without the technical argument.

~~~
jacquesm
The technical argument is what sold me, not the price point.

------
dsr_
If Jacques has been running the same system for five years, he probably missed
five Nginx CVEs and 200+ kernel CVEs. Some of those required changes that
ended up affecting the behavior of the system.

Certbot is not a replacement for a sysadmin.

DevOps is not a replacement for a sysadmin.

DevSecOps is not a replacement for a security engineer.

DevSecNetOps is not a replacement for a network engineer.

You need to have someone with the skills who applies those skills to your
systems, or else you are hosed. This has always been true and is likely to
always be true.

The good news is that cooperative projects make it easier. Let's Encrypt, the
Debian project, the Google security hole finders, the whole CVE infrastructure
-- the people working on these add an awful lot of value to the world.

~~~
jacquesm
That system is about two years old.

As for the rest of your comment, you are somewhat underestimating what I can
do but I like to automate away that sort of work because there are more
important things in life.

And I expect things, once properly automated, to continue to function unless
_I_ mess with them. These cascading failures caused by some policy change
elsewhere culminating in millions of installations having to go through
sometimes costly upgrades are not the future I had in mind when I started my
career in IT.

~~~
dsr_
I don't underestimate what you can do -- I saw the LEGO sorter. That's nothing
short of amazing.

But I am surprised at the attitude of "this automated system will continue to
function". It's never been true in connected systems that care about security,
and it's only true for limited timelines for independent systems. In the long
run, everything decays.

~~~
jacquesm
But in this particular case there was no good reason why it came out the way
it did. And that's what bothers me more than anything about it, that such a
popular package would force users to expend a lot of effort for zero gain.
It's just make-work.

------
TonyTrapp
From the very beginning I was sold by Let's Encrypt's concept but I loathed to
install their bloated client. Instead I have been using acme-tiny [0] from the
very beginning, and it saved me from all the frustration.

[0] [https://github.com/diafygi/acme-tiny](https://github.com/diafygi/acme-
tiny)

~~~
chmike
Same here. Note that acme-tiny requires to write scripts to manage the
certificate update and reload the services to take in account the new
certificates.

------
smasher164
I've been using autocert (golang.org/x/crypto/acme/autocert) to fetch and
renew certificates. A few weeks ago, I added new subdomains to my site, and
noticed that it wasn't being issued new certificates.

After searching for the error in the logs, I landed here:
[https://community.letsencrypt.org/t/end-of-life-plan-for-
acm...](https://community.letsencrypt.org/t/end-of-life-plan-for-
acmev1/88430). I missed the whole ACMEv1 deprecation notice. I thought
autocert would take care of this (and it does), except I had not updated to
the latest version of the library. Everything was resolved after updating the
library, but I still had to clear my local HSTS to verify. This would not have
been acceptable if there were actual traffic going to that subdomain.

I guess the lesson to learn is that even though many of the barriers to
automatic certificate renewal have been removed, I still have to keep up to
date on breaking changes and deprecation notices, especially if updates only
happen every couple of years.

~~~
paxys
The real lesson is to always have automated monitoring and alerting in place
for everything. Certificate renewal, API schemas, service dependencies, core
libraries and protocols - everything can and does change without notice. You
could have a single dependency on www.google.com and it would still eventually
break.

------
brazzy
In case anyone is left wondering amout what specifically has changed and
whether they might also have problems manifest suddenly:

[https://letsencrypt.org/docs/acme-protocol-
updates/](https://letsencrypt.org/docs/acme-protocol-updates/)

[https://community.letsencrypt.org/t/end-of-life-plan-for-
acm...](https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430)

So this has been planned for quite some time, and if you have a current
Letsencrypt client, it should keep working seamlessly.

But yeah, if you have servers running an old OS release that won't get a new
certbot version, that's gonna suck.

~~~
jacquesm
Why force people to upgrade if it isn't necessary? Was there a huge price on
the side of LE to keep the old protocol up and running until it was no longer
in use? Is that price balanced out by the combined cost of the users having to
upgrade their systems, something that typically costs a day or more per
installation?

~~~
brazzy
Reading between the lines, it seems the v1 protocol was basically a proof of
concept implementation and differs in some points from the final RFC
specification.

With Letsencrypt being by far the most popular ACME provider, if they kept
running the old version even optionally, there would be very little incentive
for clients to start supporting the standard instead of the POC.

I can understand how they don't want to be the IE6 of CAs.

~~~
jacquesm
Unless there was an incurable security issue in the V1 protocol I see no
reason to force an upgrade on the part of untold millions of installations.

------
boudin
I can't disagree more with the author of the post for the bad part. I manage
quite a few things with letsencrypt, some having been set years ago. Each time
I needed to take action I received an email from letsencrypt. I think they
actually do this really well. If you setup things in ways that you get no
feedback, you'll have bad surprises but that's the same with absolutely
everything in life.

The only difference with what was before letsencrypt is that certificates have
a relatively short life, so a problem can appear sooner than it would before
and it's automated. Like everything automated, it involve tools, infra and
maintenance.

------
jwr
Very true.

Generalizing: many projects would benefit from taking a more user-centric view
of the world. Your project is not the center of the universe. It is not the
main life focus for your users. Your users have things to do and places to be,
and your project is just a tiny portion of their entire life. Take that into
account when making decisions.

~~~
brazzy
The problem is that Letsencrypt is, while useful and important (and _free_ ,
let's not forget that!), one of those "if it works perfectly, you won't notice
anything at all" things. Users don't need to interact actively or regularly
with it _at all_ , that's one of its main value proposition.

So how exactly _are_ they supposed to handle a breaking change? Just never
have any breaking changes, ever? What if a breaking change is necessary for
security reasons, with security being, after all, _the_ primary reason for the
project to exist at all?

~~~
jjav
> Just never have any breaking changes, ever?

Absolutely, that is the correct answer.

If a breaking change is absolutely needed that implies a new major release
number, but never, ever, break compatibility in the current major release
family.

~~~
brazzy
Absolutely not, that is the wrong answer.

If a security-critical issue cannot be fixed without a breaking change, you
make that breaking change. Anything else is irresponsible, as is the desire of
users to keep running insecure systems to avoid the effort of updating them.

------
selectnull
I've been using Letsencrypt since the beta and I enjoy it for the simple
reason of not having to get private keys via email (remember the times before
LE?).

So far, I had exactly one frustrating moment with LE (or was it certbot?).
I've created AAAA records for a domain, but didn't set the nginx up to accept
IPv6 connections. Certbot failed to fetch the certificate with some strange
message. I figured it out. It was not that bad.

Letsencrypt is a blessing.

~~~
fanf2
If you were sending or receiving private keys via email then you were doing it
wrong. CSRs and certificates do not contain private keys.

~~~
selectnull
I was not sending them. I was receiving them as several SSL providers send
them zippped via email.

------
olavgg
I struggle with the same thing. LetsEncrypt/Certbot was the solution to
maintenance free SSL certificates. Well it's been a bumpy ride so far. Both my
FreeBSD and Linux servers has had issues with CertBot and some of the third
party softwares, like acme.sh. Like 3-4 times a year I need to spend time to
debug what has gone wrong. A $10 yearly certificate quickly becomes a no
brainer.

------
ianbooker
I agree with some of the comments, title should be not letsencrypt but
certbot, the provided implementation of certifying your server.

At least for Ubuntu 20.04, Letsencrypt does not require you to install the
Python stack for certbot anymore and switched to a Snapd package. Although at
least I had to install Snapd for that matter.

~~~
jacquesm
But that would require upgrading to Ubuntu 20.04, which is a major operation
in its own right. And it's not as if snap is without problems of its own.

~~~
dcomp
For the record, snapd is available all the way back to 14.04 [1]

I use acme.sh with dns, it just plonks a certificate in a directory of your
choice and runs a reload command of your choice. I actually upload the
certificate into a kubernetes cluster. All repeated by cron

[1] [https://snapcraft.io/docs/installing-snap-on-
ubuntu](https://snapcraft.io/docs/installing-snap-on-ubuntu)

------
charliesome
A few people have already suggested alternatives to certbot, so I thought I'd
throw another suggestion into the mix: dehydrated (formerly known as
letsencrypt.sh)

[https://github.com/dehydrated-io/dehydrated](https://github.com/dehydrated-
io/dehydrated)

I've got only good things to say about it. It's a single shell script, making
it super easy to install and start using. It's quite configurable, but has
sensible defaults and just works without demanding much operator attention.

~~~
crumbshot
I second your recommendation. After installing it on some of my now long-
running servers back in 2016, I've only touched it a couple of times since.
It's been working seamlessly since then.

Configuration was easy, I just ended up writing a simple shell script for the
deploy challenge hook, to copy the certificates and reload configurations on
nginx etc.

The two times I chose to intervene were straightforward too. The first was
because of the ACME protocol upgrade, for which I just needed to drop in a
newer copy of dehydrated, and rename references to its old name. The second
was because I needed to configure the ACME DNS challenge to get a wildcard
domain, and that just involved modifying the deploy challenge hook to
temporarily run a custom DNS server that responded to the challenge.

I did take a look at certbot, or whatever it was called back then, and decided
against it quite quickly due to the perceived complexity. Also I seem to
recall it didn't support nginx properly.

After reading this user review, I'm quite glad I did that. Though I do wonder
why the author persisted in using that client despite all the problems they
were having with it, given the choices available in the wider Let's Encrypt
ecosystem.

------
akerro
I had the same experience with LE autoupdate. It's been working on a few years
on my dedicated server and one day I woke up and all my certs were expired. LE
failed on autoupdate and was failing to renew certs. It took me approximately
2hours to get it working. Had to change DNS TXT entries, get new ACME accounts
as old accounts was creating new directories mydomain.com-0001 and
mydomain.com-0002, which would mean I would have to change config in postfix,
dovecot, nginx and more...

------
ht85
Given that OP is a self-proclaimed "update hater", I guess the reason they
don't get a 5 year commercial cert is cost? In that case whining online about
it seems uncalled for.

Also from an architecture standpoint, the post seems to suggest that OP has
certbot running on a variety of machines throughout his infrastructure. If you
have many servers relying on letsencrypt, you probably want to centralize
generating certificates somewhere, and have other machines pull from there.

~~~
SyneRyder
_> I guess the reason they don't get a 5 year commercial cert is cost?_

Web browsers no longer support 5 year certificates. The new maximum is 1 year
as of last week, with a view to make that duration even shorter over time as
certificate renewal & automated updates become more widespread.

[https://nakedsecurity.sophos.com/2020/02/24/apple-chops-
safa...](https://nakedsecurity.sophos.com/2020/02/24/apple-chops-safaris-tls-
certificate-validity-down-to-one-year/)

~~~
ht85
Thanks, wasn't aware of that, very interesting!

------
Crosseye_Jack
Personally I’ve been using acme.sh over certbot for a while now. Saves
headaches about python dependence.

[https://github.com/acmesh-official/acme.sh](https://github.com/acmesh-
official/acme.sh)

~~~
brazzy
All my personal servers are either directly provided by or proxied through an
instance of Caddy ([https://caddyserver.com/](https://caddyserver.com/)),
which has an Letsencrypt client built in.

~~~
Crosseye_Jack
I’m using Cloudfront, nginx and Apache atm between all the diff servers
(should really clean up my stuff) as well as postfix for my mail. So while it
won’t fix my needs I will keep it in mind for when I need to throw together a
server for others.

------
csomar
If you are looking for ease of use and stability, there is an alternative
solution. Issue self-signed certificates on your server, and then use a
service like CloudFlare that handles the front-user certificates. The whole
traffic is encrypted now, but you don't need a CA authority to deal with.

The people who are benefiting from Let's Encrypt are the least technical who
can't even SSH into a server. They get the certificate from their hosting
provider automagically. It's reasonable that LE and the people writing
software for it will focus on them rather than another developer doing a
manual integration.

~~~
bigiain
> then use a service like CloudFlare that

... MITMs all your user's traffic through a company that's made questionable
ethical decisions in the past.

:sigh:

~~~
maple3142
Aren't all CDNs MITM?

------
rlaanemets
Just let it get the cert (DNS or HTTP challenge) and then restart/reload the
web server. Do not let LE rewrite your server configuration. This was very
error-prone before and likely still is. People structure their Apache, Nginx,
HAProxy, etc. configuration files in very different ways and it's nearly
impossible for the certbot to work correctly in all cases. I'm not sure if
this was also the case for the article author, the article contains too little
technical information to tell it.

~~~
jacquesm
That's the one thing that actually worked once I managed to get a chrooted
copy of the latest certbot running on a copy of the config. But that took
quite a bit of fiddling to set up and it should never have come to that in the
first place.

------
edumucelli
One more than "good", actually phenomenal thing about LetsEncrypt from my
experience is their community. It is pure gold. People are super helpful in
the [https://community.letsencrypt.org/](https://community.letsencrypt.org/)
forum. I had issues on setting it up and lots of people got involved and
helped me out, the responsiveness, the cordiality, that is high quality
"support" that no money can pay.

------
lenlorijn
I usually enjoy articles by Jacques, but this one has a different tone and I
don't think I understand its point.

Saying things like

> hey should just shut up, sit in a corner and do their bloody job. seems
> unnecessarily blunt.

And if you don't like Let's Encrypt, don't use it. It's being compared to
convenient paid services, so why not use those instead if they end up being
more convenient for your usecase?

~~~
jacquesm
I don't like the 'if you don't like free service X that you've been using for
Y years then don't use it'. There is a cost to switch, and that cost to switch
alone outweighs _all_ the benefit of using LE in the first place. That's not
what I was sold when I went into it, and even 'free' users are entitled to
being treated with some care.

On the assumption that you are using a free gmail account: would you agree
that Google could shut down the free tier of gmail tomorrow unless we all move
to Chrome and it would be our problem to deal with the consequences?

~~~
notriddle
> On the assumption that you are using a free gmail account: would you agree
> that Google could shut down the free tier of gmail tomorrow unless we all
> move to Chrome and it would be our problem to deal with the consequences?

Yes.

------
codegladiator
> This is unacceptable for a project that tries to make something easier and
> where convenience, not financial considerations are a big driver in
> adoption.

Financial considerations _are_ the big driver for letsencrypt. Not sure why OP
think otherwise ? Easier and convenience is secondary and served by
certbot/acme.

------
mschuster91
> Somewhere someone has decided that Letsencrypt should auto-update even if
> absolutely nothing had been changed on the system that it was running on.

This is why I always run certbot inside of a docker container.

> The machine that this is all running on will need a much more recent version
> of its OS.

Again, same thing: use Certbot inside Docker.

~~~
reflectiv
Exactly.

[https://hub.docker.com/r/certbot/certbot](https://hub.docker.com/r/certbot/certbot)

certbot has an official docker image. By using the official docker image you
don't manage the updates beyond having to call `docker pull certbot/certbot`.
I just integrate that into the same cron task that triggers the
renewals/checks. Could not be easier and more fault tolerant.

------
dqv
Oh. I thought this was a Python library issues and had nothing to do with LE.
I too have had two occasions where I couldn't get certs with MIAB after
upgrading. The first was some weird library that was missing after upgrading
from like 10 versions behind on MIAB. IIRC it was a core Python library that
had changed. The other was something that had to do with there being multiple
"instances" of certificate chains.

Basically all it made me do is make sure I was monitoring the administrator@
account for the primary MIAB domain to make sure I'm updating it regularly.

Also I think MIAB is one of the few pieces of software that can safely be
upgraded regularly without pain. I really trust Dr. Tauberer's project.

------
Diederich
To add to the various anecdotes already shared: I've been running this for
several years now with zero attention:

    
    
      root@eighteendotfour:~# crontab -l|grep getssl
      23  5 * * * /root/bin/getssl -u -a  >> /root/bin/getssl.log 2>&1
      root@eighteendotfour:~#
    

I've not been watching the logs because it's just for my private web site, so
no biggie if it has problems.

    
    
      root@eighteendotfour:~# head -3 /root/bin/getssl
      #!/usr/bin/env bash
      # ---------------------------------------------------------------------------
      # getssl - Obtain SSL certificates from the letsencrypt.org ACME server

------
nix23
Two days ago i had a big problem with haproxy..well letsencrypt to be precise,
getting a new cert was not possible but renew one was. It was the additional
Cipher-suite setting in Haproxy ;(

------
brentis
It's a false promise for sure. Overly fragile and can bring down a system with
little warning. Have 20 instances and if it fails 9 out of 10 times it's from
LE.

------
TekMol
For me, the biggest pain point with Letsencrypt is that they do not offer any
easy to implement way for wildcard certificates.

~~~
ht85
They do though, as long as you can use the DNS-01 challenge.

~~~
TekMol
Which is not easy.

------
ed25519FUUU
> _The machine that this is all running on will need a much more recent
> version of its OS._

Does anyone have any info on this problem? I have letsencrypt running just
about everywhere (including my ubiquity edge router). It seems to "just work".
However, I use the letsencrypt-nginx plugin to handle the certificates for me.

------
icedchai
I have run into many of these same issues. Certbot auto upgrade issues, v1
protocol deprecation, and time wasted fixing it.

Still, Letsencrypt is an amazing project. I would never want to go back the
old way of buying / renewing / installing certificates. That was a true pain!

------
daneel_w
I highly recommend [https://github.com/acmesh-
official/acme.sh](https://github.com/acmesh-official/acme.sh) to solve the
annoyance of the terribly bloated and poorly built Certbot application.

------
hannofcart
Am not a sysadmin. However after struggling with LE for a while (esp. failed
renewals) I just got an origin CA certificate from Cloudflare and stuck it in
Nginx.

Haven't had to think about this again for years now.

------
jeroenhd
I've never had any of these issues, so I'm surprised by how spectacularly the
author's setup broke. I can understand some Python library failing during the
auto update process needing a rollback, but needing a complete reimage to fix
the situation? How does that even happen?

> new protocol for the letsencrypt challenges was released, announced in some
> obscure corner of the web and bit by bit the old method of issuing new
> certificates was deprecated. You would not have known this unless you
> interacted with Letsencrypt on a daily basis, but like most of these things
> their whole value lies in NOT having to interact with them all the time.

Let's Encrypt also sent out emails to people still using the old method at
least two months before the old method was phased out. You have to provide an
email address when setting up LE exactly for this reason. The only explanation
I can think of for why the author wouldn't have received an email is that they
did not execute a renew check for their certificate in the three months
leading up to the change of service, which is going straight against LE's
guidelines (you should check more often, exactly for this reason). This is the

> which is odd because after all _the situation hasn’t changed and if it
> worked last time it should still work this time_

The situation obviously _has changed_ otherwise the venv wouldn't have been
updated. Libraries get bugfixes even if those don't apply to you specifically;
deal with it. You're not the only customer here.

> I understand that Letsencrypt needs to stay with the times. But the
> Letsencrypt folks should also understand that in the greater scheme of
> things certificates are not the most important thing in life and that
> causing your end users this level of grief is in the longer term not in the
> projects’ best interest.

Web security is not a "set and forget" thing. It's not a toggle button from
"insecure" to "secure". If you want a zero-maintenance certificate solution,
get a managed server and get someone else to handle the annoying stuff for
you.

> The machine that this is all running on will need a much more recent version
> of its OS. That means I’m going to have to migrate a whole bunch of data to
> another machine in order to be able to do this without downtime.

This is unsurprising coming from someone who wrote a blog about why they don't
like to update. However, the assumption that the entire host OS needs to
update just to use a new piece of software is simply wrong. Things like Docker
and LXC make it trivial to run newer software without upgrading the entire
system. It also forces the software into a contained space to prevent the
catastrophic failure the author had to deal with before. From what I can tell
from a Shodan check for this domain, the author still seems to be running
Ubuntu 14 (which went EOL last year). That probably explains why the author is
afraid that they will need to upgrade their server. However, if you're running
on a software stack whose LTS support has already been dropped, you're just
postponing the inevitable by not planning an upgrade plan already. Blaming
Let's Encrypt for potentially not supporting a platform nobody else supports
is a silly excuse.

I agree that Let's Encrypt has its issues. Sure, their client can use some
thinning, their decisions are sometimes confusing, but the normal situation is
not nearly as bad as the author describes here. If you're bothered by the
official client updating things, use a different client; there's plenty of
them.

~~~
jacquesm
The machine was installed two years ago with what was _then_ the current
version of Mailinabox, which in turn apparently relied on Ubuntu 14.04.

So, yes, that is outdated now but it wasn't as if there was much choice at the
time and two years seems a bit short - maybe I'm wrong about that - on the
expected lifespan of a server. But setting up a mailserver, migrating all of
the data back on and then testing it is a boatload of work and I have plenty
to do as it is.

And yes, I did eventually manage to get it to work but in a pretty roundabout
way and that should just simply not have been necessary unless there was a
huge security issue with the way LE was working before. I'm not aware of any
such issues.

Essentially you are arguing that something like Mailinabox can not or should
not exist because it falls somewhere between 'managed hosting' and something
that can be set up for a few years in order to serve a purpose.

That's not a very encouraging future, maintaining basic internet infra such as
mail, web and DNS servers should not be so hard that you can only do it if you
want to pay third parties big bucks to do so.

~~~
jeroenhd
Mailinabox's use of such an old OS is the exact reason I skipped it and went
for Mailcow instead. It also placed very strange limitations on the stuff you
could do with the system once it was set up and running according to the
documentation, presumably to make sure the project maintainers had only a few
types of configurations to support.

I have nothing against the concept of Mailinabox but when you create a "hands-
off" project like that, your project should also include stable major version
upgrades to LTS versions. From what I can tell, MIAB switched from 14.04
directly to 18.04 (skipping 16.04). Their goals and ideas are noble, but I
don't think the project is very reliable if they skip major OS releases just
like that (and are very late to the party when it comes to support for newer
versions, releasing a version compatible with 18.04 only months before 14.04
support was dropped entirely).

I appreciate the enormous amount of work it takes to achieve something like
this, but marketing your product as something "you don’t need to be a
technology expert to set up" brings with it certain expectations (or even,
responsibilities). Your issue with an outdated LE client should have never
happened if the project had achieved its goals, for example.

A modern setup with MIAB based on Ubuntu 18.04 will have OS support until
2028, so the situation has improved; Ubuntu has switched to 10 years of
support instead of 5.

> Essentially you are arguing that something like Mailinabox can not or should
> not exist because it falls somewhere between 'managed hosting' and something
> that can be set up for a few years in order to serve a purpose.

The problem is that software doesn't work like that, unfortunately. Bugs and
security issues are found and need to be patched. Volunteers don't want to fix
five years of releases of their software and for companies it doesn't make
financial sense to do such work for free. A Windows server box with regular
reboots for security updates will work fine for a bunch of years, but that
guarantee is hard to make when your software runs on volunteers and occasional
corporate sponsors.

> That's not a very encouraging future, maintaining basic internet infra such
> as mail, web and DNS servers should not be so hard that you can only do it
> if you want to pay third parties big bucks to do so.

There's a reason companies have been providing paid services for this since
the dawn of the web. Hosting software on the internet isn't maintenance free
and it never was. If you desire a setup that keeps working with minimum
maintenance without the risks that come with volunteer work, you'll have to
pick a stable OS (like Ubuntu or its competitors) and software stack from the
LTS repositories and configure it. You can apt-upgrade anything from the
official Debian repositories without reconfiguring anything. You can upgrade
and It Just Works as long as you stick to the packages provided by the people
maintaining your OS. This works until your OS reaches end-of-life, which is
the next time you should do intensive system admin effort. Luckily, this is
announced long beforehand, so you can plan accordingly.

This still means you'll run into trouble when (free) external parties decide
to drop your OS because of the growing amount of software stacks and
configurations they have to support over the years. However, if you practice
running external stuff in a contained environment (with tools like Docker)
you'll get to keep the best of both worlds for as long as possible.
Alternatively, you can use setups that don't hide behind magical one-command-
fixes-all setups, like manually entering certificates instead of relying on
Some Cloud System to do it for you.

If all that is too much for you, which I can completely understand, then maybe
running your own servers might not be for you. The premise of bringing self-
hosted, federated services to the every-day consumer is nice, but I'm afraid
it'll always be little more than a dream. The modern web is either part of the
Cloud (TM) or a bunch of projects glued together by random chance and best
effort; running your own stuff unfortunately requires either money to be paid
or effort to be made.

~~~
jacquesm
I've self hosted web/mail/dns since roughly the beginning of the www and I
find it quite frustrating that all these so called improvements actually make
it only harder and harder to keep this up. Besides Google putting ever more
pressure on independently operated mailservers to join the borg by randomly
rejecting perfectly valid email.

I will also freely admit to having found that there are more important things
in life than endless fiddling with system configurations and low level stuff.
It used to fascinate me but the days when installing an operating system on
some fresh piece of hardware (or a virtual) excited me are long gone. When
you've done it 100's of times it is no longer as nice as when you did it the
first few times.

------
perryizgr8
The big fact of the matter is that https is usually unnecessary for most
websites. So this entire effort is a massive waste. Why do I need https on my
blog? What is the great risk I am exposing myself to by hosting a plain html
page? Nothing.

This is basically Google bullying the internet for no reason. So of course
lets encrypt will demand attention. It is the central artifact of your website
now, since you took the time to set it up without any use or benefit. So deal
with it.

------
mrunkel
Considering the entire post is whining about a free service, I don't get it.

Who is forcing you to use letsencrypt? If you don't like it, buy a cert and
install it.

If the certbot is too complicated or "pulls in too much cruft" then use a
different client. [https://github.com/acmesh-
official/acme.sh](https://github.com/acmesh-official/acme.sh) for example.

~~~
nix23
Ahh i see...the old black and white worldview.

------
varbhat
[https://acme.sh](https://acme.sh)

------
acp2020
It's 2020 and it's time for you to learn about containers and docker.

------
elcapitan
Can people please stop to use "The Good, the Bad and the Ugly" every time they
write some list of thoughts about some topic? It's maybe a little less
creative and unique than you think.

------
fmnxl
This is what Nix/NixOS would solve..

------
andrewshadura
You should have used certbot shipped by your Linux distribution.

------
vbezhenar
I can confirm this experience. Back in the day, when startssl or wosign
worked, I spent few minutes per year to issue a certificate and called it a
day. I think that wosign even issued 3-year certificate, but I might be wrong
about it. With letsencrypt I spent hours after hours trying to understand how
does it work and set it up properly.

But moaning does not do any good, we're not getting back in those good old
days, so it's better to adapt, I guess.

One thing that helps with broken letsencrypt setup is to monitor e-mail that
was supplied. Letsencrypt will send angry mails when your certificate is not
updated.

~~~
donmcronald
I think something like Certera, but designed more like a pull through proxy is
the solution. So instead of making queries directly to LE you use the local
proxy instead and it tracks issuances, expiry dates, CAA expectations, etc. in
one centralized dashboard.

