Hacker News new | past | comments | ask | show | jobs | submit login

A similar problem with Azure happened way back in 2013: https://www.computerworld.com/article/2495453/microsoft-s-az...

More recently, it happened with Ericsson: https://www.theverge.com/2018/12/7/18130323/ericsson-softwar...

This article has some information about how Let's Encrypt enabled an "automated process that handles renewals": https://duo.com/decipher/proposal-to-make-https-certificate-...

I wonder if such a process should be made an industry standard? Does anyone know if there are any proposals for it?




Let's Encrypt literally is an implementation of the industry standard; the standard is called Automatic Certificate Management Environment.

https://tools.ietf.org/html/rfc8555


But, somewhat annoyingly, it is only seen as applicable to the public internet. There's no effort to make ACME based CAs for non-internet usage.


In practice most non-Internet PKIs don't actually want to do proof of control which is ACME's secret sauce. As you'll see much of the document is about the proof of control, because that's both the hardest problem and the only reason for ACME to exist at all.

In private PKIs either proof of control over the name is considered an out-of-band problem or it's elided altogether. I wrote about this when Peter Gutmann was being angry about ACME years ago, Gutmann saw ACME as redundant because of existing protocols like SCEP. If you want certificate issuance automation but you don't need proof of control over the names then you don't need ACME - SCEP (or half a dozen others) are fine for this purpose and you should use those.

Now, of course if you don't want proof of control but you do want automation it's worth taking a moment to reconsider exactly what your goals really are: What is a certificate doing for me here? What exactly am I certifying, if anything? But if the technical requirement is there, regardless of whether it makes philosophical sense, SCEP delivers and ACME is overkill.

As I wrote in my thread with Peter, it's usual with things like SCEP to provide a default implementation in which the part where you only give certificates to the Right People is left marked /* TODO */. Thus often the result is pure security theatre, certificates are issued equally to bad guys or good guys without distinction and nothing at all is really being certified.

Manual processes may or may not be better in terms of actually certifying anything.


For internal use, create and distribute you own root CA with self-signed certificates.


And by letting smallstep/certificates [1] handle ACME, it's just as easy as using LetsEncrypt for public certificates.

[1] https://github.com/smallstep/certificates


This is great, thanks for sharing.


So many hidden gems on github. I’d been searching for something like this.


Sure, distribute your own root CA, but then how do people get signed certificates? I tend to work with large companies, where getting a signed certificate involves opening a ticket and waiting for someone on the other side to respond.

ACME would be ideal, but the official response of Let's Encrypt is that ACME is overkill for corporate environments and you should roll your own certificate automation.


Delegating enroll permissions is a solved problem technically at least in a windows domain. At that point it's an org policy problem and ACME won't help.


In an enterprise, you use the API that a CA provides, and build it into the ticketing system. I helped build a system that took care of this with ServiceNow a few times now.

At one place we aggressively policed external facing certificates. Don’t follow the process, your service gets whacked.

It’s a process you should look into, because the compliance regimes will start paying attention to it someday soon.


Unless your DNS public. That still allows Let's Encrypt to verify you and is actually used by RavenDB to provide SSL certificates to internal instances.


Because everyone just does normal pki via api (like vault). No need for acme flow when you control entire system


I think you can create your own acme server for that


No need to do acme when you can establish user identity and associated policies via internally deployed authn/z service


Automated renewal doesn’t mean that you don’t need to have monitoring. I’ve seen many letsencrypt-baked servers generating certificate errors because the automated renewal system broke (misconfigurations, borked updates, corporate-level firewalls/proxies blocking requests, etc.).

Even with letsencrypt you still MUST have a monitoring system, notifications going to the right person, and in general an organization that can act on this. The problem is often not technical: your organization must be structured in a way that those notifications are acted upon. If anything, letsencrypt lessens the frequency of those notifications so I posit that it’s even worse from the point of view of validating your organization: with regular certs, you get notified once every year or two.

Anyway, letsencrypt is a good thing: it’s just not a solution to this problem.


Let's Encrypt does exactly the right thing, which is to email you if a certificate is expiring without being renewed. Pretty much the behaviour you'd want from any CA, no?


That's a great start.

But in a non-technical organisation, who should those messages go to?

Often the initial LetsEncrypt setup will be handled, correctly, by some IT staff. Then it might break several months or years later for some odd reason.

The organisational challenge is to get the message through to someone who understands it and will act on it.


Yes, and: fix bugs so the setup doesn’t break. I’m constantly babysitting LetsEncrypt. It’s always failing in some stupid way, and all it can go is Email me with: “Ive been silently failing for the last couple of months and now your certificate is going to expire if you don’t drop everything and comb through my logs now LOL!”

This time the problem was LE all of a sudden decided to start storing my certificate in a directory called mydomain.com-0001 instead of mydomain.com, breaking the rest of the setup that relies on things being in the right directory. Automation is only useful when the software behaves predictably and consistently.


Comforting to know I'm not the only one who constantly has random problems with Let's Encrypt. You're on point about the silently failing bit too.


LetsEncrypt issues for 3 months by default and then tries to renew after 2 months.

So if renewal fails you should have ~30 days to fix it.

But this does work best if your tools try the 2-month renewal. (I'm looking at you, wpengine!)


Which client are you using? Certbot?


> The organisational challenge is to get the message through to someone who understands it and will act on it.

Yes, and to keep a very infrequently used channel working and up-to-date.


This is why letsencrypt very sensibly creates short-lived certificates, meaning that the channel is not infrequently used.


The channel only needs to be used when the automated process breaks.


Oh right - yeah, that sucks. Rarely used processes often break.

Hence efforts like https://tools.ietf.org/html/rfc8701 - designed to make sure extensibility options don't get messed up.


It doesn't eliminate the need for monitoring.

I'm looking for a program that is going to connect to all my SSL sites, and report back a problem if the cert is within 30 days of expiration. It is easy enough to write a client that will fail after the cert has expired, but I want one that will warn me ahead of time. And I don't want to mess with the system clock or something like that.

It shouldn't be a big program, I may just write it myself.


Nagios can do that (and you could borrow from its check_http)

  define command {
    command_name                    check_certificate
    command_line                    $USER1$/check_http -H $ARG1$ -p $ARG2$ -4 -S -C 21 -t 20 --sni
  }
A bigger challenge is getting a complete list of your https websites, and an even bigger one is finding and monitoring all those non-https certificates, eg payment gateway certificates.


https://sslping.com

is totally free, superfast and checks your TLS settings too (it's my baby)


https://letsmonitor.org/

Also my employer (well, until the end of this week) Kynd does this as part of its broader "Check cyber-security stuff" offering for non-technical people. https://www.kynd.io/


Hi, I'm a co-founder of Amixr and we've developed curler.amixr.io, which monitors a website and delivers email notifications for free and also has an integration with our flagship product Alert Mixer


Depending on your use case, environment, etc there are existing tools like Monit that can do this for free. Or paid services like Uptime Robot and StatusCake.


updown.io https://updown.io/ has a pretty generous free tier and (in addition to checking if your service is up) sends you a warning when a cert is expiring within 14 days


For sure it's not a letsencrypt problem.

But as a person who rolls LE certs across a very non-happy-path environment (many SAN domains, edge nodes which are geo-balanced).

I have a lot of issues automating this process, right now I have a HTTPd which reverse proxies the .well-known back to a central place where I run certbot and then I push out the cert to the nodes, however, sometimes one of our SAN domains will need to be removed and the whole universe comes crashing down.

DNS-01 Challenge is "nice" (although, doesn't feel super well supported); but requires domains registered with some kind of DNS server that accepts API's to change records, so Amazon route53- but it's exceptionally hard to roll your own DNS in this case. :\


Two tricks that might help you:

1. HTTP-01 challenges have a "correct" answer for a given Let's Encrypt account which depends only on the challenge (ie the part in the HTTP GET request) and on knowing which account you want to use.

Silently Certbot creates you an account, with a private authentication key and so on, for the Let's Encrypt service. When it gives you a file to prove control by placing it in /.well-known/acme-challenge/ the content of the file is always the same as the filename, plus a suffix that depends on your key.

So long as you use the same account you can thus bake this suffix into the web server, essentially causing it to answer any request from anybody: "Hey, who is allowed to issue for somename.example ?" "dijit is allowed to do that". Bad guys can't use this because they don't know your private account key, but for you now magically everything is authorised, since when it is asked your server will answer "dijit is allowed to do that" to any question it's asked.

2. DNS-01 can be redirected using CNAME. Add a CNAME, once, manually if necessary, to redirect the DNS-01 checks to a DNS server you've set up for this specific purpose.


Besides the multiple commercial DNS providers which certbot has plugins available for there is also a dns plug-in supporting rfc2316 updates. This means it can be integrated with a large number of self-hosted DNS servers. Bind, powerdns and Microsoft DNS are just some of the servers with rfc2316 DNS update support.


HTTPS/Letsencrypt would be a good thing if it was optional. But now every 2-bit static website is forced to implement HTTPS or face complaints from users whose browsers have locked them out. What a total fsck-up!


It's called ACME protocol and it's RFC 8555.

https://en.wikipedia.org/wiki/Automated_Certificate_Manageme...


Pro-tip: Renew your certs automated every 30 days and set to expire 90 days and set alerts at 35 days.


AppViewX and Venafi are the two main competitors in this space of certificate lifecycle automation - many Fortune 1000s are customers of one or the other to prevent exactly this type of problem - Venafi is the larger incumbent but AppViewX has the newer platform and is replacing Venafi in some key accounts

Challenge is discovering all the certs that exist across the org (10k+ often) and then having a fully automated renewal process requires a pretty advanced / complex platform - don’t know anyone who has built this tech successfully in house - curious if anyone has though in the HN community?

Disclosure I’m an investor in AppViewX




Consider applying for YC's W25 batch! Applications are open till Nov 12.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: