Since the advent of LetsEncrypt, ACME, and Caddy I haven't thought about SSL/TLS for more than about an hour per year, and that's only because I forget the steps required to setup auto-renewal. I pay nothing, I spend a tiny amount of time dealing with it, and it works brilliantly.
I'm not sure why many people are still dealing with legacy manual certificate renewal. Maybe some regulatory requirements? I even have a wildcard cert that covers my entire local network which is generated and deployed automatically by a cron job I wrote about 5 years ago. It's working perfectly and it would probably take me longer to track down exactly what it's doing than to re-write it from scratch.
For 99.something% of use cases, this is a solved problem.
As someone on the other side of the fence who lives primarily in IT land, this is far from a solved problem. Not every device supports SSH for copying certs across the network, some devices have arbitrary requirements for the certs themselves (like timelines, lack of SANs, specific cryptography requirements, etc), and signing things internally (so that they’re only valid within the intranet, not on the internet) doesn’t work with LE at present.
So unless you’re part of the folks fine heavily curating (or jailbreaking) devices to make the above possible, PKI is hardly a solved problem. If anything it remains a nightmare for orgs of all sizes. Even in BigCo at a major SV company, we had a dedicated team to manage PKI for internal certificates - complete with review board, justification documents, etc - and that still only bought us a manual process with a lead time of 72 hours for a cert.
That said, it is measurably improved and I do think ACME/certbot/LE is on the right track here. Instead of constant bureaucratic revisioning of rules and standards documents, I believe the solution here is a sort of modern Wireguard-esque implementation of PKI and an associated certification program for vendors and devices. “Here’s the cert standard you have to accept, here’s the tool to automatically request and pin a certificate, here’s how that tool is configured for internal vs external PKI, and here’s the internal tooling standards projects that want to sign internal certs have to follow.”
Basically an AD CA-alike for SMB and Enterprise both. Saves me time having to get into the nitty gritty of why some new printer/IoT/PLC doesn’t support a cert, and improves the posture of the wider industry.
I feel like a lot of these requirements need to be really solved from first principles. What do you need these certificates for -- specifically, TLS certificates?
If the biggest issue is "we want to encrypt traffic" then the answer really should be something more automated. To put it another way, TLS certificates used to convey a lot of things. We had basic certs that said "you are communicating with the rightful owner of domain example.com" and we had EV certs that said "you are communicating with the rightful legal entity Example Org, who happens to own example.com" and so-on and so-forth.
But the thing is, we've killed off a lot of these certificate types. We don't have EV certs anymore. Let's Encrypt effectively democratized it to the point where you don't need to do any manual work for a normal "we want to encrypt data" certificate. I just don't understand what your specific requirements are, if they aren't directly "encrypt this traffic" focused, where you actually need valid certificates that work across the internet.
Put differently, if you're running an internal CA, you can push out your own certificates via MDM tools and then not worry about this. If you aren't running your own CA but you're implementing all of this pomp and circumstances, what are you trying to solve? Do you really need all of this ceremony for all of your devices and applications?
What do I need these certificates for? I need them because browsers have started equating a vanilla http server to a malware-infested North Korean honeypot
Have they? All I see is a little message saying "not secure". They've backtracked from trying to impose a scare screen, and they've even backtracked from displaying a red line through the letters "http".
They've also blocked JavaScript access to things like cameras and microphones if you're not using HTTPS. If it were up to me they'd always block them and you'd have to install an app, but still.
> browsers have started equating a vanilla http server to a malware-infested North Korean honeypot
It isn't that they are equal, just that it is difficult to tell them apart. The change over time is that UAs have more and more erred on the side of not trusting when there is any question.
Of course HTTPS sites with valid certificates could also be malware infested hot zones, but it is less likely. Sites with invalid certs are more likely to be a problem than those with no cert (the situation might imply a DNS poisoning issue for instance), and sites with no cert are a higher risk than those with a valid one.
At least we seem to have dropped the EV cert theatre, the extra checks done before issuing one of those were so easy to fake or work around in many cases that they essentially meant nothing [source: in DayJob we once had an EV cert for a client instance, and I know how easy it was to get because I was the person at our end who applied for it and installed it once issued].
They turned out to be expensive theater. One major problem is that company names aren't actually unique. An EV cert for "Stripe, Inc" still doesn't tell you that you're talking to the correct "Stripe, Inc". The other big problem was that users had no clue they were a thing, and when browsers tried to emphasize them in the UI it just confused users and made phishing easier.
You can still buy EV certs if you want to donate money to a CA, but that's about all they accomplish.
> You can still buy EV certs if you want to donate money to a CA, but that's about all they accomplish.
To be more precise, the browsers de-emphasized them and removed any distinguishing marks or indicators that made them valuable. EV certs used to display the company name in the URL bar, but they were stripped back.
I work for government and I can tell you the guys working infrastructure are still paying for shitty SSL certificates every year, in most cases for infrastructure that doesn't even see the light of day (internal), and the reason for that is none other that not knowing any better, and being unable to get their head out of their asses for enough time to learn something novel and implement it in their workflow. So yeah, there are those types out there in thew wild still.
In our defense, it’s because we’re expected to give everything a cert but often have no say on the security and cryptography capabilities of what’s brought onto the network in the first place, nevermind the manpower and time to build such an automated solution internally. Execs bringing in MFPs that don’t support TLS, PLCs that require SHA-1, routers with a packet buffer measured in single-digit integers but with a Java web GUI, all of these horrors and more are why it’s such a shitshow even today.
Just because someone’s homelab is fully cert’d through Caddy and LE that they slapped together over a weekend two years ago, doesn’t mean the process is trivial or easy for the masses. Believe me, I’ve been fighting this battle internally my entire career and I hate it. I hate the shitty state of PKI today, and how the sole focus seems to be public-facing web services instead of, y’know, the other 90% of a network’s devices and resources.
Aye. Between Lets Encrypt and compatible vendors, and e.g. Vault for internal CA-chains, /issuing/ certs in a secure, controlled and audited way (at least for the internal ones) is indeed a solved issue. We do have a lot of internal PKI chains running through vault and most of them are pushing 72 hour TTLs.
If you can do that, do that. It's great.
That being said, deploying and loading these certs is a fun adventure, even in somewhat modern software.
We're discovering a surprising amount of fairly modern software which handles certs in stupid ways. For example, if I recall right, NodeJS applications tend to load certificates to secure a connection to PostgreSQL into memory and never bother to reload. libpq on the other hand loads them on connection startup. Even Spring wasn't able to properly reload certificates 3-4 years ago and this was only added in recent versions with dynamic certificates - a colleague patched this in back in the day. Our loadbalancers folded the ticket of "Hey, reload these three certs" into the issue of reloading the entire configuration, including opening, closing ports, handing TCP connections over transparently, and a billion other topics.
Funny enough, if there is enough stuff running, there's not even an agreement on how to store stuff. Some stuff needs PEM-files, some stuff needs PKCS8 stores, some stuff wants 1-2 PKCS12 keyrings with certs and keys in there in separate entries. I even had to patch a PKCS12 handling library because one ruby library needed everything in one entry in a PKCS12 key store and there was no go-code under the sun to find to do this.
So there is a sunny place in which PKI is indeed solved. And besides that there is a deep dark hole that goes deeper than it should on earth.
It is solved but devices you are talking about refuse to get on board with the fix so here we are.
Also, I used to do IT, I get it but what do you think the fix here is? You could also run your own CA that you push to all the devices and then you can cut certificates as long as you want.
Don't take this as a snarky comment, but that sounds quite literally as "skill issue". Not in you personally, but in the environment you work in.
> PKI isn’t a solved problem.
PKI is largely a solved issue nowadays. Software like Vault from hashicorp (it's FIPS compliant, too: https://developer.hashicorp.com/vault/docs/enterprise/fips) let you create a cryptographically-strong CA and build the automation you need.
It's been out for years now, integrating the root CA shouldn't be much of an issue via group policies (in windows, there are equivalents for mac os and gnu/linux i guess).
> Just because someone’s homelab is fully cert’d through Caddy and LE that they slapped together over a weekend two years ago, doesn’t mean the process is trivial or easy for the masses.
Quite the contrary: it means that the process is technically so trivial the masses can do it in an afternoon and live off it for years with little to no maintenance.
Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology.
They didn't tell you their needs, but you're convinced this vendor product solves it.
Are you a non-technical CTO by chance?
> there are equivalents for mac os and gnu/linux i guess
You guess? I'm sensing a skill issue. Why would you say it's solved for their environment, "I guess??"
> Quite the contrary: it means that the process is technically so trivial the masses can do it in an afternoon and live off it for years with little to no maintenance.
I'm sensing you work in a low skill environment if you think "home lab trivial" translates to enterprise and defense.
> Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology.
* Yes, I have experience with Vault. I have deployed it internally, used it, loathed it, and shelved it. It’s entirely too cumbersome for basic PKI and secrets management in non-programmatic environments, which is the bulk of enterprise and business IT in my experience.
* You’re right, the organization is the problem. Let me just take that enlightened statement to my leadership and get my ass fired for insubordination, again, because I have literally tried this before with that outcome. Just because I know better doesn’t mean the org has to respect that knowledge or expertise. Meritocracies aren’t real.
* The reason I don’t solve my own PKI issues with Caddy in my homelab is because that’s an irrelevant skill to my actual day job, which - see the point above - doesn’t actually respect the skills and knowledge of the engineers doing the work, only the opinions of the C-suite and whatever Gartner report they’re foisting upon the board. Hence why we have outdated equipment on outdated technologies that don’t meet modern guidelines, which is most enterprises today. Outside of the tech world, you’re dealing with comparable dinosaurs (no relation) who see neither the value or the need for such slick, simplified solutions, especially when they prevent politicians inside the org from pulling crap.
I’ve been in these trenches for fifteen years. I’ve worked in small businesses, MSPs, school campuses, non-profits, major enterprises, manufacturing concerns, and a household name on par with FAANG. Nobody had this solved, anywhere, except for the non-profit and a software company that both went all-in on AD CA early-on and threw anything that couldn’t use a cert from there off the network.
This is why I storm into the comments on blogs like these to champion their cause.
PKI sucks ass, and I’m tired of letting DevOps people claim otherwise because of Let’s Encrypt and ACME.
> They didn't tell you their needs, but you're convinced this vendor product solves it.
It was an example from the ecosystem of available tools but in general yes, Vault can do that. Mentioning FIPS compliance was about letting you know that the software can be used also in governative environments. It's not just a "homelab toy".
> Are you a non-technical CTO by chance?
Senior cloud engineer here. Worked anywhere from not-so-small companies (250 people, 100 engineers) to faangs (tens of thousands of engineers).
> > there are equivalents for mac os and gnu/linux i guess
> You guess? I'm sensing a skill issue.
You're attacking me on a personal level because you can't argue otherwise. That's a common logical fallacy ("Ad Hominem" - https://www.britannica.com/topic/ad-hominem). You basically have skill issue at debating =)
> Why would you say it's solved for their environment, "I guess??"
When you account Windows, Mac OS and Linux you're accounting for pretty much the totality of the desktop computing landscape. The last two macbooks I had for work came with the mac os equivalent of group policies with certificates installed etc etc. Enterprise-tier Linux distributions can do that as well (eg: Red Hat Enterprise Linux).
> I'm sensing you work in a low skill environment if you think "home lab trivial" translates to enterprise and defense.
Again, worked anywhere from companies with 250 people to FAANGs. You have skill issue at sensing, it seems.
To get back to the point: homelab "triviality". In a way, yes. Large enterprises and even more defense can spend all the money not just for software but even for consulting from various company that can bring the skills to implement and maintain all the services that are needed, and train your people at that. Things become non trivial not on the base of technical issue, but on the base of organizational issues...
If we talk government and defense... Do you know the US government has dedicated cloud regions (eg: https://aws.amazon.com/govcloud-us/)? Do you really think that cloud providers offer those services at loss? Do you really think a few vault enterprise licenses are the issue there?
And by the way, Vault is just an example of one of the possible solutions. It was meant to be an example but you clearly missed the point.
> > Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology.
> Absolutely meaningless statement.
I think it's very meaningful.
It's not 1995, cryptography isn't arcane anymore. We had hardware crypto acceleration in cpu since at least 2010 (AES-NI). The tooling is well established on both servers and clients. The skills are on the market ready to be hired (either via employment or via contracting).
The issue is not technical in nature.
Oh and by the way: I've worked closely with engineers working for the US government. I wasn't close to the US government (because I am not an US citizen) but they were. They were "close enough" that they had to work in a SCIF and could only interact with me via phone. The systems they were working on... Those systems had their own private CA (among other things).
It's feasible. It's not a technical issue. If it's not done then it's an organizational issue.
> My username is literally a cryptographic mode of operation. But you didn't know that, because you have a low skill issue.
Again, ad hominem attack... You proved yourself to be quite a fool. You can't argue, you can't back your own opinions, you are only capable of attacking on a personal level.
> This is a joke, right? You're just an LLM going through training.
My account is from 2015 and has ~11k points. Your account is 4 months old and barely has 150 points. It's more likely that you're a poorly trained LLM (whoever trained you had skill issues :P) rather than me.
I'll be dropping this useless conversation. Farewell.
That's not what ad hominem means. Ad hominem doesn't mean I can't insult you
> My account is from 2015 and has ~11k points. Your account is 4 months old and barely has 150 points. It's more likely that you're a poorly trained LLM (whoever trained you had skill issues :P) rather than me.
> It's been out for years now, integrating the root CA shouldn't be much of an issue via group policies (in windows, there are equivalents for mac os and gnu/linux i guess).
How do you do this on a proprietary device from the late 90s that runs a WindRiver VXWorks RTOS with 1 MB of SRAM? The updated (full color!) Panelview HMI is running Windows CE 6.0, so it's perhaps more likely to be compatible, but I don't think the same group policies exist on that platform.
The masses can do it in an afternoon because they can choose to only install modern equipment that's compatible with the new requirements. Some of the "heavy iron" castings for the machines we have to work with were built more than a century ago, and only later automated with tubes and relays, and then again with the first PLCs that could do the job. But now "SSL everywhere" policies and certificate expiration timelines that don't make a distinction between firewalled OT networks and Internet-facing webservers don't allow anything to run for a decade without major, risky rewrites that cost tens of thousands of dollars for highly specialized engineering services and minimal downtime. Sure, adding a cert to the SCADA server is trivial, it runs Windows Server and has a NIC that can access the Internet, but on the other NIC...there's a menagerie of 30 years of industrial oddities.
If your homelab is still working after 2 years, that's great, but if it's not running after 100 years would you call that an organizational failure?
I had to automate a Fiery controller on a Konica Minolta network printer about 20 years ago. It was running vxworks. I used expect.
I've automated certificate upgrades on everything from dodgy network appliances to IBM System i servers.
I've used everything from headless browsers to scripted 5250 terminal emulators. Generally, if you have the will, you can automate anything that you do manually. The solutions are often janky, but they can usually be verified easily by other tools (curl output can be parsed to verify cert details).
> How do you do this on a proprietary device from the late 90s that runs a WindRiver VXWorks RTOS with 1 MB of SRAM?
You weight the cost of incurring in that thing getting hacked against the cost of that thing being rebuilt on modern hardware with modern technologies and enough processing power to do TLS.
Then you pick the "rebuild route". Easy.
Anyway, it's crazy how on this forum it goes from "REWRITE EVERYTHING IN RUST!!! ANYTHING THAT'S NOT RUST IS INHERENTLY UNSAFE" to the complete opposite of "why doesn't anybody think of the poor WindRiver VXWorks RTOS!!".
> Sure, adding a cert to the SCADA server is trivial, it runs Windows Server and has a NIC that can access the Internet, but on the other NIC...there's a menagerie of 30 years of industrial oddities.
Not gonna lie, all i hear is "i'm annoyed i was able to ignore the problem for 30 years but now i have to actually fix it".
>> Sure, adding a cert to the SCADA server is trivial, it runs Windows Server and has a NIC that can access the Internet, but on the other NIC...there's a menagerie of 30 years of industrial oddities.
> Not gonna lie, all i hear is "i'm annoyed i was able to ignore the problem for 30 years but now i have to actually fix it".
You're not entirely wrong, but you're talking to the wrong guy. Clean-slate rewrites are some of my favorite best projects, and I hate dealing with legacy junk - but there's just not budget to keep everything brand new all the time.
Realize that when you flush the toilet, the reason the water level goes down is likely to be a municipal sewer system and waste treatment plant that must never stop...they replace mechanical wear items once in a while, but some of the controls are well over 30 years old. Same story at the clean water plant that fills the tank back up. An average consumer might replace their phone every 2 years, but industrial processes and infrastructure have much, much longer timelines.
> Clean-slate rewrites are some of my favorite best projects, and I hate dealing with legacy junk - but there's just not budget to keep everything brand new all the time.
Hey, I get it.
The thing is, budget is largely artificial. Unless a company is on the brink of bankruptcy, there can be budget (but the company has to eat that from profits).
The thing is, eating budget for maintenance and system upgrades is a known playbook companies use. That doesn't make it right though.
That is why i wrote that the issue is organizational, not technical...
Put a modern reverse proxy in front of the legacy thing and turn off cert verification in the proxy? Or if the problem is the reverse, tell the legacy thing to use a forward proxy configured like it’s 2001 as its router.
Alternatively, those people are dealing with legacy systems that are pathologically resistant to cert automation (looking SQUARELY AT YOU vmware) and elect for the longest lasting certs they can get their hands on to minimize the disruption.
It’s generally best to assume experts in other fields are doing things for good reasons, and if you don’t understand the reason it might be something other than them being dumb.
Generally agree, that we shouldn’t jump on conclusions, but experts in my own field are doing bad things mainly because of lack of knowledge. With the lack of will as a very close contender for the top position. “The lack of possibility” is the exception.
When I argued here that Linux is still pain to maintain on my laptops, it wasn’t because it’s not possible. I just didn’t have enough willpower.
People who jumped on me because of this were still idiots, because they thought that their even less knowledge (what were the error messages, my exact setup, etc) can be helpful for me, but still it would be wrong to state that I was unsuccessful because it’s not possible, or I had full knowledge back then.
Especially now, that I’m running Linux full time.
For example, I’m quite sure that you can solve this problem with a proxy, no matter what’s behind it. Maybe it’s infeasible, because you would need a custom proxy implementation, but it’s definitely possible.
If there are systems that are that resistant to automation, the question should be 'does this system need a publicly-trusted server certificate, the same as a blog about cats or a Shopify shop?'.
The answer is no. If it can't practically be automated, it near-certainly doesn't need to have a public cert on it.
The older I get the more skeptical I get to free services that run on others servers. They have a bunch of expenses and you are getting it for free. You are not the customer. I rather pay for a service than gamble on some free service that might be shut down at any time, or that might have malicious intents.
Let's Encrypt is run by a nonprofit organization [1], funded by corporate and individual sponsors (like Google and AWS, but also the EFF and Mozilla) [2].
That doesn't guarantee they don't have malicious intents, but it's different from a for-profit company that tries to make money with you.
I think for certs, you are not better of paying $5 for the cert, than paying nothing to get an LE cert. It is already "subsidized" into cheapness, and the $5 company will bug you with ads for EV certs and whatnot in order to make a profit off you somehow since you are now a customer.
What I think LE did was to gather the required bag of money that any cert issuer needs to pony up to get the infra up and validated, and then skipped the $5 part and just run on donations. So while LE might stop tomorrow, you don't have any good guarantees that the $5 cert company will last longer if their sidebusiness goes under, and if you go to a $100 cert company, you are just getting scammed from some company who soon will realize that most certs are being given away and that they can't prove why their $100 certs are "better" in any meaningful way so they will also be at risk of going under. In all these cases, you get to use your cert for whatever validity period you had, and then rush over to the next issuer, whoever that is left when the pay-for-certs business tanks.
As opposed to cars or whatever, you can't really put more "quality math" into the certs so they last longer, the CAs have limits on how long they are allowed to last, so no more 10-year certs for public services anyhow. You might aswell get the cheapest of the ones that are still valid and useful (ie, exists in browser CA lists) and LE is one of those. Might be more (zerossl?) but same argument would hold for those. The CA list is curated by the browser teams lots better than me or you shopping around websites that make weird claims on why their certs are worth paying $100 for.
With you in general, but in this specific case, the whole thing seems healthy:
- Many companies (including competitors) are sponsoring LE, so the funding should be quite robust
- These companies are probably winning from the web being more secure, so the incentives are aligned with you (contrary to say, a company that offers something free but want to sink you under ads)
- the vendor lock-in is very weak. The day LE goes awry, you can move to another CA pretty painlessly
There are CAs supporting ACME that provide paid services as well.
For all but one of my personal use cases, Tailscale + Caddy have even automated away the setup steps and autorenewal of SSL with LetsEncrypt. Just toggle on the SSL features with `tailscale cert`, maybe point Caddy at the Tailscale socket file depending on your user setup, then point an upstream at the correct hostname and you're done.
> Since the advent of LetsEncrypt, ACME, and Caddy I haven't thought about SSL/TLS for more than about an hour per year
I run a couple of low-stakes websites just for fun and manually updating certificates takes me about 10 minutes a year, and most of that is remembering how to generate the csr. Setting up an automated process gains me nothing except additional points of failure. Ratcheting the expiration down to 47 days is an effort to force everyone to use automation, which makes no sense for small hobby sites.
> I'm not sure why many people are still dealing with legacy manual certificate renewal
Not everyone is a professional sysadmin. Adding automation is another layer of complexity on top of what it already takes to run a website. It's fine for large orgs and professionals who do this for a living at their day jobs, but for someone just getting their feet wet it's a ridiculous ask.
> Ratcheting the expiration down to 47 days is an effort to force everyone to use automation, which makes no sense for small hobby sites.
you make it sound like setting up automated renewals is an onerous process. IME it takes about the same amount of time to set up the automation according to the directions as it does to manually renew the certs with an ACME client. the only difference is you're pasting a command into a crontab/timer instead of directly into your shell to execute.
> you make it sound like setting up automated renewals is an onerous process
The problem isn't that it's hard to set up. It's that it adds another thing that I have to monitor for failures and troubleshoot when it breaks. For a small hobby site, I might go longer than 47 days without looking at it.
fair point. for the stakes we're talking about, i've personally found it more than reliable for a handful of hobby sites i've hosted over the years, and i don't typically monitor the renewal process for those types of things.
ymmv. for me, i'll fix it if i have to, but i haven't yet :)
I run a few low-stakes hobby things, and LE cert automation took the "once a year or so figure out how to do this because I haven't done it in a year and I should write it down but when I'm done I just go to the pub instead" to "", which was a nice change. Now I only have to interact with it when I add a new vhost to the web server, and that's just run a command to do so.
I currently use XML and server-side XSLT to transform it because browser makers keep threatening to remove XSLT support (and I like to make it accessible for anyone using alternative browsers). Unless I'm missing something, Caddy doesn't support that, so that's a nonstarter unless I want to rewrite everything.
What's frankly ridiculous is that the big softwares like Nginx and Apache don't deal with this on their own. I've been letting Caddy (my http host of choice) deal with TLS for me for _ages_ now. I don't have to think about anything, I don't have to setup automation. I just... configure my caddy to host my website on https://my.domain.com and it just fetches the TLS for me, renews it when necessary, and uses it as necessary.
You don't need to be a professional sysadmin to deal with this - so long as the software you use isn't ass. Nginx will _finally_ get this ability in the next release (and it'll still be more configuration than caddy, that just defaults to the sane thing)...
I've worked with a major financial institution (let's just say that you'd definitely recognise the name) in a past job, and while I couldn't really see exactly what was going on with the certs they issued, I'm sure it was a pretty manual process given our observations of things changing then reverting again later. I don't think regulation was really the issue though, just bad old processes.
I wonder what they will do with the shorter validity periods. They aren't required to comply in the same way; it's not a great look not to but I can't believe the processes will scale (for them or their customers) to renewing an order of magnitude more frequently.
It's likely to get worse as CAs rotate roots more frequently. Cross-signing will work for a time (provided you correctly install) but at some point, older devices will drop out of support and that'll be it.
Half of all Android manufacturers suck, and many users are lazy to update, so if you target the platform, you always have to support a bunch of random old versions.
Speaking as someone who has worked in tightly regulated environment, certificates are kind of a nasty problem and there are a couple of requirements that are in conflict for going to full automation of certificates.
- Rotation of all certificates and authentication material must be renewed at regular intervals (no conflict here, this is the goal)
- All infrastructure changes need to have the commands executed and contents of files inspected and approved in writing by the change control board before being applied to the environment
That explicit approval of any changes being made within the environment go against these being automated in any way shape or form. These boards usually meet monthly or ad-hoc for time-sensitive security updates and usually have very long lists of changes to review causing the agenda to constantly overflow to the next meeting.
You could probably still make it work as a priority standing agenda idea but its going to still involve manual process and review every month. I wouldn't want to manually rotate and approve certificates every month and many of these requirements have been signed into law (at least in the US).
Starting to see another round of modernization initiatives so maybe in the next few years something could be done...
It seems like the core problem here is that certificates are considered part of infrastructure, and further that they're part of infrastructure that requires approval!
Clearly not all automated infrastructure requires approval: autoscaling groups spin up and tear down compute instances all the time. Further, changes to data can't universally require approval, otherwise every CRUD operation would require a committee meeting.
Are certificates truly explicitly defined to be infrastructure that requires change approval? If not, perhaps more careful interpretation of the regulations could allow for improved automation and security outcomes.
> Clearly not all automated infrastructure requires approval: autoscaling groups spin up and tear down compute instances all the time.
In these sort of environments, they do not.
We're talking about environments where it is forbidden to make _any_ change of any kind without a CCB ticket. Short cert lifetimes are fundamentally at odds with this. Luckily these systems often aren't public and don't need public certs, but there's a slice of them that do.
What dictates that certificate update needs to have a manual change process? I'd bet that it's just legal team saying that "this is how it's always been" instead of adjusting their interpretation as the environment around changes.
The references I'd direct you to are NIST 800-53r5 controls CM-3 (Configuration Change Control) and CM-4 (Impact Analyses) along with their enhancements, require that configuration changes go through documented approval, security impact analysis, and testing before implementation. A certificate change is unfortunately consider a configuration change to the services.
Each change needs a documented approval trail. While you can get pre-approval for automated rotations as a class of changes, many auditors interpret the controls conservatively and want to see individual change tickets for each cert rotation, even routine ones.
In some regulated places, someone is legally responsible for authorizing a change in production.
If it fails, that person's on the hook.
So the usual way is to have a manual authorization for every change.
Yes, it's a PITA.
One place I've worked changed their process to automatically allow changes in some specific parts for a specific period during the development of a new app.
And for some magical reasons, the person usually associated with such legal responsibility are the one that don't trust automatic process.
Lazy vendors. I know how to set up Let's Encrypt for web servers that I directly control, but some of the web servers are embedded in commercial vendor products. If this were just about people's directly-controlled nginx/caddy webservers this would be easy. We're not talking about homelabs here.
I currently for EIDAS certificates, I can only choose a vouched certificate provider, and it's mostly somes that requires me to in person with my ID card with someone verifying the guy who made the CSR is actually me.
The certificate is used for double SSL to authentify the server doing the request , i.e that the server doing an API call to the bank server is one I own. (I find it a pretty neat solution and much better than requiring to make a theater dance to get a token to renew every 3600 seconds )
One example is FreePBX, which supposedly supports automated SSL renewal via LetsEncrypt, but constantly overwrites the config, thus breaking the “auto.” So I have to manually renew the cert, or I have to upgrade (ie rebuild the whole system from scratch, including installing the OS) to the next major version of FreePBX and hope they fixed the issue.
So I’m really not excited about the time limit dropping from 90 days to 47 days.
I think part of this can be explained by the fact that in many corporations, code and everything supporting it, like development time, developer quality, infrastructure maintenance and software architecture aren't seen as investiment but sunk cost that needs to be reduced ad infinitum to appease shareholders, so they throw human suffering and business risk at it to make it look good in the next quarterly report.
And yet, I am a bit worried that now, most of the web depends on LetsEncrypt. That's a single point of failure. Sure, they are "good guys", really, but remember that Google used to be "good guys" too. And this is a US-based organization, dependent on US rules, which is not so bad, but alternatives would be nice.
And yes, there are alternatives, but everything is made so that LetsEncrypt is the only reasonable choice.
First, if you are not using https, you get shunned by every major web browser, you don't get the latest features, even those that has nothing to do with encryption (ex: brotli compression), downloads get blocked, etc... So you need https, good thing LetsEncrypt make it so easy, so you use LetsEncrypt.
Because of the way LetsEncrypt verification works, you get short-term certificates, ok, fine. Other CAs do things differently, making it short-term certificates impractical, so your certificates last longer. But now, browsers are changing their requirements to only short-term certificate, but it is not a problem, just switch to LetsEncrypt, and it is free too.
Also, X.509 certificates, which is the basis of https (incl. TLS, HTTP/3, ...) only supports a single signature, so I guess it is LetsEncrypt and nothing else.
There are more ACME-compatible CAs than just Let's Encrypt, should they ever become the bad guys, or if you don't want to trust them for any reason, see [0].
I understand that people get annoyed at shorter cert lifetime, for instance if you are managing appliances or use SSL certs for other reasons than the common use case. But if you just want to serve a website, there are not so many reasons not to use HTTPS today, either on Let's Encrypt or on something else.
The idea would be the ability for a certificate to accept multiple signatures, making it more of a "web-of-trust" system. So you still have your LetsEncrypt certificate, but maybe augmented by another signature from an similar authority located in another country, or some other reputable organization that has your best interests in mind.
Maybe there are problems with that, but I never really understood the limit of a single signature for certificates. Is it because of bandwidth and performance requirements? Is it really a problem nowadays? especially with ECDSA making public keys much smaller.
Does this solve any problem that isn't solved equally well by just acquiring multiple separate certificates? I guess it would make your service highly available in case of revocation, but unexpected revocations are rare enough that almost everyone is willing to run the risk of a brief outage in case one occurs.
Then anybody can maintain a database of "known fingerprints", and a web-of-trust can be established without depending on a centeral-point-of-censorship.
Fuck CA's. They're not and never have been trustworthy:
it mainly means control these days.
ive made SSL then later TLS requirements for web browsers and we had fights on this sort of stuff.
yeah encryption is needed. but then you need authentication. and then, if authentication is controlled by corporations you're f'd.
instead youd want identities to be distributed and owned by everyone. many trust models have been developed and other than raw UI problems (hi gpg/pgp) its really not a terrible UX.
Google has backtracked from requiring HTTPS in Chrome. I repeat: HTTPS is not required. Where did you get that from?
If you're talking about Android, it doesn't require you to use WebPKI CAs. You can go self-signed.
If it's about the search engine, well, that's gone to shit and you'll only be visible if Google likes you in a completely arbitrary way - no point trying to add HTTPS to show up in search results now unless you already do.
Isn't this problem already inherent to domain names, even without encryption? There's always a central authority that can take away your stuff, and always has been. (In theory you can solve this with a blockchain, but, well, gestures)
The domain system needs at least some central control at the top, or it literally won't work. Do you remember the various "alternate root" projects in the 90's? I've stopped following them.
The cert situation has vastly improved over the past 30 years. I remember paying $100's of dollars for certificates in the 90's, faxing in forms, etc.
I'm not sure why many people are still dealing with legacy manual certificate renewal. Maybe some regulatory requirements? I even have a wildcard cert that covers my entire local network which is generated and deployed automatically by a cron job I wrote about 5 years ago. It's working perfectly and it would probably take me longer to track down exactly what it's doing than to re-write it from scratch.
For 99.something% of use cases, this is a solved problem.