Sounds about par for the course for folks that think shell is a productively sustainable way of writing secure or reliable software. Not even remotely sorry about that opinion. The gall to claim ACME compat, then force require a single client, all so you can remote execute arbitrary commands. Should be enough to ruin the CA, but we know how people handle things like this "oh, won't affect me" (until it does). Seemingly just so they can avoid some reverse proxy rules to host the challenge endpoints at the right place? Holy wow. Oh there we go baby, eval'ing with arbitrary input in the shell script, complete with lazy inproper string quoting. I should probably just stop before I say more unkind things.
Was it even run through shellcheck?!
Stop writing this stuff in shell people. 95% of the time I review any (posix, nushell lacks most of these issues) shell scripts, it's obvious they would fall apart the second any string unexpectedly had a space in it. Even scripts written by darling companies of HN.
- The barrier to entry for shell scripting is tiny. You basically start with `ls` in an interactive shell and end up writing a frickin' 500-line monstrosity within a month.
- POSIX has fossilised scripting languages. I hoped we'd have something like PowerShell by now, but although we have some fine alternatives, none of them seem to be good enough to actually overcome the inertia of POSIX.
- Doing something that looks correct is easy, but doing the right thing is super effing hard. See looping through complex sets of files.
- People use statements like "It's just a script", as if scripts are somehow easy to write. Bullshit.
Script may be not easy to write, but it's universally easy to deploy. Single almost cross-platform file, no dependencies. Publish the text on a webpage or in an email newsletter so everyone can copy-paste it to their system. Brilliant.
There is almost never a good reason to use constructs like eval in any language (and it exists in many languages), just like there are barely any good reasons to use constructs like system() in C. It appears acme.sh was running eval. I haven't looked into it, but the most common reason I see eval in use is because people don't know that you do it much simpler/more directly by just geting the shell to run commands that are variables with no issue, e.g., `doit() { printf "running: %s\\n" "$*" ; "$@" ; } ; doit ls /` which works in POSIX sh, no bashisms. (Obviously you should only call it with at least the first argument being trusted)
But shell is really not hard if people stick to a few easy guidelines:
- quote all var expansion, e.g., "$var" not $var or ${var} (very rare to need otherwise, and never for untrusted data)
- use "$@" to perfectly forward arguments (not $@, not $*, not "$*" except when you want to turn many arguments into one)
- don't use eval
- use `set -eu` and explicitly handle functions/commands that may benignly return/exit non-zero, e.g., `diff -U10 ./a ./b || true`
- use printf on untrusted strings instead of echo (just use it generally, I say), e.g., printf %s\\n "$var" instead of echo "$var". One of the few times you want to use "$*" is with printf though, e.g., printf %s\\n "$*" instead of echo "$@". Try them out, easy to see why, as with one thing to format and multiple arguments, `printf %s\\n "$@"` is equivalent to `for i in "$@" ; do printf %s\\n "$i" ; done`
- when using xargs, use -0 if available, or at least -d\\n if available (busybox doesn't have it for example). also usually want to use -r
And then run shellcheck because even those of us that painful have this seared into our brains make mistakes.
Or just use a scripting language that eliminates many of these headaches (nushell) or a real programming language.
I get it, I really do. At least now I have nushell for some sanity, but I still find myself constantly writing a shell script and then realizing after a few iterations that I should've just written in nushell/Rust.
As a thought experiment, how many places rolled out acme.sh to prod and didn't bother code reviewing it or running it through shellcheck?
Every language exists on this spectrum, and pretty much every language is clustered near shell. All languages require you to be "principled" in a few key areas, and many projects (open source and closed source) don't do a great job at that.
At least shell is good at a specific and very useful thing (sequences and pipelines of other commands), and is already installed on most machines -- even windows nowadays since almost everyone has git installed. My "build system" for C projects is just doing the following from pwsh: `& "$env:GIT_INSTALL_ROOT\bin\bash.exe" .\build.sh`
I'm happy (and spoiled) that I live in a world where I get to use reliable software and not worry about it being inaccessible. With a single nix command, I'm dropped in an environment with secrets mounted by a single age key, with all programs and configuration exactly as I want. It's trivial for me to "just switch" to nushell/rust and not worry about such things. And nushell and rust are not clustered near shell for the aspects I'm talking about in this discussion.
"bash is everywhere" is just more lipstick on the pig, in my strong opinion.
People write in bash because you can expect bash to be forward compatible with how people write bash. A bash script written by the average person today will run on a bash interpreter from 2006 (or 2023). Very few languages have the developer culture to achieve this in practice even if it might be possible in theory. And it's not like Bash doesn't get new features. It does constantly. But unlike, say, Rust or JS, the developers don't instantly start using forwards incompatible features. It's a different mindset. If you're writing in shell you're writing for longevity and compatibility.
Unfortunately requiring approval and authentication from a random incorporated entity running a CA every 90 days is inherently complex. And all the ACME protocol implementations do is hide that inherent complexity by adding even more. CA TLS HTTP is very fragile but it could be made less so if corporate browsers didn't demand short lifetime certs-- a demand reasonable for businesses but not for human people or things expected to last longer than a few years without being touched.
This doesn't make any sense to me. Stable Rust is forward compatible just as much as shell scripts are. Bash, for example, does introduce things in newer versions and they're near universally ignored.
All of this also excludes that the fact that your shell script will be just as janky in 10 years as it is now. Again, I regularly, regularly see shell scripts from companies that HN fawns over that don't pass shellcheck and have inherent issues that don't occur in better scripting languages, or full programming languages.
"Forward-compat" is not unique, and not a value proposition to me if it inherently means less reliable function.
Here's an anecdote to provide sense. I recall when I was using a 3 month old (1.48.0 2020-11-19) rustc on Debian 11 (right before it was officially released) and I kept running into rust software I couldn't compile because rust devs used bleeding edge 1.50 (2021-02-11) features. This wasn't a debian is old thing. The rustc was literally 3 months old at this point. At least one of the devs was a person I knew on IRC. He was kind enough to re-write it ("plotsweep" a rust software defined radio program) using only 1.48 and older features. But that's the exception. Most rust devs assume your toolchain is from 3rd party out-of-repository and latest and think forwards compatibility is useless like you do.
Shell scripts can be and often are jank. But at least it's stable jank. Shell devs tend to write portable, stable code because that's the entire point of dealing with the shell jank.
Yeah, I think we're not going to agree on this. I very much like living in a world where I get to use stable, released software, provided by my distro, to build reliable software that is forward compatible.
Three months old software is new enough to be in Arch repository, so in this case it's not about Debian being old, it's about rust devs using bleeding edge features.
If you're contradicting a person's argument ("yeah, it was") at least provide some more explanation, otherwise your comment is rarely more than a "is too!".
Shell is a bad fit for many uses (an ACME client is more ambitious than I would build in shell, personally), but the things it's good at, it's really good at. I have yet to find anything else that's even close to as good for glue code when I have a handful of tools and/or a bunch of files that I need to string together. Unless you're writing in Ada, I promise whatever language you think is better has its own sharp edges (if we're allowing eval, then not many languages are going to be safe, really).
Well. I've been using python personally since 2007, and it never broke with update (I still occasionally run scripts written for 2.5). Also, I am working as a python backend dev, dealing with hundreds of systems (we are deploying 270+ systems and supporting another 340 user built docker images built on top of py library), and no problems there either. In my previous job, I was doing deployment to 4000+ kiosk computers, and still, no problems with python breaking.
What was definitely problematic was the update from 2 to 3. Another problem may be dependency management (people not pinning dependencies). But the python itself? What does it even mean? Interpreter not starting?
If you're using Python as a bash script replacement for various individual standalone scripts, typically you would be using system libraries without pinning dependencies, in which case you'll break on various OS updates as Python libraries don't seem to value backwards compatibility. Sure you could include a virtual environment with every script but that's not very ergonomic and will really add up...
Usually when I port shell scripts to python, I create a folder for it and throw in some setup.py / requirements.txt. But lately, I've automated most of the deb package creation process (https://copier.readthedocs.io), and I typically create the whole deb package. Of course, for single use stuff, I don't bother with either.
This is a Chinese CA, I don't think there was a danger of any of us using it, and none of them are here to receive this criticism. Most of your complaints are about the shell script that the Chinese CA is injecting and that's... not really the problem at all. It's amazing that it's not literally a rootkit; that's what I expected when they said a Chinese CA was injecting a shell script that acme.sh was running. You're missing the forest for the trees here.
This doesn't get better if they had injected a better-written script or a Rust program or whatever else. They could still have injected anything, and that's the problem. The people who wrote acme.sh (different people than the Chinese CA), which has the security vulnerability that the Chinese CA exploited, might have some soul-searching to do.
I had wrote some shell software (~5k loc without comment i think?) and agree.
I was using shell because the project want to maintain compatibility between different distros (including some weird customized one without `ps` and `sed`), but that's all. Shell is not a good choice for things other than scripting.
I think the title buries the most horrifying part of this. The HiCA certificate authority is relying on an RCE to do an end-run around the semantics of the ACME HTTP-01 validation method.
Fucked up and they should be booted from every root program for this.
They aren't in any root programs. They're just taking certificate requests and relaying them to real CAs, which is why they need to exploit an RCE in the ACME client, since the ACME client wouldn't otherwise be able to complete the validations required by the actual CA.
When confronted they just flat out shut down the service. They also donated $1000 to the project, and they've redirected requests to their payment site to the US White House's website, and they're from China.
They were also suggesting that user's ran the utility as root...
Major props and thanks to the HiCA person for engaging on the thread even though they were getting hammered. Yes they made some really (damn clever but) bad implementation decisions to use an RCE in the client to basically hack around the entire system, and then compounded it with other bad decisions to redirect to the US White House website to stop a DDoS, but the fact that they engaged, admitted, and explained what was going on I thought was really big of them.
Everybody makes mistakes people. If this person was trying to be malicious they would have disappeared, not come straight clean about it on the thread.
Curious to know if this could, maybe it should, have ripple effects to the various SSL Root CA programs. Having someone run a subCA that actually exploits an RCE against ACME clients doesn't seem very trustworthy, and any CA enabling this behaviour should probably be kicked out of the trust stores?
The sub CA is operated by ssl.com, not HiCA (which is not a trusted certificate authority). HiCA is relaying the certificate requests to ssl.com, which is properly validating the requests in accordance with all the requirements. ssl.com isn't doing anything wrong. That's why HiCA needs to exploit an RCE in acme.sh - ACME doesn't support relaying certificate requests to other CAs like this.
Someone posted a comment on github claiming they are the founder of Quantum (the sub CA of ssl.com - see https://crt.sh/?caid=200960 ) and that they are the provider of the HiCA service. So it does sound like there is a closer link here than your comment would indicate:
Quantum is not a trusted CA. ssl.com has a white-labeled intermediate CA with the name "Quantum" in it, but this intermediate CA is operated by ssl.com under all the same controls as ssl.com's other intermediate CAs. Quantum has no ability to issue trusted certificates themselves.
So the person claiming to be the founder of "QuantumCA" does not possess the private key corresponding to https://crt.sh/?caid=200960 - can we be sure the private key is only accessible by ssl.com's CA system? So the certificates listed here aren't issued by this person, but by the ssl.com's system? https://crt.sh/?Identity=%25&iCAID=200960&exclude=expired&de...
Also, why would ssl.com even create a subCA named "QuantumCA"? Are they in business with this person claiming to be the founder of "QuantumCA" who appears to be responsible for exploiting this acme.sh 0day? What does this say about ssl.com's trustworthiness? Or is the person in the github comments lying?
> So the person claiming to be the founder of "QuantumCA" does not possess the private key corresponding to https://crt.sh/?caid=200960 - can we be sure the private key is only accessible by ssl.com's CA system? So the certificates listed here aren't issued by this person, but by the ssl.com's system? https://crt.sh/?Identity=%25&iCAID=200960&exclude=expired&de...
(The audit could be flawed, but it's the same amount of assurance we have for any intermediate CA's private key - the fact that "QuantumCA" is in the name does not change the risk calculus)
> Also, why would ssl.com even create a subCA named "QuantumCA"? Are they in business with this person claiming to be the founder of "QuantumCA" who appears to be responsible for exploiting this acme.sh 0day? What does this say about ssl.com's trustworthiness? Or is the person in the github comments lying?
There is a business relationship between QuantumCA and ssl.com. QuantumCA is a reseller of ssl.com, and they've paid extra to ssl.com so that the certificates they purchase get issued from an intermediate CA named "QuantumCA" rather than one of ssl.com's usual intermediate CAs which have "ssl.com" in the name. This lets QuantumCA pretend to be a real CA. This is a common practice in the industry, and I don't think it says anything about the trustworthiness of ssl.com, because the business relationship with QuantumCA doesn't in any way subvert the integrity of the WebPKI since ssl.com retains control of the issuance. Still, I wish intermediate CA white-labeling were banned because it causes terrible confusion about who is and isn't a CA.
I find it troubling that a root CA (ssl.com) is apparently OK with lending their name in a business relationship with an actor that is actively exploiting an acme.sh 0day.
This feels a little bit like doubling down to find ways to implicate the actual CA instead of the reseller. It's clear how mismanagement by a real CA would make a more interesting story than by this random no-longer-existing pseudo-reseller, but I don't think there's evidence to support that story yet.
But it's not a random pseudo-reseller? The one github comment from "the founder of Quantum CA" seems to say they are also the creator of HiCA, which is the entity that was exploiting the 0day in acme.sh. And the crt.sh link shows an intermediate CA cert named "QuantumCA", signed by ssl.com.
So QuantumCA == HiCA == exploiters of the acme.sh 0day, it's all the same entity? The intermediate CA could just as well be named "0dayexploitersCA"? Why is it not a huge concern that ssl.com is fine with operating such a "0dayexploitersCA" intermediate?
Quantum CA (brand, not operator)/HiCA still can't issue certs for domains they don't "control" by having RCE on the systems they point to.
all CA requirements for validation still need to be fulfilled for issued certificates, as ssl.com, the Quantum CA operator, which exclusively holds the private keys, is a "proper" CA.
this does not affect the trust in the CA infrastructure or ssl.com itself; while this is morally questionable to keep the business relationship, it does not mean the CA is not following the signing requirements.
These sorts of hacks remind me of when AOL's AIM server was exploiting bugs in their own client to run verification code[0]. Hardly an acceptable practice in 2023, though.
This raise one more issue about Chinese providers.
The site using this exploit, HiCA is run by xiaohuilam on Github. He/She is also the founder of two famous SSL certificate provider in China, DigitalSign and QuantumCA. Additionally, he is also a contributor of acme.sh repository. The acme.sh repository locked issue #4659 quickly after it raise attentions in the developer community in China.
It's hard to imagine that, as one of the repository's contributor, once you have found a vulnerability, you are going to use it in your own product, instead of fix it. They are just another version of Pinduoduo (owner of Temu, and also the one who put spyware on user's android phone).
<quote>
... HiCA is injecting arbitrary code/commands into the certificate obtaining process and acme.sh is running them on the client machine.
</quote>
Even if this is just RCE in the script somehow (I doubt it, it can probably do anything the user running it can), it's horrifying. It means the certificate authority could just take your newly generated certificates and upload them anywhere they want. That's a catastrophic compromise in the TLS security model.
Somebody who cares about the acme.sh client specifically would be able to say, but in general it's not necessary that your ACME client has these keys.
What the ACME protocol wants to do is hand over a CSR (Certificate Signing Request), and get back a certificate, and to achieve that it has to explain how you'll prove you're entitled to such a certificate.
Most ACME clients will also make a suitable proof (in at least some cases), and also generate a suitable CSR from first principles, for which they will need to generate a new private key - but that's not a necessary part of the system, and it's certainly not rare to generate your own CSR, either because you must technically, or because your own security processes say strange women, lying in ponds, distributing swords is no basis for a system of government sorry, I mean, that this key is private and shouldn't be on the host running ACME services.
I have no idea, my point was that in general the ACME client can't necessarily give you the private key even if it wanted to, because if you provide a CSR the key needn't even be on the same machine, let alone accessible in its execution environment.
The CA isn't directly compromised so a third party couldn't generate any arbitrary certificate this way. Essentially though, assuming my understanding is correct, it would allow them to be a man-in-the-middle and take copies of the keys & certificates used by this tool, allowing them to use keys and certificates generated by that tool. Also, if such a tool is run by root (bad practise, but not uncommon practise) or other significantly privileged user, they potentially have access to far more.
Yeah, but that's more likely to be noticed in cert transparency and by the website operator, as there's either a duplicate cert in the log or the website server does not work.
Unfortunately this is related to acme.sh, a shell script tool to request new and replace free certificates. So far, this GitHub issue is quite disturbing.
IMO it’s just a demonstration why you don’t write complicated or security sensitive code in shell script: it’s basically impossible to get right, there’s pitfalls around every corner and it’s extremely difficult to check for mistakes.
I wouldn't say there are too many pitfalls. Quote all variables (acme.sh seem to do well on that part) and don't use eval (probably the reason for this problem).
Was it even run through shellcheck?!
Stop writing this stuff in shell people. 95% of the time I review any (posix, nushell lacks most of these issues) shell scripts, it's obvious they would fall apart the second any string unexpectedly had a space in it. Even scripts written by darling companies of HN.