
23,000 HTTPS certificates axed after CEO e-mails private keys - artsandsci
https://arstechnica.com/information-technology/2018/03/23000-https-certificates-axed-after-ceo-e-mails-private-keys/
======
DyslexicAtheist
Oops Trustico server allows arbitrary RCE incl root shell:
[https://twitter.com/svblxyz/status/969220402768736258](https://twitter.com/svblxyz/status/969220402768736258)

see also
[https://groups.google.com/forum/#!topic/mozilla.dev.security...](https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/BLvabFwcJqo)

~~~
DyslexicAtheist
did they __really __generate the certificate pair server-side?

pkiJS / WebCrypto² API are a keygen¹ replacement so that you can let the
browser generate the certificate pair client-side. If the user isn't informed
that key generation happens locally then maybe it looks as if the certificate
is "downloaded". But with these methods when implemented correctly,
certificates never goes to the server (and is saved locally).

I'm not a fan of the above approach even though Mozilla sells it as safe: e.g.
internal direct access to the OS RNG (instead of doing it in JavaScript Math-
random() which led to a lot of critic throughout history of browser based
crypto) , since the browser is still a sandbox, and IMO unsuitable for private
key storage (even just for a few seconds).

Would be interesting to know how trustico implemented this. It's somehow so
crazy and negligent a thing to imagine. There is no explanation to why they
didn't just implement it using standard software stacks for PKI (primekey has
a jboss thing they could have used). Literally copy/pasting things from SO or
some example code found in the manuals from WebCrypto would have been better
than what they did. These guys have "trust" literally in their name.

EDIT: Tippfehler

__

¹
[https://github.com/w3ctag/wiki/wiki/keygen](https://github.com/w3ctag/wiki/wiki/keygen)

² [https://developer.mozilla.org/en-
US/docs/Web/API/Web_Crypto_...](https://developer.mozilla.org/en-
US/docs/Web/API/Web_Crypto_API)

~~~
SilasX
Stupid question: I thought it was accepted practice [1] to generate the
private key server-side so long as you don't store after the user's downloaded
it? (Not that Trustico followed that part either.)

[1] though not something I would endorse!

~~~
slrz
Why would it be? It's going to end up in lots of places it's not supposed to
be. Better generate the private key where it's needed and never have it leave
the system.

~~~
SilasX
I said I don't endorse it and don't think it's a good idea but e.g. you can
have AWS generate your private key (for SSH though not for a cert AFAIK) with
the warning that they can't recover it if you lose it.

I just figured there might be some compromise to accommodate users who don't
know how to generate their own.

------
tialaramex
A question I still have is what Trustico _hoped_ was going to happen here. Was
there some arrangement where if the certs have to be revoked they get a
refund?

Trustico also don't seem very clear on what exactly they're accusing Symantec
of, the Ars article quotes them

"We believe Symantec to have operated our account in a manner whereby it had
been compromised" and "We believe the orders placed via our Symantec account
were at risk and were poorly managed"

In the relationship we're talking about here, a reseller to a CA, basically
the only valuable information that can be compromised is payment instructions.
All the important seeming cryptographic stuff? All of that is allowed to be
public by design, the CSR, the Certificates themselves, any validation
messages used - all public thanks to asymmetric cryptography.

Symantec got in trouble last year for inadequate (in fact basically non-
existent) oversight of their RAs, particularly the Korean CrossCert. Unlike a
reseller the RAs were trusted to do the domain validation themselves, so if
CrossCert said "We checked and it's OK to issue a Thawte cert for
www.example.com with this key" then Symantec just did it and didn't ask any
questions (though they do seem to have checked they got paid...). Resellers
aren't trusted with validation, they're just glorified sales people who just
take a cut and help with price discrimination, what's there to "compromise"
months after you cease the relationship?

~~~
Ajedi32
I suspect the logic was something along the lines of "Symantec has acted in an
untrustworthy manner (enough to get them distrusted by several major browser
vendors) therefore we consider all our customer's certificates that were
issued by them to be untrustworthy ("compromised") and want them revoked."

Still doesn't explain why they thought that revoking 50k of their customers
certificates with less than 24-hours notice would be a good idea though. That
really just seems like shooting themselves in the foot.

~~~
rando444
Can we take a step back even further? I can't figure out the logic that got
this started:

 _" These Private Keys are stored in cold storage, for the purpose of
revocation."_

Like what? Why? Since when do you need a private key to revoke a certificate?

I feel like I understand certificates really well, but .. am I missing
something?

------
dancek
This was very irresponsible from the CEO, but should not have been possible at
all.

The private key should be generated where it's used, a certificate signing
request created and sent to a certificate authority. After the CA makes sure
the CSR is legit, they can sign the certificate without seeing the private
key.

Copying the private key outside the server is an unnecessary risk. Giving it
to a third party such as a CA or allowing them to generate it, moreso.

~~~
cjalmeida
They said in the article they kept the private key to simplify the revokation
process. Thus, very likely they also generated the private key.

It's a bad practice promoted by issuers that want to sell a "managed" or
"value added" service corporate IT is so fond of. The lesson here is that
there's a lot of IT people that don't grasp the basics of key management.

~~~
gesman
The road to security hell is paved with intentions to simplify.

~~~
cratermoon
That's the most succinct phrasing of the problem of security vs. users (and
customers) complaining about what they perceive as the inconvenience and
complexity that accompany securing digital assets.

~~~
creeble
I like to say (and it's not original) that security and convenience are two
endpoints of one continuum.

You can't get more of one without moving away from the other.

There are a few rare near-freebies, but they always have a gotcha. And you'd
better understand what they are before taking advantage of them.

------
kerng
One part that is missing is why did the CEO say those certs were already
compromised before they he mailed them? He mailed them sort of as proof they
are compromised?? Im confused. Any thoughts?

From the article:

"When Rowley asked for proof the certificates were compromised, the Trustico
CEO emailed the private keys of 23,000 certificates"

~~~
pfg
He's arguing that the various issues that lead to Symantec being gradually
distrusted mean that they believed all their existing non-expired, non-revoked
certificate were compromised (in the sense that they should not be trusted)
and should be revoked.

Maybe it's just me, but a company who stores their customer's private keys,
sends them around in a ZIP file via email and has a trivial code injection
vulnerability on their site, with a process running as root, is probably not
the one I want to make that call.

~~~
BuildTheRobots
> Maybe it's just me, but a company who stores their customer's private keys,
> sends them around in a ZIP file via email and has a trivial code injection
> vulnerability on their site, with a process running as root, is probably not
> the one I want to make that call.

On the flip side, if a company with such little regard for security thinks
there's a problem with the Symantec, you've got to wonder. They're basically
saying "we're dangerously incompetent, however even _we_ can see that Symantec
are broken" \- and might actually have a point.

~~~
pfg
Symantec is already getting distrusted and the Web PKI community has decided
on a plan that minimizes user impact while doing that. It's very doubtful that
a company as incompetent as Trustico has the expertise to come up with a more
well-informed decision or a better approach. Certainly attempting to get
50,000 certificates revoked immediately without doing any work to reissue
those certificates prior to that is a terrible approach.

Considering the fact that they recently switched to a new CA partner, a more
likely explanation is that this was a misguided attempt to somehow keep their
existing business rather than let DigiCert pick them up somehow. Obviously it
failed in a spectacular fashion.

------
Bartweiss
> _As the holder of the root certificate used to sign the TLS certificates
> Trustico was reselling, Symantec was ultimately responsible for ensuring
> this requirement was being followed, although in fairness, there was
> probably no way for Symantec to detect a violation._

This is in some sense the scariest part in the whole mess. Root authorities
might have responsibility, but the basic logic of cert resellers makes an
error like this easy and undetectable.

------
linkmotif
> When Rowley asked for proof the certificates were compromised, the Trustico
> CEO emailed the private keys of 23,000 certificates

Voilà, compromised certs!

Couldn’t tell if that was supposed to be the idea. But amusing nonetheless.

~~~
Bartweiss
I can't tell if the logic was "we shouldn't have these keys so the certs are
blown" or "watch, I just compromised the certs!"

Either way, it's very much a case of "my neighborhood isn't safe - because I
keep committing crimes there."

edit: Oh, reading the Mozilla page it's also "we allowed code injection on the
cert generation page". That's even worse.

~~~
pilif
Also, that script was running as root

[https://mobile.twitter.com/taviso/status/969230138956136448](https://mobile.twitter.com/taviso/status/969230138956136448)

~~~
Bartweiss
Hah, _seriously_?

...I wonder whether the keys would have been more or less safe if, instead of
all that, they just uploaded them unsecured to Pastebin. At least then they
might have gone unnoticed, instead of being handed straight to possible bad
actors.

------
dsfyu404ed
"We haven't finished getting to the bottom of this and don't have the post-
mortem yet but this is very bad and you need to revoke all of these ASAP"

"We're not revoking 50k certs without a damn good reason"

"Attached is your reason"

~~~
yjftsjthsd-h
You gotta admit, it's a terribly compelling reason. And I do mean terribly.

------
INTPenis
God I hope my co-workers in the PKI group would say "hell no" if our CEO came
and asked for the private keys. Show some damn backbone.

Though on 2nd thought, maybe having a society with a good social safety net
and collective bargaining for workers would be a good foundation for a
positive outcome in that situation.

If workers are afraid of losing their jobs because they'd lose health
insurance and other vital benefits then they'd be more likely to cave.

Sorry but being from a former socialist country in the top 5 world ranking of
living standards I can't help but make it a social discussion.

~~~
Piskvorrr
The issue wasn't "CEO gets what CEO wants," but that the private keys were
even retrievable by someone else than their holder.

------
lima
The mozilla.dev.security.policy thread linked in the article is a much better
read:

[https://groups.google.com/forum/#!topic/mozilla.dev.security...](https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/wxX4Yv0E3Mk)

tl;dr: Trustico told Digicert that all 50k of their certificates should be
revoked, without specifying a reason as to why they were compromised. 50k is a
LOT of certificates, so Digicert asked for details. Trustico then sent
Digicert the private keys, and Digicert said "Welp, guess _now_ they're
compromised".

~~~
lima
> _Finally -- the CSR /key generation form page incorporates JavaScript from
> at least 5 or 6 different companies (including ad servers), which would
> allow any of those third parties (intentionally or through compromise of
> their own) to capture generated keys. This is a reckless amount of exposure,
> to the point that even if the keys were generated completely inside the
> browser and never exposed to the server (which does not appear to be the
> case), I would consider them compromised at the time of generation._

Also, wow. That's extremely reckless.

~~~
Bartweiss
Jesus. That means it's not just about "offering revocation" and storing them
on the wrong side; even people who didn't use that option are completely
compromised.

~~~
tialaramex
If you generated your own Private Key and used that to make one or more CSRs,
you can show any fool the CSR and no harm will come to you, at least on the
cryptographic side. Upload it to 4chan, print it out on a T-shirt, doesn't
matter.

Any certificates produced are worthless to everybody except you, because you
have the Private Key, and a _competent_ Certificate Authority (in this story
that's Symantec or DigiCert, not Trustico) should only issue certificates for
the names requested in the CSR, not any other names.

A customer who did things sensibly (their own Private Key, make a CSR
etcetera) and used Trustico say in 2016, needn't be concerned that these
development will end with their certificates unilaterally revoked on
Trustico's say-so. They might not want to do any more business with Trustico,
but there's no particular reason this incident should affect their feelings
about DigiCert or Comodo or any of the Symantec brands (Verisign / Thawte /
RapidSSL / etcetera) now controlled by DigiCert.

------
emilfihlman
Seems to me that Digicert is tactically forgetting to say what they said to
Trustico.

Additionally there's not an issue with Trustico having access to the private
keys. It simply means you are trusting Trustico and if you accept that it's no
different from trusting Google or Facebook or your bank with your private
data.

~~~
bllguo
So what's your response to this?

 _> Finally -- the CSR/key generation form page incorporates JavaScript from
at least 5 or 6 different companies (including ad servers), which would allow
any of those third parties (intentionally or through compromise of their own)
to capture generated keys. This is a reckless amount of exposure, to the point
that even if the keys were generated completely inside the browser and never
exposed to the server (which does not appear to be the case), I would consider
them compromised at the time of generation._

