
Distrust of Symantec TLS Certificates - asymmetric
https://blog.mozilla.org/security/2018/03/12/distrust-symantec-tls-certificates/
======
korethr
There's been downvoted comments below that, to me, seem to complain of Mozilla
unfairly blindsiding domain owners.

I disagree, based on my own personal experience. This has been coming for a
while, with plenty of forewarning.

My employer uses certificates from one of Symantec's brands. Last year, we
began to get notices that Chrome et. al. would be distrusting the certificates
issued from the old Symantec root this year, and that we would need to claim
our free replacements issued from the new trust root that is replacing
Symantec's. And it's not been just one notice, we've been getting them
regularly. And in addition to the automatic form emails, the sales rep
assigned our account personally reached out to us to make sure we were getting
this taken care of. We are not a large company, either; we have less than 100
employees. DigiCert is taking this transition seriously.

So IMO, if someone gets blindsided by their website breaking because of the
Symantec root distrust, then they have only laziness and/or incompetence to
blame, whether it's their own, or that of their website operator. Those who's
job it is to make sure that doesn't happen have been warning about it for
nearly a year now.

~~~
tialaramex
Well, a specific example that's going to bite _plenty_ of medium to large
sized corporations is that you have a pattern like this:

Big Corp's Division Z need a cert for their new web site
[https://www.division-z.example/](https://www.division-z.example/) and so Bob
buys it with his corporate credit card, and gives as contact details
bob@bigcorp.example. He buys, let's say, a Verisign SSL certificate.

Six months later Bob leaves to work at some other company. Bob's email is
probably now blackholes, or maybe it's going into a never read "Bob's emails"
folder of Bob's previous manager.

DigiCert, being conscientious, send email to bob@bigcorp.example warning that
Verisign certs were a Symantec product (Symantec bought the brand and CA keys
from VeriSign, or maybe from some other operator which in turn bought them
from VeriSign, long ago)

But that email will either get delivered and silently go unread, or it'll
bounce, but leaving no sign it needs to go to anybody else in particular
instead of Bob.

Months later the cert stops working, as scheduled, and Steve, who is now
responsible for this site, thinks DigiCert is somehow at fault. How were
DigiCert supposed to guess that they needed to contact Steve?

Role accounts are the Right Thing™ so that the email goes to the person whose
job is to care about the subject, not to some random individual who may not
even work there any more. But they aren't enough, you also need mechanisms in
place that ensure it's somebody's job to care, otherwise the role account
emails just go in a folder and are never read.

So, sure, "incompetence" and not DigiCert's fault, nor Mozilla's but it's very
widespread and you should be astonished if you work somewhere that does NOT
have this problem in some form (maybe you have SSL certs locked down, but it
turns out nobody is making sure you pay the electricity bills for outlying
offices, or there's not actually anybody in charge of making sure payroll
happens...)

Returning to SSL/TLS certificates though, any medium sized or larger
organisation ought to be paying attention to the Certificate Transparency
system. To do this properly you need to know all the eTLD+1s your organisation
controls (this may be a non-starter in really sprawling organisations, but
that's already a problem that needs fixing) but then you can know exactly what
certificates exist for your names, who issued them, and why.

In most cases you don't _want_ Bob buying certificates on the company credit
card anyway. Not just because it's cost inefficient (ignoring Let's Encrypt
bigger organisations can get a commercial CA to cut them a deal for a fixed
price or a steep discount per cert in exchange for doing one big Purchase
Order rather than hundreds of credit card payments) but because it's
organisationally a problem, it has security consequences, and it's another
asset you're probably not tracking properly as a business.

~~~
korethr
Oh I know that it's widespread, it happened here; the guy who's job it was to
care left and didn't pass on knowledge when he did. It is because there was a
a role account assigned to comms about TLS certs that the notices were, well,
noticed. I've worked in large orgs where comms are even worse.

But my point was that this is not some reckless sudden move by the browsers or
DigiCert. In my observation, they have been earnestly and diligently working
in good faith so people don't get bit by the transition.

~~~
mehrdadn
Question: with such big news going on for such a long time, how does not even
one person -- whether manager or otherwise -- just take 5 seconds to ask if
they are prepared for it? It's not like the reporting on this was just in some
dark corners of the internet... /some/ people in a given company should have
at least heard something was going to happen to Symantec certificates.

~~~
forgotmypw
Because there are tickets to close before the whistle.

------
lbriner
Wow, I didn't realise how many non-conformances there were with Symantec. It
certainly looks like they had enough chances to get their houses in order and
didn't!

I wonder what the root problem was? They didn't care, they didn't think anyone
would do anything or they are just a large sloppy corporate who can't run a
group properly?

~~~
hannob
My impression: For years it was common practice that when CAs messed something
up it would cause some complains, but no real consequences. The large CAs
thought they were "too big to fail". They thought it would just go on like
that. They (and many others in the industry) didn't believe that Google was
serious when they threatened with browser removal. When they realized Google
was serious it was too late to change their course.

~~~
ashleyn
It also probably helped that the CA business wasn't Symantec's bread-and-
butter.

------
AndrewDucker
Chrome doing likewise: [https://security.googleblog.com/2018/03/distrust-of-
symantec...](https://security.googleblog.com/2018/03/distrust-of-symantec-pki-
immediate.html)

~~~
kzrdude
Chrome did this first, so any wide effect (in yall's organsiations) should
already have been noticed by now, due to this.

~~~
oasisbob
As I understand it, Chrome and Firefox are both operating under the same
policies with about the same timetables. Full distrust doesn't come until
Firefox 63 / Chrome 70 - neither of which are stable releases yet.

The limited distrust (for older certs issued prior to June 2016) was activated
in Firefox 60 and Chrome 66 much earlier this year.

~~~
user5994461
That's factually incorrect.

Google blacklisted ALL Symantec certificates since Chrome 66, in April. The
roadmap announced the change for October but they did it 6 months earlier,
ignoring their own roadmap.

As the OP says, organizations using Symantec have already been hit very hard,
by surprise.

~~~
oasisbob
Do you have a source for that? Google's KB articles still reference Chrome 70
[1], and I can't find another reference to this anywhere else.

Paypal.com is still operating with a Symantec signed cert - issued by
"Symantec Class 3 EV SSL CA - G3". Works fine in Chrome 68. (and not in
Firefox with the security.pki.distrust_ca_policy override set)

[1]
[https://support.google.com/chrome/a/answer/7662561?hl=en](https://support.google.com/chrome/a/answer/7662561?hl=en)

~~~
user5994461
I worked at a large company whose sole supplier was Symantec. Everything has
been blacklisted since April.

~~~
brazzledazzle
Similar environment except for the blacklisting didn’t happen to us. Any
chance some wires were crossed and someone pushed out a policy explicitly
distrusting the CA?

------
mjlee
You can enable this now in Firefox 62 (latest stable)

> In advance of removing all trust for Symantec-issued certificates in Firefox
> 63, a preference was added that allows users to distrust certificates issued
> by Symantec. To use this preference, go to about:config in the address bar
> and set the preference "security.pki.distrust_ca_policy" to 2.

------
sam0x17
It's been obvious to me for quite a while that EV etc only really tells you
"this person paid $$$ to get a cert" rather than anything about the site being
trustworthy or being who it says it is. I wouldn't bat an eye if 10 years from
now major browsers distrusted everything but letsencrypt. Once the letsencrypt
project comes out with a comparable solution for code signing, there is really
no more reason for paid cert companies to exist. They don't check shit and
it's super easy to get fraudulent certs so the value of what they provide is
$0.

~~~
majewsky
I would bat an eye because it's never good to rely on a single point of
failure.

Imagine, for instance, that at some point, LE is the only trusted CA for code-
signing. If its root key gets compromised for some reason, how are we supposed
to move to a new CA if the auto-updaters cannot trust the old CA anymore?

~~~
sam0x17
True. They should maybe consider breaking up their operation into 20 or so
root certs.

~~~
sterlind
As much as I'm tempted to agree, decentralization perhaps risks even more.
Opsec compromise is only one SPOF for a signing service; the state might
coerce the CA into handing over the keys or issuing an intermediate for MitM.

Do you shard across 20 nations? If so, you have to contend with 20
intelligence agencies. It might make more sense to keep all the eggs in one
basket, and have new baskets ready to go if a canary triggers.

------
oasisbob
It would be nice to have the date reflected in the title of this post (March
2018). It's relevant and a nice reminder because the full-distrust is coming
soon, but there isn't anything new here AFAICT.

~~~
craftyguy
Right, nothing new here. I was confused when I read the title, thinking "it
was already distrusted, is there some further amount of distrust possible or
is it groundhog day?"

------
josho
This is huge as it affects GeoTrust, RapidSSL, Thawte, and VeriSign.

I was aware of the Symantec issue and checked my own certs and didn't see
their name, but skimming the article I noticed RapidSSL and thought I'd double
check and sure enough my certs are about to become bogus.

------
natch
When such an error (see article) is presented, it’s a teachable moment not
only for the user surfing the site, but also for the owner of the site who is
going to field questions about the error message from users.

It would be great if Mozilla would include some wording that instills alarm in
site owners about how the site’s information, not just the user’s, is at risk.

For example, most non-malicious site owners probably would not want
maliciously altered content served from or made to appear as though it came
from their site. Yet most of them are, I suspect, unaware that this is a
danger when their site does not use TLS.

It would be great to see Mozilla take this chance to highlight this danger so
that the people most in a position to make changes would become aware of this
additional good reason to do so.

~~~
sebazzz
> It would be great if Mozilla would include some wording that instills alarm
> in site owners about how the site’s information, not just the user’s, is at
> risk.

I'd expect the reseller to inform the buyer about the distrust of these
certificates. The SSL reseller we use informed us before April this year.

~~~
natch
I’m not talking about the distrust though. I’m talking about how to convince
sites why they should give a care about the distrust. Many of them have no
cares to give because they are missing this part of the picture.

------
acd
The CA trust model is totally broken! We pay certificate of authority CA
companies a lot of money for certificates that are not fully confirmed to be
authentic.

Here is a better model. You normally register your company with the local
government corporate registration authority. The local government knows who
this company is, who the corporate registrars are.

One should use a digital ID card to apply to register a company. The founders
sign the registration of the company with their personal digital ID. The
company is then registered. To apply for a domain name for a company should be
digitally signed. The registration government authority should handle
certificate of authentic not foreign CA registrars.

All corporations should use Government CAs all other types of certificates can
then be issued by Lets Encrypt having proper DNS validation in place should
help there too.

All else is broken security by design.

~~~
clon
> One should use a digital ID card to apply to register a company. The
> founders sign the registration of the company with their personal digital
> ID. The company is then registered. To apply for a domain name for a company
> should be digitally signed. The registration government authority should
> handle certificate of authentic not foreign CA registrars.

You must be from Estonia. Or on crack. I'm guessing Estonia.

A lifetime of experience has convinced me that we will have hoverboards, hell,
self-flying hoverboards, before such a scheme becomes the universal norm.

------
SpikeDad
Apple is also distrusting Symantec CAs which started in Aug 2018 - fully by
Fall 2018

[https://support.apple.com/en-us/HT208860](https://support.apple.com/en-
us/HT208860)

------
ghostbrainalpha
It's just insane that they haven't been able fix this issue and get back into
good standing with 6 months warning.

~~~
klodolph
> Root cause: Symantec was willfully disregarding industry regulations by
> issuing trusted certificates without proper authorization.

Source:
[https://sslmate.com/certspotter/failures](https://sslmate.com/certspotter/failures)

“Willfully” is the key word here. The business side of running a CA is
fundamentally at odds with the security side. A few short-sighted decisions by
business-minded managers with their eyes set on profits can completely
eviscerate the security side of a company, and it’s possible that nobody
remains at Symantec who has the combination of security knowledge + internal
political power + will.

There’s also the Trustico fiasco. Trustico revoked 50,000 Symantec-issued
certificates in a shockingly bad way. Because policies allowed for
certificates with compromised public keys to be revoked, Trustico
intentionally compromised the private keys in order to achieve the desired
revocation.

[https://groups.google.com/forum/#!topic/mozilla.dev.security...](https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/wxX4Yv0E3Mk)

> As one of Symantec's former largest partners - my personal opinion and
> personal experience is that Symantec is a company that thrives on
> recklessness and one that I wouldn't trust nor deal with.

You can see more bad behavior here—Symantec is seen as a reckless company, and
Trustico (recklessly) cuts ties with Symantec for its business needs. Both
actors are bad enough that their relationship reflects poorly on both of them.

I’m sure there are some people who could fix this problem in six months, but I
bet they don’t work for Symantec or don’t have the power to do it.

~~~
pitaj
> The business side of running a CA is fundamentally at odds with the security
> side.

Well, as we're seeing here, they can only be out of phase to a certain degree
before both suffer.

------
retlehs
PayPal's site is affected by this

~~~
twblalock
That's surprising, because Chrome has distrusted Symantec certs for a few
months and it's odd that Paypal would not have fixed it by now.

~~~
cpeterso
Chrome and Firefox have only distrusted Symantec certs in their pre-release
versions. The Chrome 70 and Firefox 63 releases in mid-October are when the
hammer will fall.

[https://security.googleblog.com/2018/03/distrust-of-
symantec...](https://security.googleblog.com/2018/03/distrust-of-symantec-pki-
immediate.html)

~~~
user5994461
The hammer fell in April already. Google published a roadmap, the link you
gave, but didn't respect it.

~~~
plttn
Your anecdotal evidence doesn't prove they didn't respect it. I was just able
to load PayPal with a Symantec cert on Chrome 69 on Android, which I realize
is dueling anecdotes, but I'm just reinforcing the status quo, you're the one
making a bold claim.

------
cpeterso
Firefox bug 1484006 has a list of some sites that are still using distrusted
Symantec certs:

[https://bugzilla.mozilla.org/show_bug.cgi?id=1484006](https://bugzilla.mozilla.org/show_bug.cgi?id=1484006)

------
ibotty
[https://www.ssllabs.com/ssltest/analyze.html?d=www.paypal.co...](https://www.ssllabs.com/ssltest/analyze.html?d=www.paypal.com&latest)

I was getting frustrated because I could not visit Paypal (and Ebay) with
Firefox nightly, because thy still use the old Symantec TLS certificates. I
wonder whether there is _any_ major party that should be more concerned with
their certificates than Paypal.

------
King-Aaron
I've got a question about this, as I've been hurled into the admin of our
certs, and honestly I probably shouldn't be the one to do it, lol.

But, we run root certs from DigiCert with the rest of the chain provided by
RapidSSL. However, I have not received any depreciation warnings via email or
any notification in the console output on these sites.

Is there a utility available to "check" our sites, in the same way vendors
provide utilities to verify your cert installations? Or should I assume that
our stack is going to get caught up in this, and move to find alternative
certs now?

~~~
discreditable
Chrome's dev console shows a warning for affected certs.

------
fosco
anyone know if Microsoft is following suit? I searched briefly and could not
find any comments...

~~~
gsnedders
Microsoft rarely comment publicly prior to making changes to trust levels, but
I've heard from several people that they will similarly distrust them.

Apple are yet to state when they will perform the final, total distrust, but
it is planned: [https://support.apple.com/en-
gb/HT208860](https://support.apple.com/en-gb/HT208860)

------
peterwwillis
There's two problems this exposes. One, it takes a very long time to dis-trust
a root cert. This means some people's connections could be exposed for a very
long time.

Two, the method for update shown here is to upgrade your browser. Not everyone
_can_ upgrade their browser. Corporations often lock down browser updates, and
take a very long time to upgrade. Sure, it's fine for you to say "that's the
corporation's fault, too bad for them!" but the users of those companies still
have to suffer in the meanwhile - to say nothing of vendored smartphone OSes
with slow updates...

The other problem with upgrading is legacy computers run very old browsers. I
don't know if you've tried to browse the web on an old computer with a new
browser, but here's a secret: _it doesn 't work_. New browsers have so many
"advancements" that they bloat and crawl on older machines. So effectively,
the means of being able to use the internet requires you to buy a new machine.

If operating systems immediately shipped patched CA lists, and browsers
immediately used them, that would patch the legacy browsers. But it would not
prevent sites from immediately breaking. So no matter what, either we wait
forever to dis-trust certs, or we break sites.

Clearly we need an option C that will allow site owners to upgrade their keys
immediately and without issue, and users to update their CA lists immediately
and without issue. ACME is a good start, but it too has issues that need to be
solved.

In addition, the whole idea of trusting hundreds of root certs to sign for
every domain is just crazy. We need a method to sign certs only by the
organization who actually has responsibility for ensuring the ownership of the
domain: the registrar.

CAs are a great "hack" because they allow browsers to verify certs of sites
without ever putting any onus on the registrar, but they also have a wacky
"trust" model. Any of hundreds of organizations can verify who controls the IP
space of a domain, one time, and issue a magical assurance of this, which is
trusted until the assurance expires in several years. This can be overridden
at any time, and it has nothing to do with who _actually controls the domain_
, which is the registrar and the user who registered it. All the current
system really verifies is who controlled the DNS at one time, which is merely
pointed to by the registrar, and can be hacked independently of the registrar,
meaning there are extra attack vectors.

Yes, lots of little extra "hacks" have been added as stop-gaps, like CAA, and
Certificate Transparency, the now-defunct HPKP, and the future implementation
of cert issuers verifying the DNS and host integrity from multiple ASNs. But
these are just to keep the status quo limping on, and ignore the unnecessary
risks the current design imposes. We need innovation and better design, not
hacks.

~~~
Kalium
It sounds like you could be describing a desire for trust stores in operating
systems that users and system administrators can manage. Is that the case?

Moving on past that, I think you've hit the nail on the head. CAs are about
having a root for tree of trust. CAA and DANE are about ensuring that. I'm
very curious how you imagine an improved, innovative solution to assuring
trust!

I've seen proposals like Namecoin, which are innovative but fail the test of
being improved.

------
yuhong
My favorite story is when Mark Shuttleworth sold Thawte to VeriSign and used
the money to start Canonical. This shows one of the problems of the current
debt-based economy.

~~~
sobani
> Mark Shuttleworth

> Canonical

> debt-based economy

What do any of those things have anything to do with Symantec certificates
being distrusted?

~~~
yuhong
Nothing, just mentioning the history. It is worth mentioning since VeriSign
was one of the first CAs.

------
frandroid
Wow, talk about an obscure warning that tells nothing to the domain owner.

~~~
jleedev
Nobody who runs a website can pretend to be ignorant of this fiasco.

~~~
Spivak
I mean with the huge number of set-and-forget Wordpress installations I could
absolutely believe people are ignorant of this. I mean why would anyone
outside of professional webdevs even care? It's not like this news escaped the
tech bubble.

~~~
fpoling
Those forgotten Wordpress installs may not even have https support.

------
xmichael999
All I can say is Firefox and Mozilla need to go away.

~~~
judofyr
Google, Apple and Microsoft are doing the exactly same thing in this matter:
[https://news.ycombinator.com/item?id=17920093](https://news.ycombinator.com/item?id=17920093)

------
user5994461
FYI: Your site has been unreachable since April if you use Symantec
certificates.

Chrome announced the depreciation for October but they didn't stick to their
own roadmap and blacklisted Symantec since April instead (Chrome 66 release).
[https://security.googleblog.com/2018/03/distrust-of-
symantec...](https://security.googleblog.com/2018/03/distrust-of-symantec-pki-
immediate.html)

~~~
phobosdeimos
Yeah I have seen ecommerce sites that were using Symantec. Nobody cares about
Firefox I guess but once Chrome starts enforcement shit will hit the fan.

~~~
user5994461
Point being. Chrome already started enforcement in April.

