
Symantec CA Response to Google Proposal and Community Feedback - mentat
https://www.symantec.com/connect/blogs/symantec-ca-proposal
======
MichaelGG
>require these applications to be recoded, recompiled and redistributed.

Aka "updated".

The entire post is basically "ok how about we be really good from now on and
suffer no consequences, cause it'd be really shitty for us if we had to be
penalised".

They also posture a lot talking about how big their customers are, almost
boasting about how inflexible and slow these big companies are, as if that's
somehow Google's or general Internet users' problem.

~~~
eykanal
This comment irritates me.

1) The point of the first part of their post is not "we're too big", it's "a
move like this would disrupt the lives of an awful lot of people." Even more
so, consider what would happen if Google were to update Chrome to not accept
these certs. For internal applications, the IT departments of all these
companies—which likely total a few hundreds of thousands of users in
total—would simply mandate that Chrome is no longer an acceptable browser. For
external applications, companies would have to engage in an enormous PR effort
to communicate to customers that their product doesn't work with Chrome. These
companies have done nothing wrong, but they're the one being punished, as
customers don't know and don't care about cert issues.

2) "Updating" can be an enormous effort. As the blog post states explicitly,
there are a lot of use cases where updating a deployed app is very difficult,
almost impossible, or cost prohibitive. Again, remember that the person being
punished here is not Symantec; you're publishing some poor development company
who happened to use Symantec—one of the most popular cert providers on the
planet—for an SSL cert. Not good.

I'm not taking sides on the issue, because I honestly don't know that much
about it. But comments like the above really trivialize the huge impact that
something like this will have.

~~~
tptacek
Part of the premise of Google's plan is that Symantec's customers will
experience disruption. If you poll Symantec's customers, the overwhelming
majority of them will day "no, don't punish our CA". We didn't need Symantec
to tell us that.

On the other hand, Symantec's CA was caught mis-issuing certificates. That
affects everyone, not just Symantec's own customers.

~~~
mentat
The "survey" that accompanied the original customer facing message was so
biased as to be ridiculous. Like "Will Google's proposed change be disruptive,
massively disruptive, or business ending for you?"

------
ahmeni
There is a significant amount of "we will" language in here, rather than "we
have started" or "have already begun". The choice of language speaks volumes
that they only appear to be willing to take necessary audit action if Google
agrees to it, rather than something that needs to be done regardless.

------
durkie
> Embedded devices that are pinned to certificates issued by a Symantec public
> root to communicate to resources over the Internet or Intranet. Replacing
> these certificates would result in immediate failures and the need to recode
> and reimage the firmware for these devices.

> Mobile applications that have pinned certificates. Replacing server
> certificates would require these applications to be recoded, recompiled and
> redistributed.

Are either of these relevant? Isn't google's proposal to eventually have
Chrome stop trusting the Symantec CA system? That change seems like it would
have no effect on 1) embedded devices that aren't running Chrome 2) things
that already are in place that trust the CA.

~~~
ploxiln
Imagine the embedded device or mobile app connects to a
[https://api.example.com/..](https://api.example.com/..). and so does their
web app. They need to use a new certificate for api.example.com, signed by a
different CA root, because when Chrome tries to run the web app it won't trust
the old certificate, nor a new cert signed by the old root. So they get a new
cert signed by a different root so Chrome works, but then the embedded or
mobile apps stop working.

In this case the story is more complicated - I think new certificates signed
by existing Symantec CA roots would be trusted if their duration is 90 days or
less. Historically, most CAs offered one, two, or three year duration.
Presumably, the "big customers" would not accept the need to rotate their
certificates for this purpose in less than 90 days, they need their 3 years
because it takes over 90 days to get something through their bureaucracy. And
presumably there are other things they can't do in 90 days, like update an
embedded device configuration or firmware, or a mobile app.

Of course it depends on the particulars, but of course these "big customers"
can't figure any particulars out in 90 days. But that's why Google proposed a
slow sliding scale that would gradually reduce the max duration allowed over
the course of a year, so random unknown stuff breaks more gradually throughout
the year, not in one huge event like Symantec's blog post suggests.

~~~
X-Cubed
An easier way around this sort of issue is to change their web app to use a
different domain, and only put the new certificate from a different CA on that
domain. It's not necessary to update every device.

Embedded devices => [https://api.example.com](https://api.example.com)
(Symantec cert)

Browsers => [https://new-api.example.com](https://new-api.example.com) (New
trusted cert)

Another approach is that if the devices are old enough they're probably using
a deprecated version of TLS, so the existing
[https://api.example.com](https://api.example.com) server could choose which
certificate to issue based on the version of TLS offered by the client.

~~~
toast0
You have to have planned for this (or be lucky) though. Some people just put
their api on www.example.com; assuming you have clients in the field which
have pinned BIG CA, and you've got HSTS preloaded (because you're forward
thinking). Unless you can detect Chrome N+1 from your existing clients during
the TLS handshake, you have to pick between browsers or existing clients.

If Chrome ships this at the same time as re-enabling TLS 1.3 support, that
would be a decent way to target, since presumably very few of your distributed
clients are using TLS 1.3.

~~~
majewsky
I know it's not going to help companies which pinned Symantec, but that's
_exactly_ why every serious guide to HPKP will have you pin two separate CAs.

------
Aaron1011
> This cohort is an important constituency that we believe has been under-
> represented to date in the public commentary that has been posted to the
> Google and Mozilla boards since large organizations rarely authorize
> employees to engage in such public discussions, particularly in an area
> related to security.

Are these large organizations somehow incapable of putting out official
statements regarding CAs? If they're being 'under-represented', it's their own
fault for not speaking up.

~~~
mdip
To me it was really bold of them to take the tone and stance that they did
with regard to these large organizations. They're making the case as to _why_
Google needed to do what it did ... all of these large organizations that will
be severely impacted by having to replace all of these certificates are
relying on a CA that has shown itself to not be worthy of the trust that CAs
are relied upon to provide.

I'm at a loss for a good analogy so I'll pick a lousy one: it'd be like if
Samsung's response to the exploding Note 7 recall mandate were "Don't do this
-- you're going to keep some of the biggest customers -- users of our phones
-- from being able to make emergency calls/check their e-mail/respond to
texts" ...yeah, but I'm still not OK with the small possibility that carrying
my phone in my front pocket _might_ result in third-degree genital burns.

~~~
skuhn
As it happens, your hypothetical situation was exactly what did transpire:
Samsung wanted to remotely deactivate the phones, but Verizon refused on those
grounds. [http://bgr.com/2016/12/09/galaxy-note-7-recall-verizon-
samsu...](http://bgr.com/2016/12/09/galaxy-note-7-recall-verizon-samsung/)

~~~
mdip
... and here I thought comparing security to burning genitals might have been
too much of a farce, but that takes the cake, there. :o)

------
tyingq
I don't think Google was soliciting for a counter proposal from Symantec. Will
be interesting to see their reply, and whether it's a literal reply or just a
version push of chrome with their original plan.[1]

[1]
[https://groups.google.com/a/chromium.org/forum/m/#!topic/bli...](https://groups.google.com/a/chromium.org/forum/m/#!topic/blink-
dev/eUAKwjihhBs)

Edit: They did ask for community feedback, comments on risk, etc. But they do
already have a timeline. See link above.

~~~
toomuchtodo
"This site does not support Chrome. Please use a browser that does not take
unilateral CA authority action." might very well be the response of orgs
married to Symantec.

As a user, you need your bank (or other large org) more than you need your
preference of browser.

~~~
cjbprime
Mozilla seems in total agreement with Chrome on this, and it also takes
unilateral CA authority action. This is not a negotiation where the orgs have
the right side of the power dynamic.

~~~
toomuchtodo
It's always a negotiation. He who has the least to lose wins.

"Why isn't my browser working??" "So sorry about that, those darn nerds made a
change on us."

I have seen this exact scenario play out. Mozilla and Google might be
respected in the tech community; to most people they're an extremely tiny part
of their life. "Your the IT man, just make it work"

~~~
5ilv3r
You're arguing "Right" vs. "Easy".

~~~
toomuchtodo
Easy there Tex, I'm not arguing. I am simply describing the behaviors between
large orgs in a dysfunctional ecosystem/marketplace.

As you get older, you transition from "this is how it should be!" to "this is
how it works, what is a realistic amount of change that can occur?"

~~~
joncampbelldev
i would say the relevant "this is how it works" is having a globally trusted
CA system that isn't abused by large CAs just because they think no one will
dare punish them. What would any of these large Symantec customers says if in
the future their security was compromised due to mis-issued certificates? I
doubt they'd be thrilled they were saved some dev time getting their
certificates signed by another root in exchange. it's very clear from
Symantec's language here and elsewhere that they wouldn't be doing shit about
this is chrome weren't taking it so seriously

------
bitmapbrother
Symantec is just going through the 5 stages of CA grief. First they were
oblivious to it, then they were angry about it and called Google
irresponsible. They're now at the proposal stage. Next will be depression and
finally acceptance for their incompetence.

------
geofft
> _These customers include many of the largest financial services, critical
> infrastructure, retail and healthcare organizations in the world, as well as
> many government agencies. This cohort is an important constituency that we
> believe has been under-represented to date in the public commentary that has
> been posted to the Google and Mozilla boards since large organizations
> rarely authorize employees to engage in such public discussions,
> particularly in an area related to security._

... well that's their problem, right?

You can't simultaneously say "These are some of the most important
organizations in the world and you'll cause worldwide chaos" and "Won't
someone listen to these poor companies, I am the Symantorax, I speak for the
cohort, for the cohort has no tongues."

~~~
mdip
> These are some of the most important organizations in the world and you'll
> cause worldwide chaos

Thanks for that -- reading this comment connected a few thoughts I was
struggling with.

Their response felt like "if you don't follow through with your plan to punish
us we'll do all of these nice things for you." It reminds me of when my kids
get caught doing something wrong and I get "but I'll do all of my chores and
treat my siblings nice" along with a handful of other promises of future deeds
which I know will be partially or completely not-done. How different the
result would have been if they came to me with the chores done and some/all of
the promises fulfilled already. It would have been much better for Symantec to
have responded with these items in the "completed" category with a smaller
number of "we will's".

But to the quote I highlighted: the issue at hand is the past failures
indicate that "some of the most important organizations in the world" need to
_not_ be trusting Symantec in the first place. They've had a big enough
history of failures as a CA to get the attention of Google, who I'm sure isn't
terribly interested in degrading the browsing experience of their customers.
The choice becomes: (1) Banhammer Symantec, take the resulting pain all at
once, visibly and obviously, and get the problem fixed painfully (but it's
fixed) or (2) Trust that things will be different _this time_ and if they're
not, have another security failure of some kind that results in these
critically important company's customers' data being exposed, which is a
circumstance that will almost certainly go unnoticed for a period of time,
will be covered up if the exposure is minimal and covering up the exposure is
possible or may not be discovered _at all_. The CA system is centered around
_trusting a handful of rather powerful bodies_ with an _enormous
responsibility_ around the overall security of the Internet. It's not a great
model to begin with, but it's what we have and I haven't read about any
alternatives that are viable. The _only_ way to keep a model like this secure
is to ensure that the CAs themselves are actually _trustworthy_ and one of the
more likely ways that will happen is to enforce violations of that trust in
the manner Google is suggesting.

~~~
geofft
Yup. It's very interesting that the argument from Symantec is that their
customers have to buy Symantec certs indefinitely - not that they can't
replace existing certs in servers, but they've hardcoded the Symantec _root_
in enough clients, such that buying certs from a Symantec competitor will
break their clients.

Among other things, if what Symantec is saying is true, this means there is
zero financial incentive for Symantec to behave. The largest certificate
customers in the world have all voluntarily made it infeasible to choose a
Symantec competitor, so as long as Symantec has any product at all of any
quality, the market will keep buying Symantec.

If what Symantec is saying is true, the mere fact that they went to their
largest customers and said "Suppose you had to buy certs from one of our
competitors, what would you do" and they replied "We've never thought about
that before" is itself a major victory.

~~~
mdip
I hadn't read it that way and that's a really good point.

If you have too many trusted-third parties in a trust model like this, you run
into difficulties with enforcement due to scaling problems, but what we're
seeing is the converse of that. Symantec went out and bought up other CAs
(Verisign is the only one I can remember for some reason) and they've become
so large that them losing CA status would be prohibitively damaging to the
internet-at-large. This affects not only enforcement (we don't want to move on
a bad actor because the consequences of enforcement would be more severe than
the consequences of a rule violation), it also negatively affects the trust
model -- now if I'm a bad actor, Symantec is my only target -- even though I
_could_ attack the others and yield some of the same benefits of attacking
Symantec, if I breach Symantec, the rewards are larger by several orders of
magnitude depending on what they store and how they store it.

To digress a little ... today ... I'm not all that surprised that Symantec has
reached sort-of monopoly status in this area. Not because Symantec is a well
managed company (I'm making no statement for or against that despite my
personal opinions), but just because so much of The Internet has turned out
this way. If you would have told me in 1998 that for the very vast majority of
US Internet users, there'd be one search engine, one e-commerce company, one
(expensive) ISP, one e-mail provider, all of it would be secured by services
provided by one company's CA and all of that wouldn't matter much because the
typical Internet experience would basically be a web-enabled, bluer version of
AOL[0], I'd have called you some unfriendly things, but that's where we are.
It's a centralized system operating on decentralized technology.

[0] Oh, and that the "one e-commerce company" was that bookstore web-site and
would be responsible for hosting such a large part of the _rest_ of the
Internet that when they had a major failure and people had that handful of
rare clicks off of FAOLBook, they'd have about a 50% chance that whatever
their target was wouldn't load.

------
lsh123
CA infrastructure business is built on trust. Symantec lost the trust and got
thrown out by Chrome. I hope that as the result Symantec will lose significant
amount of CA business thus making a show case for other CAs to demonstrate
that good processes and trust are key for staying in this business.

------
jupp0r
Notably missing from their post:

This is how we make sure issuing certificates for gmail.com, etc will never
happen again. An external audit every 3 months is not going to fix anything.

------
auditplan
Read the statement carefully. Symantec still does not offer a verifiable audit
trail like Google's Certificate Transparency requires. They just offer to have
3rd parties review the certificates sold to customers. Without a COMPLETE
audit trail Symantec can still continue to secretly issue additional
certificates not part of any audit trail. The whole announcement rather
decreases my trust in Symantec even more.

------
kaishiro
Does anyone else get a blocking "Symantec Connect" and "Loading your community
experience" for a good 10 seconds before it loads? Could just be my mobile
connection but man is that lame.

~~~
joncampbelldev
I've heard elsewhere in this thread that blocking ads/tracking breaks their
site.

------
jwilk
Archived copy, which can be read with JS disabled:

[https://archive.fo/FMemh](https://archive.fo/FMemh)

------
chaz6
The irony is, for an industry based on trust, if this abuse is not penalized,
then the industry as a whole is pointless.

------
dtemp
I'm still trying to figure out if my 36 month wildcard carts from RapidSSL are
going to be distrusted. Their intermediate is signed by GeoTrust which is
owned by Symantec, and a blog post says with Chrome 59, certs with a period
over 33 months will be distrusted. Chrome Canary on v60 shows them still
working.

------
unixhero
"Community" \- as if we are downstream, something easily managed.

I see it as a polite way of saying "You kids".

------
revelation
I love how none of the example "dependencies" they give should be using public
CA in the first place.

~~~
ahmeni
Given some of the internal CA systems I've dealt within the past, I'd almost
prefer a public CA in some cases. Sometimes your internal CA is just the group
with manual access to the certificate provisioning and signing systems with
either no API or some awful re-implemented API.

~~~
revelation
What API do you need? The signing system should be airgapped or you end up
with the same shit that is the public CA system such as roots sitting on
public FTP servers.

It's a bunch of command line scripts because if you are using it any different
way you are probably doing it wrong.

~~~
viraptor
There's more to it than you've seen so far. Think about environments with
hosts starting on demand. You need to encrypt data in flight -> how do you
automatically assign a new certificate to an instance on startup? You can't do
it with an airgapped system.

~~~
revelation
If you're going to automatically create certificates you are shifting the
entire authentication burden to that automated system.

This is the exact issue with the public CA system, their automated processes
have been found lacking and now what they certify can not be trusted.

~~~
viraptor
The topic was internal CAs. Those have completely different requirements and
possibilities than public ones. The authentication will look completely
different as well. You just can't compare those issues.

And you're missing the point I raised as well - if you have hosts coming up
automatically, you either need automatic certificates, or shared keys. As long
as you rely on SSL, there's no other way to go.

(For example of authentication: public CA verifies your domain, internal CA
can ask the hypervisor to verify a token on your instance - not remotely in
the same class)

------
varunvkrishnan
Incompetence​ needs to be penalized. Period. Symantec's reasoning is pathetic
and infact a veiled blackmail attempt.

