
Certificate Revocation Issue - directionless
https://downloads.globalsign.com/acton/fs/blocks/showLandingPage/a/2674/p/p-008f/t/page/fm/0
======
andrem
The handling of this has been quite terrible. The article is the first "good"
communication about an event that started 7 hours prior. And according to the
communication will take another 4 days to solve completely.

Before this, the Technical Solutions Director tweeted solutions that did not
work for end users, but highlighted a typical IT centric approach to problem
resolution ("Works, What's the problem?") [1]

For anyone not already aware, check out Let's Encrypt. I am evaluating it for
about 200 domains now in earnest after having it on my horizon for some time.
At least to have it ready as a fallback. [2]

Getting 200 EV certificates in a hurry from a different CA has been costly
this morning.

[1] -
[https://twitter.com/vanbroup/status/786548172864626690](https://twitter.com/vanbroup/status/786548172864626690)
[2] - [https://letsencrypt.org/](https://letsencrypt.org/)

~~~
mnx
Let's Encrypt doesn't offer EV, and AFAIK, they don't plan to.

~~~
nailer
CertSimple only does EV (and we happily recommend Lets Encrypt, CloudFlare,
Heroku and other free sources if you only need a DV cert).

We're 18 months old and the main reason we exist is to make the identity
verification EV adds not a pain in the arse.

Here's a list of ways we do EV differently from other providers:
[https://certsimple.com/about](https://certsimple.com/about)

I've been on HN for nearly as long as pg has so feel free to give me a shout
here if you have any questions. It's past midnight in the UK right now so I'll
answer them tomorrow morning. If you prefer email: mike@certsimple.

~~~
AbacusAvenger
Tried it out for the heck of it. Unfortunately Dun & Bradstreet has old
address information for my company and their account creation system is broken
(error 'CEPDEV101' when creating an account). So the "3 hour" validation will
take a lot longer, through no fault of CertSimple's.

~~~
nailer
Hi Steven! Check the 'Next steps' email which has specific instructions for
your company - we don't rely on Dun & Bradstreet to update.

------
samuelb
In case you can't access their website

Dear Valued GlobalSign Customer,

As most of you are aware, we are experiencing an internal process issue
(details below) that is impacting your business. While we have identified the
root-cause, we deeply apologize for the problems this is causing you and
wanted to ensure you that we are actively resolving the issue.

GlobalSign manages several root certificates and for compatibility and browser
ubiquity reasons provides several cross-certificates between those roots to
maximize the effectiveness across a variety of platforms. As part of a planned
exercise to remove some of those links, a cross-certificate linking two roots
together was revoked. CRL responses had been operational for 1 week, however
an unexpected consequence of providing OCSP responses became apparent this
morning, in that some browsers incorrectly inferred that the cross-signed root
had revoked intermediates, which was not the case.

GlobalSign has since removed the cross-certificate from the OCSP database and
cleared all caches. However, the global nature of CDNs and effectiveness of
caching continued to push some of those responses out as far as end users. End
users cannot always easily clear their caches, either through lack of
knowledge or lack of permission. New users (visitors) are not affected as they
will now receive good responses.

The problem will correct itself in 4 days as the cached responses expire,
which we know is not ideal. However, in the meantime, GlobalSign will be
providing an alternative issuing CA for customers to use instead, issued by a
different root which was not affected by the cross that was revoked, but
offering the same ubiquity and does not require to reissue the certificate
itself.

We are currently working on the detailed instructions to help you resolve the
issue and will communicate those instruction to you shortly.

Thank you for your patience.

Lila Kee Chief Product Officer GMO GlobalSign

US +1 603-570-7060 | UK +44 1622 766 766 | EU +32 16 89 1900
www.globalsign.com/en

~~~
liquidise
One thing that shocks me about this is the browser responses. Opera would not
allow me to visit the sites affected, citing a revoked cert and possible
attack. While I appreciate the warning, it was odd to see the site fully
disabled.

Safari on the other hand silently failed the https auth and served the page
regardless. Concerning behavior for a revoked cert.

~~~
wichert
That might depend on the specific Safari version. I am using Safari 10 on
macOS Sierra, and it would not allow me to access any site using one of the
affected certs: instead it would just serve an error page.

~~~
pilsetnieks
I think that was more because of the site, not the certificate - the grey
failure page would appear for sites that use HSTS[0]. If the site didn't use
HSTS or if you visited it for the first time, you'd get an ordinary
certificate error alert.

[0] [https://en.wikipedia.org/wiki/HSTS](https://en.wikipedia.org/wiki/HSTS)

------
SysArchitect
This worked for me on OS X:

    
    
      sqlite3 ~/Library/Keychains/*/ocspcache.sqlite3 'DELETE FROM ocsp WHERE hex(serialNum) IN ("040000000001444EF03E20", "040000000001444EF04247");'
    

What a pain in the behind :/

~~~
gdeglin
That didn't work for me, but this did:

    
    
      sudo rm /var/db/crls/*cache.db
    

From:
[https://support.globalsign.com/customer/portal/articles/1353...](https://support.globalsign.com/customer/portal/articles/1353318-view-
and-or-delete-crl-ocsp-cache)

~~~
SysArchitect
That removes all caches for OCSP. That seems a little heavy handed IMHO.

but if it works.

------
0x0
How does their SSL warranty play into this? Will they have to pay $1.250.000
for each OrganizationSSL certificate?
[https://www.globalsign.com/repository/globalsign-warranty-
po...](https://www.globalsign.com/repository/globalsign-warranty-policy.pdf)

~~~
andrem
Interesting - 3.3 seems to be applicable here:

"3.3 Intentional or accidental errors This Warranty Policy warrants against
intentional or accidental errors introduced into a Certificate by any
personnel of a GlobalSign RA or a GlobalSign LRA due to failure to use
reasonable care as required under the CPS."

Not a lawyer, but will push this with our legal team for sure.

This could destroy GlobalSign, so there must be a clause to exempt them from
liability ... ah, here it is:

"8.1 Total amount for warranty exhausted When the maximum limit allocated for
warranty payments is exhausted, GlobalSign shall have no further obligation to
refund any Beneficiary, unless required otherwise by applicable law."

~~~
delinka
"introduced into a Certificate"

They didn't introduce any errors _into_ a certificate. They affected the
certificate chain, but not the certificate directly. I'm certain this is how
their own lawyers view this.

~~~
0x0
If _THIS_ isn't covered by warranty, what would it take to invoke this
warranty?

Btw I've always thought these SSL insurance warranties sounded like a scam.

~~~
nathanaldensr
It seems likely we'll find out if they are, at least for GlobalSign's.

------
johnjuuljensen
I bought new certificates, for a new set of domains, through AlphaSSL today.
One hour later customers starts calling, complaining about revoked
certificates. Initially I assumed they had screwed up somehow and revoked our
old certs, but after reports saying that it worked with some browsers and
failed on others I started googling for recent related issues, and found out
about GlobalSign.

Man, do they suck at communication. We're now 14 hours into the incident. 6-7
hours ago they posted a trouble shooting guid, promising new intermediate
certificates for AlphaSSL and I've just been informed by their support that
it'll be another hour before they're ready.

It's now 02:00 in dk, so I can expect the new certs at 3 and be done by 4.

Fun night.

Thanks GlobalSign.

P.S. Also thanks to the guy who made their marketing department stop tweeting
iot crap while this is going on. That pissed me off.

------
gdeglin
This broke bootstrapcdn.com, bootstrap's official CDN. So the effects are
extremely widespread even for non-globalsign clients.

------
byuu
Since this may take days to resolve completely, here's a temporary workaround
for Chrome users -- launch your browser with this flag:

    
    
        chrome --ignore-certificate-errors
    

You'll still know when sites have bad certificates due to the red line drawn
through the [https://](https://) portion of the URL. But you will be able to
access these sites. But be sure to stop using this flag as soon as you can. It
could leak secure cookies to a MitM with a fake cert. _Very_ slim odds of
that, but still undesirable. Yet Chrome left us with no other option here.

I must say though, I'm increasingly frustrated by software vendors trying to
strip away control over our own machines. There is no option at all from the
standard error message, even under advanced, to indicate that you know about
this problem and wish to proceed. And I'm sure it's only a matter of time
before they remove this command-line option as well.

I get that novices probably need some protection, but I really wish there were
a way to say that, "yes, I _really_ do know what I'm doing, please stop
treating me like a toddler." So instead, I'm forced to use a much less safe,
hidden command-line option or be locked out of various sites for four whole
days.

~~~
pfg
> I must say though, I'm increasingly frustrated by software vendors trying to
> strip away control over our own machines.

I get the sentiment, but if you look at how common malware is on Windows
versus (walled-garden) platforms like iOS, it becomes pretty clear that it's
the most effective way to keep users safe. Inconveniencing tech-savvy folk is
a small price to pay for that, IMO.

> I get that novices probably need some protection, but I really wish there
> were a way to say that, "yes, I really do know what I'm doing, please stop
> treating me like a toddler."

Typing "badidea" when faced with the interstitial should do the trick.

~~~
byuu
But that's just it, I'd have to download the source to Chromium, find where
this error is thrown, and add a UI element to allow me to bypass the error.
That's ... a whole lot more than just inconveniencing me. Flipping a switch in
chrome://flags would be an acceptable inconvenience.

This command-line switch works, but it requires accepting a larger risk with
respect to secure cookies and MitM attacks.

~~~
pfg
That's why "badidea" exists.

~~~
byuu
... wow, I thought you were saying that would be a good way of implementing
it. Didn't realize that _actually_ worked o_O

Thanks, that's certainly superior to the command-line flag! A touch
patronizing, but I guess it could be worse :P

------
gcr
What reuptable certificate authorities are left besides LetsEncrypt?

~~~
CamperBob2
Symantec is expensive but they seem to have their act together.

~~~
mnordhoff
Not particularly.

[https://security.googleblog.com/2015/09/improved-digital-
cer...](https://security.googleblog.com/2015/09/improved-digital-certificate-
security.html)

[https://security.googleblog.com/2015/10/sustaining-
digital-c...](https://security.googleblog.com/2015/10/sustaining-digital-
certificate-security.html)

~~~
agwa
Also:

[https://groups.google.com/forum/#!topic/mozilla.dev.security...](https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/VjRDU_VAOf8)

[https://cabforum.org/pipermail/public/2016-January/006519.ht...](https://cabforum.org/pipermail/public/2016-January/006519.html)

[https://groups.google.com/d/msg/mozilla.dev.security.policy/...](https://groups.google.com/d/msg/mozilla.dev.security.policy/siHOXppxE9k/OOBlANs4BAAJ)

[https://security.googleblog.com/2015/12/proactive-
measures-i...](https://security.googleblog.com/2015/12/proactive-measures-in-
digital.html)

[https://sslmate.com/blog/post/ct_redaction_in_chrome_53](https://sslmate.com/blog/post/ct_redaction_in_chrome_53)

[https://www.agwa.name/blog/post/domain_validation_vulnerabil...](https://www.agwa.name/blog/post/domain_validation_vulnerability_in_symantec_ca)

And all of this is just from the last year.

Symantec is one of the worst CAs there is. Avoid them like the plague.

~~~
mnordhoff
Now i have ten tabs open catching up on tweets from you and @sleevi_

[https://bugzilla.mozilla.org/show_bug.cgi?id=1261919](https://bugzilla.mozilla.org/show_bug.cgi?id=1261919)
is fun

------
directionless
Updates
[https://news.ycombinator.com/item?id=12712279](https://news.ycombinator.com/item?id=12712279)
[https://downloads.globalsign.com/acton/attachment/2674/f-06d...](https://downloads.globalsign.com/acton/attachment/2674/f-06d2/1/-/-/-/-/globalsign-
incident-report-13-oct-2016.pdf)

------
ComodoHacker
>some browsers incorrectly inferred that the cross-signed root had revoked
intermediates, which was not the case.

So it's not their failure but browsers'? Which ones then and what versions?

~~~
gcp
At least Safari and IE, it seems.

~~~
duaneb
Beta chrome as well. Stable chrome shows revocation if you click into details;
beta marks the cert as revoked.

~~~
eridius
Stable Chrome for me was definitely broken. And in fact, trying to view the
details on the revoked certificate caused Chrome to beachball.

~~~
duaneb
Weird! Completely different than my experience.

------
jolux
I didn't even know that this was what was going on until I saw this on HN and
I've been experiencing it all day today.

~~~
akkartik
I thought it was Opera's fault: sites would load fine on Firefox or Chrome,
but not on my primary browser.

------
typicalrunt
This killed one of our main webapps, but since the site was hosted on AWS, I
provisioned an ACM certificate (free) in about 5 minutes and manually applied
it to the ELB listener. Couldn't have been easier.

Today's weirdness and communication around it has made me trust Globalsign a
lot less now.

------
nodesocket
Affecting SoundCloud, though I'm not seeing any ssl issues on my end.

[http://status.soundcloud.com/day/2016/10/13](http://status.soundcloud.com/day/2016/10/13)

------
okket
Revoking keys (and the necessary checking that it requires) will never work
IMHO. The only way to solve this problem is short key signature lifetimes,
automated signatures and, if compromised, just no re-signature.

Let's Encrypt is one way, although the lifetime with 3 months is a bit too
long. One month or even less would be better. Additional verification and
checks via DANE/DNSSEC help to shorten the impact of a compromised key.
Constant checking for revocations do not. Again: IMHO.

~~~
BillinghamJ
If you're going to do that properly, why not more like 1 hour?

~~~
ceejayoz
That'd be pretty rough if Let's Encrypt had an outage.

~~~
duskwuff
It'd also be incredibly hard on Certificate Transparency servers.

------
kirankn
This issue affected me as soon as I upgraded my chrome to V54 yesterday. It
broke all my CDN hosted files which were using the AlphaSSL Wildcard
certificate. We were experiencing low traffic and realized this may have been
the issue. Got into Chat support with ssl2buy who provided me with a Comodo
Wildcard certificate. It was a pain to recreate and install the certs
everywhere. But we didn't want to lose any more traffic.

------
Rufal
[https://support.globalsign.com/customer/portal/articles/2599...](https://support.globalsign.com/customer/portal/articles/2599710-ocsp-
revocation-errors---troubleshooting-guide)

They are in the process of fixing certificates. I know of many that are now
ok. Too bad I already bought new ones, won't be going back,.

------
ziggrat
Once we understood that this was caused by the Chrome update we contacted them
and we got a free Komodo certificate from AlphaSSL.

------
novaleaf
it looks like this is impacting sites hosted on google cloud using the load-
balancer (you upload your cert, and I'm using a globalsign cert). I am getting
502 errors via mobile but via desktop it's fine.

anyone else use globalsign via google load balancer who can confirm?

~~~
novaleaf
downvoted eh? tough crowd.

FYI the google load balancer issue seems to be real, but not related.

[https://status.cloud.google.com/incident/compute/16020](https://status.cloud.google.com/incident/compute/16020)

------
sinatra
I've been wanting to switch all our certificates to Let's Encrypt for almost
one year. But, there was always something else which needed my attention more
urgently.

So, I guess, thank you Global Sign for forcing me to finally make the switch!

------
rahkiin
Luckily I don't have many tenants yet: I replaced my wildcard AlphaSSL with a
couple of Lets Encrypt certs to fill the 4 day gap. Four days of
inaccessibility is just not acceptable.

This is a real mess.

~~~
byuu
Did the issue directly affect your AlphaSSL site? So far mine's continued to
work. May have dodged a bullet by using OCSP stapling, though.

~~~
rahkiin
Yes it did. I have had multiple emails from clients who could not access the
platform.

------
gamache
My company uses Google Firebase and Fastly CDN, and we're affected by this
issue through both hosts.

------
sparky_
I assume this explains why so many people are experiencing random SSL issues
today.

------
terom
Easy, they just need to revoke the revocation.

What do you mean, X.509 doesn't support that? :P

