
WoSign Incidents Report Update [pdf] - xnyhps
https://www.wosign.com/report/WoSign_Incident_Report_Update_07102016.pdf
======
terom
Awesome use of load balancing for request retry across multiple backend
servers:

> This was caused by the CMS (Certificate Management System), when it sent the
> signing request of the certificate to the signing server A, which had no
> response, then the CMS sent it to the other newly added signing server B.
> After a while the signing server A signed the certificate and sent to the
> CMS and also to the subscriber, then the subscriber installed the cert in
> its website and hat's why Censys recorded this certificate; in the meantime,
> the signing server B also signed this certificate some time later (in
> seconds) and sent it to the CMS, the CMS accepted it and rewrote it in the
> DB.

> This issue happened after adding another signing server on Jan 5th 2015, and
> found it on April 9th 2015. When had the two signing servers added a load
> balancer, but the configuration was not properly done because it didn't lock
> the request.

Mind you, that's a perfectly legit technical bug. Maybe they were using nginx
for load balancing POST requests? :)

[https://news.ycombinator.com/item?id=11217477](https://news.ycombinator.com/item?id=11217477)

~~~
therein
Yup, retrying PUT/POST is a bad idea unless you're really careful.

------
zamber
Cross linking the earlier WoSign scandal reports:
[https://news.ycombinator.com/item?id=12389573](https://news.ycombinator.com/item?id=12389573)
[https://news.ycombinator.com/item?id=12582534](https://news.ycombinator.com/item?id=12582534)
[https://news.ycombinator.com/item?id=12617659](https://news.ycombinator.com/item?id=12617659)

So if I get it right, WoSign will cease to exist as a CA given the 1y
probation proposed by Mozilla and the general distrust that will follow?

~~~
inimino
That was Mozilla's action plan, this is Wosign's response, now it is back in
Mozilla's court.

------
ctz
> "Many customers in China find it important to use a domestic CA for purposes
> of security."

That's not how the CA system works. Your security is unaffected by what CA you
choose; it is invariably the minimum of all trusted CAs.

~~~
hueving
It certainly is if you strip down the certs trusted.

~~~
ctz
Well, true. But that would mainly affect availability :)

------
nikcub
Interesting aside that came out of this case is the issue of cross-signing
intermediary and root certs and it not being disclosed.

The WoSign roots were cross-signed by[0] Comodo and StartCom (owned by WoSign,
but we didn't know that), so even with WoSign roots being revoked, there would
still be a verification path.

Nice to see that now there is an effort to disclose all of these[2][3],
and[1]:

> Mozilla now requires the disclosue of all intermedidate certificates,
> including those cross-certificates.

[0]
[https://wiki.mozilla.org/CA:WoSign_Issues#Cross_Signing](https://wiki.mozilla.org/CA:WoSign_Issues#Cross_Signing)

[1]
[https://groups.google.com/d/msg/mozilla.dev.security.policy/...](https://groups.google.com/d/msg/mozilla.dev.security.policy/k9PBmyLCi8I/5c1c6L7JFwAJ)

[2] [https://crt.sh/mozilla-disclosures](https://crt.sh/mozilla-disclosures)

[3]
[https://secure.comodo.com/products/publiclyDisclosedSubCACer...](https://secure.comodo.com/products/publiclyDisclosedSubCACerts)

------
cperciva
Is the CEO stepping down because fraudulent certificates were signed, or is he
stepping down because they got caught?

~~~
mcbits
He is being "relieved of his duties as CEO" according to the report. Saying
he's "stepping down" is IMHO like saying a fugitive "turned himself in" when
police arrived at his house with an arrest warrant. (Although I don't think he
is being accused of anything criminal here.) Most of the incidents appear to
stem from technical errors and poorly implemented processes, but the whole
list points to his inability to lead this kind of company effectively.

I'm always skeptical of whether replacing a few executives can actually fix
the cultural problems that were fostered/ignored in a company over time. The
new leadership has to overcome a lot of inertia, and that's assuming they're
any better than the old. They're also still answering to the same investors
and pressures.

------
koolba
When you have a business that effectively prints money, why do something this
stupid?

I also wonder how much more effort they thought it was to write the code to
backdate the certs (rather than use "now") v.s. code for upgrading to SHA-256.

~~~
0x0
It's not about a lack of support for SHA-256 on the CA side. But backdated
SHA-1 certs are in demand because they allow other systems to keep working,
systems like embedded/PoS computers that lack support for SHA-256, while
simultaneously dodging errors on modern browsers that only allow SHA-1 certs
if they were issued before a certain date.

Wosign was certainly capable of issuing SHA-256 certificates. But the
customers needed SHA-1 certificates with a backdated issue timestamp. And
Wosign was willing to fake the issue timestamp on new certificates, probably a
lucrative market because no other reputable CA would be willing to do so.

~~~
koolba
> Wosign was certainly capable of issuing SHA-256 certificates. But the
> customers needed SHA-1 certificates with a backdated issue timestamp.

So they had a special side deal to get backdated certs? There's no way they
were doing this for everybody or for "regular price" right?

~~~
kuschku
Interestingly, several western sites also have side deals to get SHA1 certs
backdated or otherwise whitelisted, including Google and CloudFlare.

~~~
0x0
This is a very interesting allegation. Do you have a link to any such
backdated or whitelisted certificates? Where is the whitelist, and how does it
work? Does all major SSL libraries, browsers, http clients (openssl, libressl,
polarssl, gnutls, microsoft, apple, java, etc etc) implement the same
whitelist?

~~~
kuschku
There is a public exception process to handle SHA1 certificates, and for the
rest, they get special certificates of old root certificates that are only on
older devices.

Obviously, all these options are not available to anyone except a handful of
large companies.

~~~
0x0
That's at best off topic again. We're discussing the actions and
trustworthyness of a currently trusted root CA here.

Edit: The exception processes also do not involve fraudulently backdating
anything. Which is kind of a big deal when you are in the trust business.

------
nly
A shame. WoSign were super generous with their free certificate offering long
before LetsEncrypt was a thing. They were a a handy alternative.

We should be thanking them for their free certs, and thanking them again now
for giving us another example of how the PKI is a farce. The chances are there
are a bunch other 'reputable' CAs out there playing these games.

~~~
bigiain
Interestingly - StartCom's free certs used to only have 1yr validity - one I
got last week is good for 3 years. Cynical-me suspects they're doing that to
improve their chances of sitting out a limited 1yr timeout. It'd be almost
indistinguishable from a death penalty if all their existing client certs
expired while they were unable to renew them - I wonder how long they've been
signing certs for 3yrs instead of 1?

------
dredmorbius
This story is a bit of a mess to make sense of coming in cold and reading a
Google Groups summary. Here's my read, which may help clarify the story for
others.

Mozilla have an excellent explanation document covering the backdated certs in
detail here:
[https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBG...](https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBGv6szuSB4sMYUcVrR8vQ/preview)

(Thanks to @xnyhps for the link in a reply to this comment.)

WoSign, described elsewhere as China's largest certificates authority, are a
CA who have been found to have backdated SHA1 ceritificates to work around
browser restrictions on SHA1 cert issueances. SHA1 is no longer considered
secure. Resolution of that issue is discussed in new
mozilla.dev.security.policy Usenet group peered by Google Groups:
[https://groups.google.com/forum/#!msg/mozilla.dev.security.p...](https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/1XI3Y7PJ1Uc/qc9PvezXFwAJ)

A better source for WoSign's update to the story is in the PDF posted to the
newsgroup, here:
[https://www.wosign.com/report/WoSign_Incident_Report_Update_...](https://www.wosign.com/report/WoSign_Incident_Report_Update_07102016.pdf)

Titled "WoSign Incidents Report Update". Which is even less descriptive than
the title presently given on this HN post, though perhaps what HN posting
guidelines prefer. I'll let @dang wrestle his conscience on that one.

In that document are several issues listed, the one relevant to this HN post
appears to be:

"9\. Issue S: Backdated SHA-1 Certs (January 2016)

"WoSign has issued certificates after January 1st 2016 but backdated the
notBefore date to be in December 2015. This has the effect of avoiding the
blocks in browsers regarding SHA-1 certs issued after January 1st 2016. The
number of certs affected is probably 67, but may be a few more or less."

Following down from there, several corporate restructuring steps are
mentioned, including:

 _360’s Corporate Development team has been notified to execute the process to
legally separate Wosign and Startcom and to begin executing personnel
reassignments. StartCom’s chairman will be Xiaosheng Tan (Chief Security
Officer of Qihoo 360). StartCom’s CEO will be Inigo Barreira (formerly GM of
StartCom Europe). Richard Wang will be relieved of his duties as CEO of
WoSign._

There is background on the story from:

"WoSign Mis-Issued SHA-1 SSL Certificates [Updated]" (August 24, 2016)
[https://www.thesslstore.com/blog/wosign-mis-issued-
sha-1-ssl...](https://www.thesslstore.com/blog/wosign-mis-issued-sha-1-ssl/)

"Mozilla Ready to Ban WoSign Certificates for One Year After Shady Behavior"
(September 26, 2016)

The second article details Mozilla's issues with WoSign, including purchase of
an Israeli CA, StartCom [http://news.softpedia.com/news/mozilla-ready-to-ban-
wosign-c...](http://news.softpedia.com/news/mozilla-ready-to-ban-wosign-
certificates-for-one-year-after-shady-behavior-508674.shtml)

I'm not claiming anything other than a 15 minute familiarity with the
situation here. I may have heard earlier rumblings but really haven't followed
this at all and wasn't consciously aware of particulars.

~~~
xnyhps
The best place to start with this story would be this very well writen
document by Mozilla:
[https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBG...](https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBGv6szuSB4sMYUcVrR8vQ/preview).
This is WoSign's first public response after that document was published.

~~~
dredmorbius
Thanks, I've updated my comment with that.

------
gogopuppygogo
Mirror of the PDF version of the incident report if they take down the
original or it goes down:
[http://clicky.strapr.com/3g1s1y1k0Y2J](http://clicky.strapr.com/3g1s1y1k0Y2J)

------
ajdlinux
Direct link to PDF report:
[https://www.wosign.com/report/WoSign_Incident_Report_Update_...](https://www.wosign.com/report/WoSign_Incident_Report_Update_07102016.pdf)

------
tarancato
So much money they are making and they can't hire anybody who can write
English properly or write a PDF that is not composed of a handful of font
faces and sizes.

I also find it funny they have fired the CEO (the PDF does not say he stepped
down voluntarily) but he's the one sending that link to the mailing list. I
call bs.

~~~
johnnyfaehell
> So much money they are making and they can't hire anybody who can write
> English properly

If you're going to make fun of someone's English then you should probably
write better English yourself. At the end of the day English isn't their first
language and they'll probably very rarely have to use English. I think some
slack should be given on this front.

~~~
Normal_gaussian
tarancato isn't making any money from these comments, so it seems reasonable
that they are not using some form of proofreader.

~~~
johnnyfaehell
I am not bothered by their English or using a proof reader. I just think it's
hypocritical to throw stones at people for not having good English when they
don't operate in an English speaking market when yours isn't perfect either.

Also do you honestly think that was written by the business team, do you think
the business team would have been able to properly edit a technical document
in another language?

~~~
cshimmin
He's not throwing stones at people, he's throwing them at the company, WoSign.
FWIW, I agree. They should have gotten (i.e. paid) someone to write a coherent
document for this important communication.

~~~
johnnyfaehell
> FWIW, I agree. They should have gotten (i.e. paid) someone to write a
> coherent document for this important communication.

Seriously you'll be hard press to find a paid translator that can translate
technical documents.

Also the document is actually reasonable coherent. Picking on the language
level of the document over the technical content of the document seems kind
petty.

~~~
michaelt
The CA Browser Forum puts out a lot of documents in english - often full of
both technical terms and complex multi-layered clauses and subclauses.

You'd hope they'd have a good technical translator already on their payroll.

~~~
uxp
I'm capable of reading and writing a number of languages, but I'm not
proficient at it. This also means that I probably have the writing level of a
lower grade school child in those languages, and if I was to try and speak,
I'd sound like I was mentally disabled since my vocabulary and general
knowledge of the language is lacking.

Japanese is one. I've many times read comments written in Japanese in the Ruby
language source code and understood their meaning, but there is absolutely
zero ability for me to visit that country and be able to interact with anyone.

The author of this incident report is leagues more capable than I am in
writing a report in a foreign language. They didn't insult my mother, or my
dog, or call me names by accident. I don't see the justification in insulting
their English.

------
lifeisstillgood
So I am wondering how this mess got started

Is it just incredible incompetence on WoSign's part or is there a stronger
reason why China's largest CA is trying to keep issuing weak certificates? Is
is tinfoil-hattery to assume that the Chinese government is not ready for SSL
to start getting unreadable again?

~~~
nikic
> The charged 42 backdated certificates are an intentional activity that we
> try to help the desperate customers since there are more than 3M users still
> using Windows XP sp2 in China. We like to make things simple that don’t
> realize how serious this solution was.

I think this is all there really is to it. Some customers wanted to have
backdated SHA1 certificates so they could continue supporting platforms that
do not accept SHA256 certificates. WoSign decided to oblige them.

