
Revoking certain certificates on March 4 - teddyh
https://community.letsencrypt.org/t/revoking-certain-certificates-on-march-4/114864
======
terom
For context in terms of what caused this, here's the PR which I assume fixed
the bug in question:
[https://github.com/letsencrypt/boulder/pull/4690](https://github.com/letsencrypt/boulder/pull/4690)

It looks like a nasty and subtle pass-by-reference of a for-range local
variable, although I'm having trouble figuring out where the reference is
stored:
[https://github.com/letsencrypt/boulder/blob/542cb6d2e06e756a...](https://github.com/letsencrypt/boulder/blob/542cb6d2e06e756aeeca964965ae6fc04c3dc62a/sa/sa.go#L1409)

I've spent plenty of time hunting down similar bizarre bugs in Go code as
well, where the called function ~implicitly~ takes a pointer to the iteration
variable and stores it somewhere. Each iteration of the for loop updates the
stack-local in-place, and later reads of the stored reference will not read
the original value. It's hard to spot from the actual call site :/

EDIT: This was an explicitly taken `&v` reference, but the same thing can also
happen implicitly, if you call a `func (x *T) ...` method on the variable.

~~~
heavenlyblue
People say Rust’s borrowing rules are only useful for multi-threaded
environments. This is one of those issues that they are supposed to solve.

~~~
Thaxll
I'm not sure I understand, it's a business logic bug, would have happen in any
language.

~~~
chc
How do you figure this is a business logic bug? It looks like a pretty clear-
cut implementation bug to me. Rust would 100% have caught this bug, and in
fact I'm pretty sure it would have caught the bug at least two different ways:

1\. The reference outlives the original value.

2\. You can't have multiple mutable references at the same time.

~~~
cesarb
> 2\. You can't have multiple mutable references at the same time.

If I understood the issue correctly, only one of the references would be a
mutable one, so the way Rust could have caught the bug would instead be the
related rule: "you can't have an immutable reference and a mutable reference
at the same time".

------
wbond
Emailing users and giving them only 24 hours before their certs are revoked
seems very unreasonable. Say you are down and out with a stomach bug or on
holiday for a day or two.

My understanding is that the 90 day lifetime is largely because revocation can
be thwarted. Thus the practical difference between 24 hours and one week is
meaningful for server admins, but inconsequential if someone is staging an
attack.

~~~
pfg
The Baseline Requirements for publicly-trusted CAs (section 4.9.1.1) require
timely revocation of mis-issued certificates - either 24 hours or 5 days
depending on the reason. I'm not entirely certain which is applicable here,
but I'd assume Let's Encrypt's hands are tied in this case.

~~~
wbond
That is a very useful bit of info. I guess if the mis-issuance happened on
Friday evening PT, then fives days is March 4th.

~~~
thenewnewguy
The misissuences have happened over the last several months (since at least
December 2019), but it does seem that it was _discovered_ on Friday.

------
talkingtab
So lets see, the deal is that I get free certificates on any and all of my
domains. I get an easy way to install and update my certificates that works
with my nginx services. I can move the service to a new address and instantly
get a new certificate. The certificates are universally trusted. I get
notified by email when there is a problem along with a way to detect and fix
the problem available to me.

I'm speechless. I used to pay real money to get certs without half the service
I get now for free.

Thanks letsencrypt.

~~~
wowaname
>I get an easy way to install and update my certificates

You're lucky, because it took me forever to devise a renewal system that met
my requirements of being able to renew certificates for domains that are used
across multiple machines, both physical and virtual, many of the virtual
machines which share IPv4 behind NAT thus cannot sanely use HTTP-01 renewal.
Fine, use DNS-01, but: having to deal with my strict DNSSEC setup where I sign
zones at home on a trusted device, keeping my keys off live servers, ensuring
keys don't end up on untrusted hardware, and wanting my TLS private keys only
on the servers they're being used on—all that suddenly makes my deployment
more complex.

What I'd rather do is, since my nameserver runs a cronjob to pull signed zones
from my home device (unauthenticated because the signed zonefiles are public
information), I wanted to incorporate that into my DNS-01 ACME verification
too. But all the existing tools (certbot, dehydrated, acme.sh, et al) seem not
to support this style of setup well, where I can batch my DNS challenges, wait
for my nameserver to pick them up, and then verify the challenges en-masse an
hour or a day later (LE challenges last for seven days, so this is acceptable
delay for me).

But I'm stuck with the CNAME/alternate NS approach where I have to run yet
another network-facing service from home, for the duration of renewals,
because I simply do not have time nor patience to sift through ACME's
specification and implement a better tool suited for my needs.

>I get notified by email when there is a problem

LE has a mailing list for critical announcements in the interest of current
customers and interested prospectors? Do tell, because all I know about is
their blog and the Discourse forum, neither of which are solely for critical
announcements.

~~~
ferzul
> LE has a mailing list for critical announcements in the interest of current
> customers and interested prospectors? Do tell, because all I know about is
> their blog and the Discourse forum, neither of which are solely for critical
> announcements.

I think they plan to improve their communication after this mishap.

But you can use an rss reader to subscribe to the incidents category (search
incidents.rss in this page), which is very low traffic. It's not “action
required” level only, but with only two or three a year it may be suitable.

~~~
wowaname
Yeah, I subscribed to that feed after seeing it posted here. It's good that
they have _something_ , but I'd much rather be opted in to "important updates
regarding my account" like literally every other service does for me at this
point.

------
owenmarshall
> We confirmed the bug at 2020-02-29 03:08 UTC, and halted issuance at 03:10.
> We deployed a fix at 05:22 UTC and then re-enabled issuance.

Wow. That is a truly impressive way to handle a security bug and I know it’s
not the first time Let’s Encrypt has responded extremely quickly.

I would love to hear how their engineering practices make this possible.

~~~
d33
And I'm curious about the work culture park. They're a small organisation in
terms of workforce, but somehow managed to respond within minutes on Saturday.
How does that work? Do they have shifts or somehow the workers are so devoid
of private life that they have to respond during early morning on weekend?

~~~
tialaramex
For me at least 24/7 incident response is completely acceptable in a properly
compensated role so long as it's accompanied by the culture that says
_preventing_ such incidents in the first place is Job #1

That is, I'm OK with being woken at 0200 to try to understand and if
appropriate fix or recover from a disaster _only_ so long as if I'd suspected
this might happen the people expecting me to be awake at 0200 would have given
me the resource (money, people, whatever) to fix it. If I feel like I don't
have that support, I'll only start looking at your disaster during my working
day.

My impression is that ISRG pays a lot of attention to preventing disasters, so
if I worked for ISRG (not very practical since they're based on the US West
Coast and I live in England) I'd be comfortable taking a call in the middle of
the night to fix things.

~~~
ithkuil
operations based on US west coast could definitely benefit from a few people
on the other side of the world to achieve a 24/7 coverage while keeping a good
work-life balance.

~~~
myself248
Local nerds can be noctournal too. Letting people pick their preferred shift
is just as important as accommodating other kinds of physiological diversity.

~~~
pgporada
Can confirm.

------
rswail
It also only affects you if you are issuing the certificate for more than one
domain name if I'm reading it right.

What's supposed to happen:

    
    
      For each fqdn in the request
        if challenge succeeds (eg dns-01)
          check whether caa record exists
          if (it doesn't) or (it does and allows issue)
            issue certificate
    

In the step on "check whether caa record exists", instead of using the domain
name that is being issued in this loop, it uses the first one it found (or one
of them, it's unclear which one). So theoretically, if you wanted a cert for:

    
    
      domain1.example.com, domain2.example.com
    

and you had a CAA record for domain1 that allowed letsencrypt but then a
different CAA record was added between the CAA check on domain1 and the CAA
check on domain2 (which wouldn't happen because of the bug) you could get a
cert for domain2 that the CAA record said not to issue.

~~~
captncraig
Kinda frustrating having certs revoked when there have never been CAA records
for any names involved. I know they can't know that to be true historically,
but I wish they could do some additional filtering.

------
terom
Here's some quick&dirty stats from the list of revoked certificates:
[https://gist.github.com/SpComb/6338facd12e020ec4fe561ca91f32...](https://gist.github.com/SpComb/6338facd12e020ec4fe561ca91f3254f)

There's 3M "missing CAA checking results" in total, of which 2M are dated from
2020 and 1M from last month. FWIW the only certs of mine affected were old
certs from 2019-12 which had since already been renewed in Feb, and the
renewed certs are not affected?

The largest account has 445k certs revoked, and the most revoked certs from
last month (most likely to still be in active use?) is 43k for a single
account. I hope your rate-limits are in order if you're going to start
reissuing all of those before midnight :/

BTW account number 131 at the top of the file seems to mostly be
akamaiedge.net sites :)

------
ge0rg
There is a list of all affected certificates posted under
[https://letsencrypt.org/caaproblem/](https://letsencrypt.org/caaproblem/) \-
and it looks like they are also leaking the account IDs from the list, so now
you can map different domains/certificates to the account that got them
issued.

~~~
tialaramex
Yeah, it does seem like it'd have been sensible _not_ to list the account ID
in this file. It's convenient if you know your account ID and want to pull out
just your certs, but for most people this associates all their certificates
together.

If you own both [https://www.happy-rainbow-
nursery.example/](https://www.happy-rainbow-nursery.example/) and
[https://hardcore.bdsm-videos.example/](https://hardcore.bdsm-videos.example/)
you probably go to some lengths to avoid visitors realising the connection.
Nothing you're doing is illegal or even unethical - but it's obviously going
to cause uncomfortable conversations so why not avoid that altogether. Let's
Encrypt aren't doing you a favour if tomorrow a mom at nursery says now she
knows why you sound so much like Masked Mistress Martha...

~~~
djsumdog
Yea, this is really bad. I've done some searching of the data. Sometimes it
doesn't matter. It looks like whoever is currently running gab.com is probably
a big consulting company with like 100 other clients, so there's no big
relation there. But if you run a small personal blog and use the same e-mail
address for maybe more controversial sites that are hosted on different IPs,
now you could get doxed.

I'm guessing customer IDs are associated with e-mail addresses? This seems
like a good case of using different e-mails for ever cert. There are open
source tools like anonaddy.com you can host yourself or buy from them (they
have a decent free tier).

I feel like this list seriously needs to be pulled. There is some serious lack
of oversight here.

~~~
pfg
> I'm guessing customer IDs are associated with e-mail addresses?

They are (on Let's Encrypt's end), if an email address was provided.

It's a 1:n relation, the same email may be used for any number of ACME
accounts. Roughly speaking, for most clients, the ACME account maps to a
specific ACME client on a specific host. If you run three servers with
separate ACME clients, you're probably using three ACME accounts (even if
you're using the same email and issuing certificates for the same domain).

Large or custom implementations may reuse the same ACME account across many
servers and domains. (Issuance would typically be centralized and operated as
a separate system in these scenarios.)

------
appleflaxen
There are several links in this thread, but the following page allows you to
enter your hostname and check online.

(no SSH, terminal access, etc) and it's from the letsencrypt team (linked in
blog post).

[https://unboundtest.com/caaproblem.html](https://unboundtest.com/caaproblem.html)

~~~
zimpenfish
Alas, only works for things on port 443 which is a bit of a problem for most
of my certificates...

------
b4d
If anybody needs a bulk check:

for domain in $(cat domains.txt); do printf "$domain :" && curl -XPOST -d
"fqdn=$domain"
[https://unboundtest.com/caaproblem/checkhost;](https://unboundtest.com/caaproblem/checkhost;)
done

------
MrStonedOne
>CAA

What is a CAA? Letsencrypt: please dont use initialisms in customer facing
blog posts without using the FULL name at the first use. Makes things more
learnable and googleable.

~~~
lvh
CAA is a term of art for a standard encoded in DNS. It stands for
"Certification Authority Authorization", but most people who know what a CAA
record is probably do not recognize it written as words. (I know I would need
to read it several times to know that's what they meant, and I do this for a
living.)

~~~
tialaramex
Expanding other DNS record names isn't very helpful either. I know what a
Canonical Name for something is, but CNAME seems clearer, Mail Exchanger
definitely isn't a helpful way to think about what MX records are for...

PTR and TXT aren't initials they're just short for "pointer" and "text"
neither of which is much help divining what they're actually used for, and
presumably AAAA doesn't actually stand for anything at all (?) other than it's
four times bigger than the A record.

~~~
captncraig
4 A's for ipv6 of course, and 1 A for ipv4. How could that possibly be
confusing?

Had I been given a vote we would be using AAAA and AAAAAA records.

------
rswail
This bug only affects you if you got domain validation (eg by dns-01) but
didn't immediately issue a certificate.

Letsencrypt validates the domain ownership for 30 days, so the bug allows you
to issue a certificate within that window, even if you added a CAA record
after validation that says "don't allow issue by letsencrypt, or only allow
issue by MyCA.example.com".

But if you have everything automated, you're checking for renewal and issuing
every day and probably validating as part of that, so unlikely to encounter
the bug, unless you validate in one step and then sometime 8h+ later, issue a
certificate.

~~~
kn100
I don't think this is true - I don't recall using domain validation. I think
it's more related to multi domain certs.

------
deweller
Will a nightly certbot invocation will replace these revoked certificates
without manual intervention?

~~~
simias
Apparently you have to manually use --force-renewal for certbot to regenerate
new certificates (I just did it just in case, even though I'm 99% sure that
I'm not concerned).

I assume that by default certbot only checks the expiration date of local
certificates against the system clock, it doesn't ping any external resources
so it can't be aware that the certificate might have been revoked even though
it hasn't expired.

I agree that it would be nice if there was such an option, although I assume
that it would increase the server load significantly if certbot connected to
letsencrypt's servers at every invocation so maybe that's why they didn't do
it.

~~~
lightswitch05
> I assume that by default certbot only checks the expiration date of local
> certificates against the system clock, it doesn't ping any external
> resources so it can't be aware that the certificate might have been revoked
> even though it hasn't expired.

I think the actual issue here is that the certificates have not been revoked
yet. We know that they will be revoked, which is why we have to run with
--force-renewal, but there is no process for certbot to know that a
certificate, although not revoked, will soon become revoked. I would expect
certbot to automatically renew the next time its ran post-revocation.

------
shakna
Thankfully it's fairly painless to see if you're affected:

    
    
        curl -XPOST -d 'fqdn=example.com' https://unboundtest.com/caaproblem/checkhost
    

Replace example.com with your Fully Qualified Domain.

~~~
Ayesh
All my recent certificates are affected. I think it is the same case for the
majority of us.

~~~
tialaramex
The most likely cause will be that you issued similar certificates a few days
(more than eight hours) earlier with the same Let's Encrypt account.

Suppose on Wednesday you get a cert for example.com and www.example.com, and
then on Thursday you realise you also need images.example.com - you use the
same ACME account (if you run Certbot this will happen by default if you use
the same machine and user account, it silently makes you a free account if you
don't have one already) and so Let's Encrypt can see that this account showed
proof-of-control for two of these names on Wednesday, so only fresh proof-of-
control of images.example.com is needed. Unfortunately this bug means Let's
Encrypt forgot to re-check CAA for the old names, and so there's a risk they
technically were no longer authorised to issue for these names and shouldn't
have given you the Thursday certificate.

Rather than try to argue about whether it's appropriate to disregard this
check, Let's Encrypt decided to revoke all ~3 million affected certificates.
That's maybe 2-3 days worth of normal issuance in the last 90 days, so lots
but hardly "the majority".

------
gindely
If like me you have several hundred certificates to check, please do something
like this:

cd somewhere-nice

wget [https://d4twhgtvn0ff5.cloudfront.net/caa-rechecking-
incident...](https://d4twhgtvn0ff5.cloudfront.net/caa-rechecking-incident-
affected-serials.txt.gz) gunzip caa-rechecking-incident-affected-
serials.txt.gz

for i in $(cat domains); do (openssl s_client -connect $i:443 -showcerts <
/dev/null 2> /dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number
| tr -d : | tail -n1) |tee serials/$i; done

cat serials/* | tr -d " " | sort | uniq > serials.collate

grep $( cat serials.collated | head -c-1 | tr "\n" "|" | sed -e 's/|/\\\|/g' )
../caa-rechecking-incident-affected-serials.txt

It will take a moment and then it may tell you that letsencrypt misspoke when
they said they sent emails to everyone whose contact details they have.

~~~
Operyl
I thought I was in the minority there! We have 45 certificates (of many more)
that were affected, and our account id was listed, and it has an email contact
associated. I got no email whatsoever, but I'm glad I had the foresight to
check anyway.

~~~
gindely
I just noticed I got an email at 1949 UTC. I guess they're still sending them
out. Presumably some people will receive their emails after the revocation.

~~~
Operyl
I spoke to someone from the team, they’ve got another 10% to go (presumably
much less now). I finally got mine as well, and they’re still coordinating to
figure out the timeline to revoke. Presumably they’ll wait for the emails
first.

------
tialaramex
I suppose this is one way to answer my question

I asked (on m.d.s.policy) on the 29th how many issuances were affected, Jacob
replied saying they intended to spend that day figuring out the answer, but
then there was nothing further from him. The incident doesn't seem drastic
enough to prompt urgent answers so I intended to revisit later this week if I
heard nothing further.

Now we have a complete list of affected certificates instead, (the answer to
my original question is about 3 million)

I was sort of hoping the answer was going to be like five thousand or
something manageable. Alas. In hindsight I guess this was to be expected.

~~~
thenewnewguy
Well, from my understanding, they have no idea of whether or not a given
certificate was misissued as long as it meets the baseline criteria for
triggering the bug (which seems to be just "issued after X date with more than
1 domain").

So while the number of certificates that should not have been issued due to a
blocking CAA record is likely small (or possibly even 0), they have to revoke
every cert that could have triggered the bug, as they have no way to travel
back in time and find out what the CAA records they didn't check would have
been.

~~~
tialaramex
The criteria also require that the challenge answers used to validate control
were old (more than eight hours old).

If all proof-of-controls were fresh the CAA checks are also fresh for those
proof-of-controls so there's no bug. That's why the big list is "only" about
three million certificates.

Suppose you own example.com, example.org and example.net and all you do is
every 60 days or so you spin up Certbot once to get a certificate for six
names, the three domains and the associated www FQDNs - that won't trigger
this bug because each time your old proof of control have expired and new
fresh ones will be used, triggering fresh CAA checks.

You're right though that it's likely the number of truly mis-issued
certificates may be zero because the most common way to have a CAA record
deliberately changed to forbid Let's Encrypt after having successfully done a
proof-of-control (the scenario that would trigger their bug) is researchers
looking for bugs in CAA checking, and of course such researchers would have
reported this to Let's Encrypt triggering exactly the same incident but
probably at a more friendly time like a Monday morning.

------
kn100
Yeah my domain was affected. Renewed, that would have sucked if I hadn't seen
this! Also, I didn't get an email and I'm pretty sure my certs were generated
with my email!

------
mholt
FWIW, I believe any Caddy sites will not be affected by this since caddy does
not manage multi SAN certificates. Even if it did, Caddy will immediately
replace a certificate when it sees a Revoked OCSP response. So, if you're
using Caddy, there's probably nothing you need to do. But if you are a caddy
user and are impacted, let me know.

------
paulfurtado
Does anyone know of a generic way to detect that letsencrypt will revoke a
certificates soon?

The goal would be to have our automation automatically rotate the certificates
when similar issues occur in the future.

~~~
pfg
To my knowledge, there's no such mechanism in any of the relevant protocols
(i.e. ACME and OCSP).

------
ck2
That thread enlightened me to a great trick to force cert renewal even if it's
been done too recently: add a second (sub)domain and make a new cert with both

------
geocrasher
Here's a bash one-liner to check all the domains you have:

for domain in $(cat list-of-domains.txt); do curl -s -X POST -F "fqdn=$x"
[https://unboundtest.com/caaproblem/checkhost](https://unboundtest.com/caaproblem/checkhost)
;done | sed '/is OK./d'

------
chrisMyzel
Oh Let's Encrypt - what a chaos tuesday this was for me - I was informed
yesterday, 19:00 (UTC+7) and it was a fun evening - many befriended companies
and their clients where not informed at all.

Funny to find out the otherwise awesome traefik has no force-renewal other
than messing around with acme.json until it renews

------
low_key
PSA: The unbounded checker doesn't seem to work if you have certificates
issued for both ECC and RSA keys. For some of mine, it passes the check with
status "OK" and shows the serial number of the certificate for the ECC key.
The certificate that is going to be revoked is not shown.

~~~
tialaramex
If you have more than one certificate in use, regardless of what flavour, they
only see one and assess that. Maybe the checker should emphasise that. For
small users they probably only have one certificate in use, so this avoids
some problems.

The issue would probably also affect people who have geographically separate
certificates e.g. if you have two servers in different regions and decided
rather than make things more complicated for key distribution you'll just have
them each get their own certificates for the same name - that's totally fine
with Let's Encrypt (it doesn't scale, but if you had 500 servers not 2 you'd
probably redesign everything) but obviously this test only sees one of those
servers and won't check the other certificate.

There's no way to know, given that two (or more) valid certificates exist for
a name, and seeing one of them, whether the others are still actively used
anywhere.

It would obviously be pretty easy to build a web form where you can type in an
FQDN and get told if any certificates matching that name will be revoked, but
then you get false positives where it says yes, this certificate for
some.name.example will be revoked, you rush to replace your certificate for
some.name.example but maybe actually the one that will be revoked is from 20
December 2019, and you already got a newer one which was unaffected in
February.

~~~
namibj
I wish they'd issue short-term scoped CAs under the same criteria as they
currently use for wildcards.

No significant load on their infrastructure, and you'd not have to break the
"private keys don't move over _any_ network" rule.

~~~
tialaramex
[https://datatracker.ietf.org/doc/draft-ietf-tls-
subcerts/](https://datatracker.ietf.org/doc/draft-ietf-tls-subcerts/)

Delegated Credentials is the proposed mechanism to do what you actually want
to achieve here. The certificate issuance is mostly the same except there's an
OID 1.3.6.1.4.1.44363.44 and the digitalSignature KU is present meaning this
certificate is intended to be used with Delegated Credentials and thus can
sign things.

Although tightly constrained subCAs (which is roughly what you're describing)
might be a way to achieve your goal, it's seen as disproportionately
complicated and risky, so hence this proposed much more narrowly defined new
feature for TLS.

------
karimmaassen
Yes, let's.
[https://www.reddit.com/r/ProgrammerHumor/comments/7x2ugb/let...](https://www.reddit.com/r/ProgrammerHumor/comments/7x2ugb/lets_encrypt/)

------
qXlihgad7n
Is there an RSS feed that you can subscribe to for alerts like this? I can't
see one.

~~~
thenewnewguy
[https://community.letsencrypt.org/c/incidents.rss](https://community.letsencrypt.org/c/incidents.rss)
perhaps?

(Pro tip: you can append .rss to many pages on discourse to get an RSS feed)

------
robjmills
Is there anything to suggest if this relates to Certbot or API provisioning
only? or both? We've checked a whole bunch of API v1 provisioned certs against
their tool and nothing has been listed so far.

~~~
robjmills
FWIW I think the reason we're unaffected (as far as we can tell so far) is
because we're not re-issuing certs within a short time period. The bug their
end was to do with checking CAA records, if you re-issued the cert for a
multi-domain cert within a short period of time after the initial provision
then it wouldn't re-check the CAA records. This meant that subsequent CAA
changes wouldn't be checked and theoretically a cert could be re-issued
despite a CAA record being added to prevent this. As i'm reading it, if you
didn't re-issue within this timeframe then your cert can be assumed to be
correct as the original CAA check wasn't a problem.

------
bobmaxup
Only 13 out of ~3,500 of certificates I manage required renewal

~~~
vermontdevil
3,500? wow. What are you managing if I may ask?

~~~
bobmaxup
A CMS platform with customer provided domains.

------
Tomte
I haven't got a mail (I think), and I don't see that on their web site or on
their blog.

Is wading through Discourse threads now the new minimum requirement for using
services?

~~~
gindely
No, you can get your information from Hacker News.

(Do check your domain even if you didn't get an email, since they have not
delivered emails to everyone who is affected.)

------
TwoNineFive
They have known for over 30 days and they didn't notify me until about 10
hours ago that several of my domains were affected.

That's not okay. That's serious negligence.

EDIT: My bad. They have known about this for maybe a week, and they did a blog
post 4 days ago. I was basing my thought that they had known for 30 days based
on the title of the post and in the URL:
[https://community.letsencrypt.org/t/2020-02-29-caa-
recheckin...](https://community.letsencrypt.org/t/2020-02-29-caa-rechecking-
bug/114591)

~~~
forbiddenlake
I'm confused about what you mean; 2020-02-29 was 4 days ago, not 30.

They didn't know the extent of the bug until the 29th of February:
[https://bugzilla.mozilla.org/show_bug.cgi?id=1619047#c1](https://bugzilla.mozilla.org/show_bug.cgi?id=1619047#c1)

------
IceWreck
Yes, I just got their mail. Four certs out of six, all issued at the same time
were affected. The other two were not.

------
low_key
Looks like LE will be adding to the billion certs they've issued!

[https://news.ycombinator.com/item?id=22434466](https://news.ycombinator.com/item?id=22434466)

------
donatj
Can we get a “(some)” for the title?

------
rb808
Certificates are such huge a maintenance problem. So many mines waiting to
blow up. We really need something better.

------
32gbsd
So it starts

~~~
GuyPostington
So what starts?

~~~
32gbsd
random bugs

~~~
wowaname
So, nothing new

