
The Impending Doom of Expiring Root CAs and Legacy Clients - fabian2k
https://scotthelme.co.uk/impending-doom-root-ca-expiring-legacy-clients/
======
geerlingguy
I begrudgingly bought a 'Smart TV' just because I wanted a 4K monitor. But I
use it as a dumb monitor and watch app-based content through an Apple TV. At
least I'm reasonably sure the Apple TV will be updated for 5-10 years, and is
a lot more disposable than a giant TV.

My last TV worked great for about 12 years. No way Panasonic would've kept
supporting it that long.

Honestly, the thing I hate most about smart TVs, especially budget-conscious
ones like the one I bought, is they have terribly slow processors so just to
turn on the TV takes like 12 seconds. My old TV had a picture up within 3
seconds of turning it on.

Does any manufacturer make a good TV, or are they all smart now?

~~~
hanklazard
Someone on HN recently floated the idea of a company that would make quality,
non-smart electronics and appliances that would also be easily serviceable. I
wish this existed and would pay significantly more for products like this.

To address your question, I think they’re all “smart” now. I ended up getting
a low-end Samsung 4K a few months ago and it’s been good so far. It starts up
very quickly, maybe 1-2 seconds. We use it exclusively with an AppleTV because
of privacy / advertising concerns that have been raised. I’m just waiting for
the day when all TVs _require_ a working internet connection and we have to
start hacking their hardware and software to protect our data.

~~~
koolba
Never add your WiFi credentials you your TV. It doesn’t let you remove them so
you’re only option is to create a “dummy” network (ex: phone as a hotspot) and
switch to it.

Also, watch out for clueless^helpful friends and relatives who try to “fix”
your TV’s WiFi.

~~~
gindely
(ot: should that be clueless^Whelpful, or is there some new convention I'm not
aware of? ^ still means "ctrl" right?)

~~~
castillar76
It's an old Unix convention: ^W (Ctrl-w) would remove a word using the Emacs
keybindings. So in place of striking out a word, you'd put ^W and the
replacement, as you would when typing it out ("clueless...err, no, wait...
_ctrl-W_...HELPFUL").

~~~
Drdrdrq
Gp missed "w".

------
tialaramex
If you have a current Firefox that first diagram may not be correct for you.
Let's tell a brief story about why.

Servers, as Scott explains, are supposed to present a "chain" of certificates,
in practice it's one leaf and then just anything else that might be useful for
clients to assemble a trust path. If you do this everything works.

But lots of servers are misconfigured and present only the leaf certificate,
not least because a lot of older software makes configuring the chain
needlessly confusing/ complicated and it often "seems" to work anyway in some
client software.

Given just a leaf, and a set of roots it trusts, a client can't make a useful
trust decision, there's a gap. One option in this circumstance is to just give
up and just tell the client's user this isn't secure. That's a poor UX.

Another idea is called AIA Chasing, many popular web browsers do this. When it
can't get from the presented certificates to a trusted root the browser
inspects the Authority Information Access field of a certificate to find a
URL, hopefully this URL points to an intermediate certificate that plugs the
gap. A problem with AIA chasing is that it reveals information about your
browsing to the CA who otherwise have no reason to know. This privacy concern
led Mozilla to reject AIA Chasing.

Yet another idea is to cache intermediate certificates for some period of
time, then use that cache to fill any "gap" when making a trust path for other
servers you connect to. This causes hard-to-predict errors for users, maybe
the funny cat video site you enjoy works fine... unless you try to visit it
first thing after turning on the PC. It can potentially represent a privacy
risk within a session.

So ultimately Mozilla's workaround in current Firefox is to give Firefox a
fairly complete set of intermediates, just as if it had cached them already.
Essentially that right hand "embedded in browser" part of the diagram extends
left to the intermediates too.

It made that happen through two means: Firstly m.d.s.policy required all root
CAs to tell it about all unconstrained intermediates they had ever created,
which was itself a revealing process because there sure are a lot of (mostly
older) garbage intermediates that clearly are untrustworthy and shouldn't
exist. More recently Certificate Transparency means any intermediates still in
use must be in the CT logs because otherwise the leaf certificates won't be
accepted by those logs, so Mozilla can "just" fetch that data.

~~~
diroussel
Interesting note about Firefox there.

I've certainly expirenced the missing "missing intermediate cert" problem
before. It is a bit of a nightmare to debug.

1\. Had issue, raise support ticket: TLS not working

2\. Ticket closed as can't reproduce

3\. Try myself again locally, also can't reproduce. Hmpf!

4\. 2 months goes by..

5\. Experience same issue. Debug more carefully locally, use openssl go get
proof of missing intermediate

6\. Raise support ticket with platform team, then try to convince them that
just because most people are not affected, it still needs to be fixed!

~~~
tialaramex
You can often engage people, especially technical people who enjoy things
being gameified - by showing them that Qualys gives them a lousy score for
what they've done.

[https://www.ssllabs.com/ssltest/](https://www.ssllabs.com/ssltest/)

Also this lets you out-source the decisions about what's important versus what
really doesn't matter to somebody else, and unless you've got (or can hire
someone who has got) hours per week to read and digest work in this area
that's likely going to mean better security in practice.

~~~
user5994461
ssllabs is great and only works for publicly accessibly websites.

The challenge in my experience is to resolve the issue for internal sites, the
thousands of internal tools and test domains from every department. None of
the public tools can reach them.

~~~
PostPlummer
If you like ssllabs, you will love
[https://www.hardenize.com](https://www.hardenize.com)

Much more intel, better performance and a better UI.

No affiliation, just a fan since day one.

------
topkai22
Certificates suck. They are monumentally user unfriendly and complex. How many
major internet properties have had outages driven by certificate renewal in
the last several years? Half? I know that enterprises I work with seem to have
some sort of work impacting certificate problem every month.

The ecosystem needs to become a lot more robust and user friendly in general.
Unfortunately, I’m just a user of certificates not a cryptographer, so I’m not
really qualified to design security critical aspects of the system, but here
are some rough features that I’m looking for:

\- certificate warning date: a time period defined like expiry date that
indicates that clients should warn of impending expiration but still click
though

\- Make it easier for people to acquire and renew certs. Let’s encrypt is an
amazing start here, but it’s not a universal solution yet

\- Formalize a way to solve the key distribution problem described in the
article. Again, I’m not a security expert, but perhaps embedding a “replaces”
concept into a certificate would work.

~~~
danpalmer
While I empathise with your viewpoint, that the ecosystem is hard to work
with, I'm afraid I don't think there are any quick solutions to this.

> certificate warning date

If this is user-visible then for most companies this would be nearly as bad as
expiry, and if it's not user visible then it's not useful.

> Make it easier for people to acquire and renew certs

Let's Encrypt pretty much solves this problem. The places where it doesn't
work are companies that lock down the use of that sort of software. I don't
think there's an easy way around that.

> Formalize a way to solve the key distribution problem described in the
> article

This problem isn't just the problem described in the article, it's _THE_ Key
Distribution Problem. It has a Wikipedia entry, it's a chapter in most
security textbooks, it's a lecture in most cryptography modules at
universities. It's a _hard problem_. The practice of including Root CA certs
on devices is the best thing we've come up with yet that has the security and
scalability trade-offs necessary.

If the "replaces" concept was viable in the general case I think we'd have
likely implemented it, but I suspect it's too vulnerable, and likely only
pushes back the problem.

~~~
topkai22
I suspect we need standards/protocals defined at layers above the crypto-
layer. For the certificate warning date, a yellow lock rather then giant red
screens browsers show me now would be good, so at least in some cases users
can do the notification. However, the real solution is standardized tooling
that helps the organization detect the issue.

Let's encrypt is definitely in this vain, however I think it needs to be the
point where software defaults to user friendly behaviors like auto-renewal
unless explicitly told not to. This seems like it could be address in
standards making process, but maybe that's the wrong venue.

------
teddyh
To quote myself¹ from two years ago:

Any Internet-connected device is, in fact, a server, and must be seen and
managed as one. This means strict control of installed services and, first and
foremost, regular _updates_ of all its software components (including
firmware). If you acquire and install such a server which either can’t be
updated or one which you know, realistically, won’t get any updates six months
after installation, that’s asking to lose.

1\.
[https://news.ycombinator.com/item?id=18019343](https://news.ycombinator.com/item?id=18019343)

------
jakub_g
Smart TVs are the worst. Unrelated to TLS: I work for a video streaming
company, and while almost all of desktop/mobile Chromium-based traffic is
Chrome v80+ or at least v70+, the smart TVs have webviews based on Chromium
v45~55 (most TVs) or ~v60 (cutting edge modern Samsungs), and some older ones
(~2016) are even worse, based on a very outdated webkit which is more or less
Safari 8.0 - basically hardly better than IE11.

But coming back to TLS again: even if the old devices have an up-to-date root
store, they typically have an outdated TLS stack which does not support TLS
1.2 ¯\\_(ツ)_/¯ and this one is probably even harder to update. And since most
of the companies want to drop TLS 1.1- support on their servers soon, this
should also doom those outdated devices apart from the root stores issues.

~~~
kstrauser
If this is the newspaper-whack-to-the-nose that gets people to stop using sly
TVs as more than dumb monitors, then I think we'd all be better off for it.

------
devit
Having expiring certificates/key rotation might be a net negative: if you keep
the private key secure, there is no need to rotate it, and it avoids a lot of
hassle. Also, if you have a revocation mechanism, then rotation doesn't add
that much for keys that are only used for signing and not for encryption (like
the CA keys).

Of course in some cases like domains it's necessary since the domain can be
transferred, but this is solvable by DNSSEC+DANE.

~~~
gregmac
It's a good point. At any given time, the key is either compromised or it's
not. And you may or may not know either way.

If it's compromised, and you know, then it should be revoked -- if your only
mechanism to revoke it is waiting some amount of time (several days, months,
or even a decade) you have a pretty big problem.

If you don't have a way to detect if it's compromised, rotating a precaution
almost makes sense -- but then the valid time should be very short. I'm not
exactly sure how short, but definitely the decade or two used for CA certs is
too long. Make it shorter than a year and automated renewal becomes necessary,
but if you have that mechanism why not check for revocations?

The only other reason I can think of to expire non-compromised certificates is
to force them to be regenerated to use newer signature algorithms, but even
that's difficult to predict. We're currently using SHA-2 for PKI, but is that
still going to be reasonable in 5, 10 years? Someone could figure out a
potential weakness at any time, which would start the move to something else.

~~~
GoblinSlayer
I'd say sign the next root certificate with previous root key, then you can
rotate root key automatically. Browser update works the same way: you download
new browser with new root certificate authenticated by previous root key.

~~~
WorldMaker
Most roots are required to be cross-signed today by all the major browser and
OS vendors. What you have is a classic web of trust issue because it's the new
key that has the signatures attached so you could take the new key/certificate
at its word, but if the problem is the previous root expiring, how do you
trust that the new root certificate was signed by the previous root before it
expired? You can't trust the now expired root can you? If you restrict it to
the subset of time where both roots are unexpired, you have the exact same
problem as the intermediate game that the BBC has to play in the article.

~~~
GoblinSlayer
Same as with the browser update: you receive it before the previous
certificate expired.

~~~
WorldMaker
Most of this article is about the pain caused by that we cannot assume devices
will update in _years_. There's no window you can set for both certificates to
be valid side-by-side that will be long enough to update every device.

Again, roots are _already_ cross-signed and the article points out they still
take 2+ years to pass other validity checks for root adoption. Even if they
"virally" propagated in this manner after all those checks passed, there are
plenty of devices that are only powered on once a year or less; there are a
lot of devices whose support lifetimes for any upgrades at all are less than 2
years (that's a specific call in the article: we need security support
lifetimes extended to decades at least, probably).

If they propagate "virally" you have a lot of questions of whether or not a
device will even see that there is an updated certificate. Not every user or
device visits a web page in a certificate chain to any given root routinely
much less _ever_.

Comments to this article even point out that there is a semi-viral update
process already in place in most browsers called AIA chasing, where
certificates may point to URLs to look up their parent authorities, and
Mozilla intentionally doesn't AIA chase because it's very definitely a privacy
risk, even if you don't agree that it is a security risk (bad certificates
sending you to bad URLs). The security risk is why the browsers that do AIA
chasing only allow it for intermediate certificates and will not trust new
roots found by AIA chasing.

------
mschuster91
I got hit at work by the AddTrust issue too, quite a bit of a nightmare to
chase down, especially Java keystores. And that was mostly because there exist
clients who barf if they have a valid working chain but _also_ another chain
where one of the certs has expired!

For embedded devices - it's not just smart TVs but all IoT devices! - I think
that the solution is a legal requirement: a condition of a device being able
to be sold on the EU / US market must be the commit of the whole source tree
and build chain including any and all key material for signing firmwares to
the national library or other government-run secure escrow, to be released to
the public once the manufacturer discontinues the product support. Similar to
like you can't sell a product without adhering to the CE and electrical safety
standards, manufacturers should not be able to sell products without having
them certified for IT safety and e-waste/"planned obsolescence" issues!

------
Androider
TIL about Network Error Logging[1], which is having browsers send you reports
that your DNS isn't resolving, your load balancer is timing out (HTTP / TCP
errors in general), or your TLS certificate is invalid. Basically all types of
issues that don't end up in your own application server logs.

Report URI should lead with this (it's not even mentioned on the front page)!
It's super useful for every site, I can't think why anyone wouldn't want this.
CSP monitoring on the other hand, which seems to be their focus, is a much
harder sell, a pain to setup and maintain, and a bit questionable value in my
opinion.

[1] [https://report-uri.com/products/network_error_logging](https://report-
uri.com/products/network_error_logging)

~~~
parliament32
This is the actual tech, rather than a commercial solution:

[https://w3c.github.io/network-error-
logging/#introduction](https://w3c.github.io/network-error-
logging/#introduction)

[https://developer.mozilla.org/en-
US/docs/Web/HTTP/Network_Er...](https://developer.mozilla.org/en-
US/docs/Web/HTTP/Network_Error_Logging)

~~~
Androider
Report URI is run by Scott and was featured in the article.

By definition you have to run such a service entirely separate from your own
infrastructure: you should use a different CA than your other site(s), a
different domain name for sure, preferably a different infrastructure host
etc. It makes a lot of sense to use something like the Report URI service for
it.

------
imhoguy
That is why we need open hardware specs, open-source firmware and a right to
reverse-engineer out of market devices. Like OpenWRT or LineageOS, they keep
old devices usable for long and reduce electronic waste.

------
tialaramex
> If we take a look at similar data for iOS, it's a very different story.

This fits the prevailing narrative, and for all I know it's true, but Scott
does not in fact illustrate it.

The "similar data" to a table of all Android API versions versus popularity
turns out to be a chart of recent iOS versions in which everything that isn't
current is just grouped as "Other" at 16.28%

Maybe if we ungrouped "Other" it would prove Scott's point, maybe not, we
shan't find out answers in his article. If those "Other" iOS users are all
running 10.3 it's a very different story than if they're on iOS 6...

~~~
idbehold
He links directly to the source of that data and on that site you can download
a CSV of the aggregated data.

    
    
      "iOS 13.3",27.03
      "iOS 12.4",16.19
      "iOS 12.3",14.2
      "iOS 13.1",10.22
      "iOS 12.2",7.89
      "iOS 12.1",4.16
      "iOS 13.4",4.02
      "iOS 13.2",2.68
      "iOS 10.3",2.31
      "iOS 11.4",2.14
      "iOS 9.3",1.94
      "iOS 11.2",1.03
      "iOS 12.0",0.86
      "iOS 13.0",0.82
      "iOS 11.0",0.81
      "iOS 11.3",0.72
      "iOS 6.0",0.45
      "iOS 10.2",0.4
      "iOS 11.1",0.32
      "iOS 7.1",0.18
      "iOS 13.5",0.17
      "iOS 10.1",0.17
      "iOS 9.1",0.15
      "iOS 7.0",0.15
      "iOS 10.0",0.14
      "iOS 6.1",0.13
      "iOS 5.1",0.12
      "iOS 9.2",0.12
      "iOS 8.4",0.11
      "iOS 8.1",0.1
      "iOS 8.3",0.06
      "iOS 9.0",0.06
      "iOS 5.0",0.04
      "iOS 8.0",0.03
      "iOS 4.3",0.03
      "iOS 8.2",0.02
      "iOS 3.2",0.01
      "Other",0.02

------
AshamedCaptain
"Impending" doom?

Devices from 10 years ago are _already_ useless online.

Not only are the root certificates completely expired, but let's see which
protocols does an openssl 0.9.8 build from a decade ago have in common with,
say, current Gmail's IMAPS server: NONE.

~~~
account42
I am really conflicted when configuring my servers. I can see the need to
deprecate insecure protocols but deliberately cutting off compatibility with
old clients feels wrong.

------
afwaller
We had to deal with the sectigo / addtrust expiration. It was annoying because
we reached out to our cert vendor and they had claimed it would not affect us
(only affecting “legacy” systems or old browsers), but then of course it did
affect us. It was an easy resolution, but still annoying.

Of course, we still have people working on our stuff. If it were some
abandoned hardware product I could see this being a disaster.

~~~
rmetzler
We too were affected by the Sectigo / Addtrust expiration. Google's Uptime
Checks couldn't verify the validity of the certificate and reported our
website was down. While this was a false alert and it was easy to verify and
circumvent it just shows that even companies like Google aren't prepared for
something like this.

------
hamiltont
Has anyone taken the offered training?

I'm pretty good with TLS, but could use more hands-on time with openssl and
PKI, especially a deeper understanding of certificate chaining issues. Would
be interesting to know if the training is targeted at "total beginner" or
"intermediate looking to be advanced"

~~~
payamb
I’ve taken the training and I recommend it, Keep in mind though that TLS and
PKI are broad subjects and covering every intermediate/advanced use case in a
two days training won’t be possible. But yeah it does cover certificate
chaining issues, cross signing etc.

------
grishka
This might be a stupid question, but... Why do root certificates have to
expire at all? The article presents this as if it's an immutable law of
physics.

~~~
MaxBarraclough
We have no real means of revoking certificates whose private keys have leaked.
Eventual expiration seems better than nothing. Also, ciphers tend to become
obsolete, so enforcing churn will help keep things moving forward. I'm not an
expert here though, there may be other reasons.

~~~
grishka
I feel like if a certificate gets compromised, the fact that it expires in 5
years won't be of any help from the security standpoint. There are already
mechanisms in place to revoke certificates before they expire.

[https://en.wikipedia.org/wiki/Certificate_revocation_list](https://en.wikipedia.org/wiki/Certificate_revocation_list)

[https://en.wikipedia.org/wiki/Online_Certificate_Status_Prot...](https://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol)

------
syats
This will also be an issue with some docker images built some years ago with
an outdated cert store somewhere in there. Unmaintained images are, in effect,
frozen in time.

I'll just mention glassfish here in case someone in the near future wonders
why this or that application is failing: it has it's own cert store (different
from the JVM one), and it is a bit out of date. Yes, I just ran into a github
project whose last commit was 3 months ago (march 2020) and that wouldn't work
with LE certificates, all because deep down the docker matrioshka, there was a
glassfish with outdated certs.

------
lol768
This post honestly reads like an ad for the training course.. did it really
need 8 mentions on the page?

------
stormdennis
We have our smart TV just acting as a display for a mini PC via HDMI
connection. TV knows nothing about our WiFi so no phoning home.

~~~
tomxor
Have you had fun hunting for and turning off all of the awful picture
"enhancing" features yet? and then figuring out how to make it persistent.

Modern TVs really seem to ruin the video even though they are perfectly
capable of just showing a color adjusted signal, quickly becomes obvious when
you plug in a computer.

~~~
stormdennis
I don't know to be truthful. The TV remote has loads of buttons bottom the
only ones I use are on/off, source to find the HDMI and volume sometimes. I'd
love just to have a dumb monitor of the same size but they're not available. A
bonus then would be no need to have a tv licence.

~~~
jfk13
If you're referring to a UK TV licence, it doesn't matter what kind of monitor
you're using; it's about what you watch (live TV, whether over the air or
online, or any use of BBC iPlayer). Watching TV via a computer and a dumb
monitor still requires a licence.

[https://www.tvlicensing.co.uk/check-if-you-need-
one](https://www.tvlicensing.co.uk/check-if-you-need-one)

(Edited to add: Conversely, merely owning a TV does not require a license; if
you're only using it as a computer or gaming monitor, and not watching
[online] TV programmes on it, you should be fine.)

~~~
stormdennis
Unfortunately not. Having a tv set in the house here requires a licence even
the thing is out of order

~~~
jfk13
Interesting! I'm curious where "here" is, if you're happy to say...?

~~~
wiredfool
Potentially Ireland. That seems to match the licensing scheme here.

------
dehrmann
You almost need something in HTTP for a client to signal which root
certificate is being used, then if it's old, push the new one (signed by the
old one) in the response. There's plenty of potential for abuse with this,
though.

Maybe root certs need to expire _more_ often so implementors have to do OTA
updates.

------
rootedbox
Who hasn't had a root ca issue in production. It's v painful.

------
gambler
A working device should never stop working because of an expiration of some
made-up entity. It is a signs of compete failure in basic system design.

~~~
withinboredom
For sure. Add to the fact that in every crypto-library I’ve used, you can’t
tell it to ignore the expiration date without also ignoring actually harmful
certificates, and this is what you end up with. If devs could just say: yeah
we are not ever going to update the certificate, I’d be ok with that. An
expired certificate is not inherently untrustworthy, it should just mean that
it can’t sign anything. They’d then die of their own accord or be revoked.

Tin foil hat: To me, it smells more like CA’s wanted to make some cash by
forcing their customers to buy certificates every so often instead of actually
solving the problem and are now being bitten by their own rules.

------
superkuh
Once again the persuit of security theater to meet the requirements of
commerce (which most of the web doesn't share) cause problems accessing the
large parts of the normal web.

When are people going to learn that centralized cert authorities are just for
commerce and only hurt the non-commercial web?

~~~
Drdrdrq
Curious - what do you suggest for non-commercial web, taking into account both
privacy and security?

~~~
superkuh
If those are your motivations then making the site accessible as an onion
service over tor in addition to the http interface (and https self signed)
seems reasonable.

