

New TLS certificate for .herokuapp.com hostnames - paulfurley
https://devcenter.heroku.com/changelog-items/1813
We run a backend API app on Heroku and for simplicity our frontend calls it via the herokuapp.com subdomain `&lt;our-app-name&gt;.herokuapp.com`.<p>We haven&#x27;t bothered with a custom domain SSL certificate as the herokuapp.com subdomain has been just fine.<p>Fortunately I was monitoring the endpoint as I started getting SSL expiry warnings a few weeks ago.<p>It seems heroku is serving an old certificate for &lt;our-app-name&gt;.herokuapp.com, issued April 2019 and expiring 22nd June:<p>```
$ curl -v --head https:&#x2F;&#x2F;&lt;our-app-name&gt;.herokuapp.com&#x2F;
* Connected to &lt;our-app-name&gt;.herokuapp.com (52.19.225.66) port 443 (#0)
[snip]
* Server certificate:
*  expire date: Jun 22 12:00:00 2020 GMT
*  subjectAltName: host &quot;&lt;our-app-name&gt;.herokuapp.com&quot; matched cert&#x27;s &quot;<i>.herokuapp.com&quot;
</i>  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
```<p>It&#x27;s a wildcard cert for <i>.herokuapp.com but it&#x27;s different from the current one I see if I curl the root domain:<p>```
$ curl -v --head https:&#x2F;&#x2F;herokuapp.com&#x2F;
</i> Connected to herokuapp.com (34.194.84.166) port 443 (#0)
* Server certificate:
*  expire date: Aug  2 02:13:11 2020 GMT
*  issuer: C=US; O=Let&#x27;s Encrypt; CN=Let&#x27;s Encrypt Authority X3<p>```<p>It seems they&#x27;ve transitioned to Let&#x27;s Encrypt for the wildcard domain, but it isn&#x27;t being served for app subdomains. I&#x27;ve checked a few other subdomains and see the same thing:<p>* govuk-prototype-kit.herokuapp.com
* heroku-status.herouapp.com
* juice-shop.herokuapp.com<p>I&#x27;ve been raising this with support since T-30 and they just keep saying things like:<p>&gt; Our concerned team is aware of it and they are actively working on the renewal process. We&#x27;ll get the new cert in there well before the expiration, and there will be no disruption of service.<p>Now we&#x27;re at 7 days I&#x27;ve lost confidence that support has even forwarded my ticket to the right team.<p>I suspect in 7 days we&#x27;re gonna see a lot of things break...
======
jrockway
Something that's nice about Let's Encrypt is that it forces you to change
something every few months. After the first couple months, you'll probably get
your issues worked out. If you just change certs every few years, then every
few years you have some sort of disaster because of the "well we fixed it, we
don't have to worry for two years" effect.

A broader lesson is the importance of "trying out" rare events, even before
that rare event actually happens. If depends on a service with a certain SLA,
it's pretty dangerous when that service has 100% uptime. You never get to see
what happens when it does go down, which it did promise you it will. Some
people track their error budget, and at the end of the accounting period,
break their service in accordance with the SLA. Then you get to see what
happens when it does go down. (Ref:
[https://queue.acm.org/detail.cfm?id=2371516](https://queue.acm.org/detail.cfm?id=2371516))

~~~
tialaramex
Although, speaking of Let's Encrypt, there will be a series of disruptive
events over the next 18 months or so.

* Soon (although when exactly I'm not sure because it has been delayed at least once) the Let's Encrypt systems will tell compliant ACME clients that the "correct" intermediate is Let's Encrypt's ISRG-signed X3 intermediate. This is a different certificate for the same X3 private key you're used to but not signed by the same trust root. If you use a correct client and have done things properly, this may cut off TLS clients for your systems that don't trust ISRG (the charity which runs Let's Encrypt). Six year old Android phones, the Windows XP system you know should have been retired, a VoIP desk phone running out-of-date firmware, stuff like that.

* In March 2021 the X3 Intermediate expires. If your certificate software was not compliant with ACME, or you manually overrode it to use the old certificates to avoid the problem in the previous item, things break now. More things, and worse. Although...

* Maybe before March 2021 the Let's Encrypt systems stop issuing from those soon-to-be-obsolete Intermediates and use newer ones instead perhaps named Y3 and Y4. In this case if you've jury rigged things (in an ACME non-compliant way) to keep using the old X3 intermediate that'll break suddenly after your renewal. Common web browsers may not trust the nonsense you're emitting, exactly which browsers break may vary depending on exactly what stupid things you did, but chances are you haven't tested and don't know. If you are using a compliant client then modern browsers are all fine, but archaic stuff breaks suddenly.

* In September 2021 the DST Root X3 root expires. If you have somehow clung on to trust via this root, whether through your own effort or via trust path discovery code inside client systems, that goes away instantly. Any systems that don't trust ISRG will refuse to trust your certificates, no matter how often you re-issue them and reconfigure things, those clients themselves need updating urgently and you probably have no way to do that. Oops.

~~~
Denvercoder9
That sounds like just one maybe-disruptive event that manifests itself
differently if you keep working around it instead of dealing with it properly.
If you need to deal with it at all - I suspect most systems that still need to
connect to the internet trust the ISRG root nowadays.

~~~
EB66
> I suspect most systems that still need to connect to the internet trust the
> ISRG root nowadays.

There are tons of systems that do not -- particularly in the enterprise. I
manage web servers for a mission-critical healthcare-related SaaS. We
occasionally encounter TLS issues even with Globalsign root certificates --
far more distributed than ISRG.

We ended up switching to DigiCert last year and it helped reduce the number of
TLS-related failures reported to us.

We could never switch to Let's Encrypt / ISRG for that reason. Even if ISRG
has 95% distribution of their root certificate, that's not good enough for
mission-critical enterprise.

I'm not at all surprised that Heroku had to roll back their TLS certificate
back to DigiCert -- DigiCert is what you want if need compatibility with the
highest number of clients.

------
bad_user
Why would you pin a certificate that you did not generate for a domain that's
not yours?

~~~
godzillabrennus
My guess is that it is some larger corporate client with a middleware app they
pinned the cert into. An app they built on Heroku hurriedly because it was
fast and cheap to get started and they didn’t expect to need to scale. Then as
can happen it became important and they scaled anyway. They probably lost the
talent that built it so they don’t even remember how it works.

All of this is assumption based on how they seem to buy enough compute at
Heroku to have sway over rolling back this change of cert provider.

~~~
yjftsjthsd-h
Ugh, yes, I assure you that some corporate clients will even try to pin the
actual leaf certificate; pinning an intermediate or root is almost good
behavior for them. (Honestly, the number of times I had to tell _our_ support
people that no, we would _not_ support customers trying to pin our AWS-issued
certificates, and no, I _couldn 't_ promise to notify them even if I wanted to
since AWS could just rotate them at will...)

------
jamil7
Heroku also doesn't enforce any verification that you own a domain name.
Another user can simply add any domain they like to their app if you haven't
claimed it by adding it to your app first. Regardless of ownership and you
will no longer be able to add your own domain to your app getting a generic
"domain is already in use" error. Happened to me a few years ago, had to reach
out to support and prove I owned the domain. They made me verify I owned it
and fixed it but said theres nothing they can do going forward. Granted it's a
total edge case but was still an unnexpected experience, maybe it's fixed now
who knows.

Edit: Looks like this is fairly common on PAAS so my original comment isn't
that relevant.

~~~
swiley
How does GitHub’s gh pages handle this? I don’t remember them doing anything
either.

~~~
derision
Pretty sure you have to set some DNS records for gh pages

~~~
swiley
Yes you have to point a record at the gh pages if you want to use dns with it,
but I don’t think their server checks for that.

~~~
awirth
Is it possible to subscribe to a firehose of .COM NS record updates through
one of those fancy things like dnsdb? If so, perhaps there's an opportunity
here for exploiting that race condition en-masse for services that support
direct NS-style delegation, like netlify.

------
photonios
It already broke a few days ago:
[https://status.heroku.com/incidents/2045](https://status.heroku.com/incidents/2045)

~~~
tuna-piano
Yep, we were down during that time. I don't know what Heroku is doing with
this certificate thing, but seems like a mess.

~~~
jdxcode
If you're pinning the SSL cert then it's not something they can fix, it's
something you need to change on your end.

~~~
photonios
We went down. We're not pinning the SSL cert. We're behind CloudFront, and
CloudFront didn't trust the new cert and stopped forwarding traffic.

------
abhisuri97
Wow. I really hope they’ll get it done before the expiration date but I always
thought they’d be renewing months in advance at minimum. Are they trying to
negotiate something?

~~~
tialaramex
Get what done?

Heroku replaced their wildcard certificate in plenty of time but many
customers do not anticipate anything changing ever and will fiercely resist
this simple fact, so for those customers stuff blew up.

Remember the Y2K problem is the result of software written not in 1901 or even
just 1981 but even well into the 1990s with the calm certainty that all years
begin 19xx.

Heroku can try brown out policies, but this sort of customer intransigence is
very difficult to defeat. The customer is quite certain they're right, who
ever heard of "change" anyway? Everyone knows that the world is a flat plane,
fixed in space, eternal and unchanging, this pinning rule I wrote in 2019
worked then, therefore it is still correct now.

~~~
tinus_hn
This is called ‘enterprise’ and everyone in it will tell you a million excuses
why it’s the only way.

------
_raul
There's more context in this Heroku Changelog
[https://devcenter.heroku.com/changelog-
items/1813](https://devcenter.heroku.com/changelog-items/1813)

"On Tuesday June 9th 2020 Heroku changed the certificate used for terminating
TLS for built-in <appname>.herokuapp.com hostnames from a certificate issued
by DigiCert to one issued by Starfield/AWS. This change was rolled back on
June 10th because a small subset of Heroku customers had pinned apps to the
DigiCert certificate or had apps that could not establish a chain of trust
with the new certificate for other reasons.

A new DigiCert-signed certificate will replace the current one before June
22nd (when it expires)."

