
Technical Details on the Recent Firefox Add-On Outage - headalgorithm
https://hacks.mozilla.org/2019/05/technical-details-on-the-recent-firefox-add-on-outage/
======
jzl
Looks like a good read. I haven't finished reading it yet, but there's
something I still don't get ...

Windows and macOS both have a signing infrastructure for apps. The rules of
that infrastructure dictate only that apps must have been signed by a valid
certificate at the time they were signed. That way old app downloads don't
need to be periodically re-signed just to account for expiring certificates. I
can download a 5-year-old version of 7zip or whatever and it runs just fine
because it was signed with something valid to the timestamp in the signature.
The process of distributing desktop apps would be utterly insane if this were
not the case.

Not following this model for browser plugins seems unnecessarily cumbersome.
Is it really worth requiring all browser plugins to be signed by a _currently_
valid certificate? Is there a document or blog post where this is argued to be
more appropriate?

I get that it arguably leads to more stringent security, but I'm not convinced
by the delta improvement of that model over the desktop model, given the
additional downsides. And the "let everything expire after a few years and
resign it" process should not be used as a substitution for revocation. After
all, if it were determined retroactively that a malware extension was signed
does it really help everyone that it can't be loaded in a year or two, given
the damage it could cause right now?

~~~
johncolanduoni
For the time stamp method to work, you need a trusted mechanism to attest that
the timestamp is correct, otherwise the mechanism is useless (an attacker with
an outdated private key can just backdate the timestamp in the executable and
then sign it). Windows code signing uses a server Microsoft runs to provide
this, and Mozilla would need to do the same.

I’m not saying they shouldn’t, but it is a significant piece of complexity.

~~~
StavrosK
Can you explain why you need the _current_ time anywhere in this? Say I
download a ten-year-old addon, it's signed by a valid signature from that
root, with a valid signing date. What's the problem?

Are we worried that someone will steal an old/expired cert _and_ have control
over a user's clock?

~~~
cortesoft
Imagine you have an old expired key... you take your new malicious extension,
and sign it with the expired key and a time stamp that says it was signed at a
time the key was still valid.

Without some other verification mechanism, you can't tell the difference
between this and an actual signature signed when the key WAS valid

~~~
josteink
> sign it with the expired key and a time stamp that says it was signed at a
> time the key was still valid.

The whole point of a trusted timestamp is that such signature cannot be made
for a fraudulent date, otherwise it would be utterly pointless.

This scenario and threat model does not exist if timestamping is correctly
implemented.

~~~
tialaramex
Nope.

RFC 3161 timestamps - which is what we're discussing here - can be
fraudulently constructed for any timestamp value by someone who has the
private key for the TSA (Time stamping authority). So what your parent
described is easily possible: A system that relies on RFC 3161 timestamps has
to trust that

* any cryptographic hash algorithms used remain safe

* any public key signature methods used remain safe

* the TSAs retain control over their private keys for as long as you continue to accept timestamps from that TSA

This is a big ask, and in practice the code signing systems you're probably
thinking of just don't care very much. A state actor (e.g. the NSA) could
almost certainly fake materials for these systems, we know that this has been
done (presumably by the NSA or Mossad) in order to interfere with the Iranian
nuclear weapons programme in the past.

You _can_ build a system that has tamper evident timestamping, but it's much
more sophisticated and has much higher technical requirements. That's what
drives the Certificate Transparency system. CT logs can prove they logged a
specific certificate within a 24-hour period, and monitors verify that their
proofs remain consistent, the to-be-built Gossip layers allow monitors to
compare what they see in order to achieve confidence that logs don't tell
different stories to different monitors. But to achieve this a CT log must be
immediately distrusted if it falls off line for just 24 hours or if an error
causes it to not log even a single timestamp certificate it issued. Massive
Earthquake hit your secure data centre and destroyed the site? You have 24
hours to get everything back on line or be distrusted permanently. Bug in a
Redis configuration lost one cert out of 4 million issued? You are distrusted
permanently. Most attempts to build a CT log fail the first time, some outfits
give up after a couple of tries and just accept they're not up to the task.

~~~
stupidthrottle
> can be fraudulently constructed for any timestamp value by someone who has
> the private key for the TSA

Sure. Which is why these are heavily secured and guarded. Just like the keys
for any cert, and highly trusted root certs in particular.

Any private/public crypto system can be compromised if the private keys are
leaked. Everyone knows that.

That however is in no way a good argument for not using timestamps.

~~~
tialaramex
RFC 3161 timestamps are used because they let people do something Mozilla
doesn't care about at all and which was largely irrelevant here.

Alice the OS Vendor wants to let Bob the Developer make certificates saying
these are his Programs, she is worried Bob will screw up so his cert needs to
have a short lifetime, but her OS needs to be able to accept the certs after
that lifetime expires so users can still run their Programs. So, Bob makes
certificates and uses Trent's public TSA that Alice authorised to prove they
were made when they say they were. Alice only has to trust Trent (who is good
at his job) for a long period, and Bob who can be expected to screw up gets
only short-lived certificates.

But Mozilla's setup doesn't have these extra parties. There is intentionally
no Bob in Mozilla's version of the story, they sign add-ons themselves, so
timestamping plays no role. If a 25 year TSA would be appropriate (hint: it
would not) then a 25 year intermediate cert would be just as appropriate and
simpler to implement for Mozilla.

------
DangerousPie
My point of view as a long-time Firefox user that cares about privacy but also
knows we live in an imperfect world: It obviously sucks that this happened but
I think they handled it very well.

The bug was fixed so quickly that I wouldn't even have realized it had
happened if it hadn't been for the thread here on HN. My extensions hadn't
even been disabled yet by the time the patch came out. And pushing out the
hotfix through studies followed by a new version probably ensured that a large
fraction of the "average joe" userbase didn't even realize there was a
problem.

So obviously there are some improvements to make for the future but I think
some of the criticism over the last few days has been a bit harsh. Firefox is
still my preferred browser by far.

~~~
SECProto
> The bug was fixed so quickly that I wouldn't even have realized it had
> happened if it hadn't been for the thread here on HN

Maybe this is a timezone thing, but I was in East Asia, and I had to deal with
the internet for close to 36 hrs (android) with no ublock. It was almost
enough to look for a new browser (but browsers with adblock on Android are few
and far between - so instead I just didn't use the internet as much for a day
or two.). Part of that delay was play store being slow to push it, as I recall
seeing the binaries somewhere a while sooner.

~~~
worble
There's a couple of options of Android; you could use the DDG browser[0] or
the Privacy Browser[1] (which I know sounds dodgy but seems legit). They don't
have 'adblock' exactly but I think they implement a lot of the same lists such
as EasyList.

[0][https://f-droid.org/en/packages/se.johanhil.duckduckgo/](https://f-droid.org/en/packages/se.johanhil.duckduckgo/)

[1][https://f-droid.org/en/packages/com.stoutner.privacybrowser....](https://f-droid.org/en/packages/com.stoutner.privacybrowser.standard/)

~~~
SECProto
Thanks for those! I usually browse on desktop, so hadn't looked into other
browsers, but I'll keep these alternative browsers in mind (and comment
history) in case a situation comes up again. The only other instance where I
considered switching was the whole forced Mr. Robot addon debacle a year or
two back.

------
sohkamyung
Related to this: Mozilla has deleted Telemetry data for those users who
enabled Telemetry to get the hot-fix [1]

[1]
[https://twitter.com/firefox/status/1126593558490693632](https://twitter.com/firefox/status/1126593558490693632)

~~~
sohkamyung
Edit: a error on my part. Mozilla is deleting ALL Telemetry data collected
during that time period, not just from certain users. From the post linked
from the tweet [1]

>In order to respect our users’ potential intentions as much as possible,
based on our current set up, we will be deleting all of our source Telemetry
and Studies data for our entire user population collected between
2019-05-04T11:00:00Z and 2019-05-11T11:00:00Z.

[1] [https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-
thi...](https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-things-go-
wrong/)

------
jrochkind1
Interestingly, it seems like it's not just that they didn't realize the
certificate expire date was zooming past; they didn't even realize that
particular certificate expiring _was a thing_. If they had noticed 6 months
ago it was about to expire -- they still would have had to figure out _what to
do about it_ , there wasn't an actual documented procedure in place to swap in
a new non-expired cert without disruption to FF users with existing signed
add-ons installed. If they had known the expire date was upcoming, they just
would have had more time to figure out what to do about it in a leisurely
fashion.

I wonder if the engineers who implemented the original system had a clear idea
of what would be done when the cert expire approached and it just never got
documented (or FF accidentally changed to make it no longer applicable?), or
if they just figured they'd figure that out years later when the date
approached.

------
jeffchien
I would've liked to see some reflection or even just acknowledgment about the
fact that they intentionally disabled the "xpinstall.signatures.required"
setting on Windows and OSX. I hope it's at least in the formal postmortem.

~~~
callahad
That's covered at [https://blog.mozilla.org/addons/2015/04/15/the-case-for-
exte...](https://blog.mozilla.org/addons/2015/04/15/the-case-for-extension-
signing/), which is linked from ekr's post. I encourage you to read the
rationale as a whole, but the specific question you're asking is addressed
here:

> Many developers have asked why we can’t make this a runtime option or
> preference. There is nowhere we could store that choice on the user’s
> machine that these greyware apps couldn’t change and plausibly claim they
> were acting on behalf of the user’s “choice” not to opt-out of the light
> grey checkbox on page 43 of their EULA. This is not a concern about
> hypotheticals, we have many documented cases of add-ons disabling the
> mechanisms through which we inform users and give them control over their
> add-ons. By baking the signing requirement into the executable these
> programs will either have to submit to our review process or take the
> blatant malware step of replacing or altering Firefox. We are sure some will
> take that step, but it won’t be an attractive option for a Fortune 500
> plugin vendor, popular download sites, or the laptop vendor involved in
> distributing Superfish. For the ones who do, we hope that modifying another
> program’s executable code is blatant enough that security software vendors
> will take action and stop letting these programs hide behind terms buried in
> their user-hostile EULAs.

~~~
Wowfunhappy
Does Windows et al _really_ not provide a mechinism for saving privileged
settings in a tamper-resistant way? I frankly find that hard to believe. How
does other software solve this problem?

There are of course always workarounds on an open platform like
Windows/Mac/Linux, but the threshold isn’t “impossible”, it’s just “as
difficult as injecting into the browser’s code.”

Edit: For example, what if the config file contained a checksum of the file's
contents + the user's hardware? If the setting is changed by the user within
Firefox, the checksum is updated and everything works—if the checksum is
invalid, settings are reset to the default.

Video games will occasionally do this type of thing with their config files.
Modders often figure out the formula—but again, the idea here isn't to make
editing the file _impossible_ , it's to make it _as difficult_ as injecting
into the Firefox executable.

~~~
djsumdog
I mean, the Windows UAC and the MacOS password prompt that elevate user
access, but people often click through those. I can see their argument here.

~~~
TeMPOraL
No amount of security measures can stop users from being tricked by social
engineering attack into self-pwnig themselves. At some point we need to stop
taking control away from users, because this road ultimately leads to turning
computers into cable TV.

------
lllr_finger
It might not have helped here, but updating certs every several years always
leads to problems from my experience - people move on, processes get lost or
outdated, etc. I'll accept the yearly annoyance to avoid those issues every
time.

~~~
Someone1234
Agreed.

My initial reaction when Let's Encrypt had you re-issue every 90 days was
negative, but I was wrong. Very wrong. A 90 day re-issue forces you to have
working re-issue infrastructure and procedures, and therefore you're less
likely to get stung by an accidental expiration.

Long expirations are a trap, a very easy trap to fall into.

~~~
drtillberg
Why not update long-duration keys every 90 days? That way you're never close
to expiration, ever, best of both worlds.

~~~
cyphar
Because if you make it optional then nobody does it (except you) and then
you're back to square one. By mandating a 90-day expiry, LetsEncrypt forced
people to automate the process -- and everyone is on the same page.

It should be noted that most LetsEncrypt tools will renew a certificate when
it is 30 days from expiry, so if you run the renew script every week (or day)
you're also never close to expiry.

~~~
bscphil
>By mandating a 90-day expiry, LetsEncrypt forced people to automate the
process -- and everyone is on the same page.

Ironically, I had the opposite problem. I used to be on top of things like
cert expirations, but now I just let certbot do everything. The problem (at
least in my case) was that even though certbot updated the cert on time, it
doesn't restart / reload nginx so that it picks up the new cert. My site was
up for the full ~30 days between the renewal and the expiration of the old
cert. So my site went down _because_ of letsencrypt's cert renewal policy.

(I now have a script set up that reload's nginx's configuration whenever the
certificate is updated.)

------
ghostly_s
If anyone's wondering if this answers the Actual Question of why the cert was
allowed to expired, don't waste your time, it doesn't. I guess implicitly
that's a "social detail"?

~~~
sfink
The post-mortem hasn't happened yet. ekr's post is a preliminary description
of what happened, everything he felt confident saying in advance of the post-
mortem and so necessarily heavy on technical details but light on process
details.

~~~
djsumdog
I mean, it's fair to say it's oversight? I wonder if the post-mortem will
reveal if it was on someone's radar and slipped through the cracks or if it
was never put into the schedule system they use for certification expiration
to begin with.

------
tedunangst
> For the other groups we are developing a patch to Firefox that will install
> the new certificate once people update. This was released as a “dot release”
> so people will get it — and probably have already — through the ordinary
> update channel. If you have a downstream build, you’ll need to wait for your
> build maintainer to update.

Why not link to the xpi that can be installed now?

~~~
yellowapple
> Why not link to the xpi that can be installed now?

This is the crux of my remaining frustration with how Mozilla handled this
issue. That XPI should've been front-and-center in all the articles that
detailed the fix. And yet, instead of something like...

"If you have Studies enabled, a fix should apply automatically. If it hasn't
yet, or if you have Studies turned off (or are using a version which does not
support Studies), you can install the hotfix add-on [here](URL to XPI)."

...pretty much all the official messaging ended up like so:

"If you have Studies enabled, a fix should apply automatically. It may take up
to 6 hours; please be patient and wait for it. If you don't want to (or can't)
turn on Studies, you're SOL until we push out a point release (and further SOL
if you're at the mercy of a Linux package maintainer or you want to use a
version of Firefox that still supports XUL-based addons)."

The notion that this was a deliberate ploy to get more people to turn on
Studies is surely conspiracy-theorist mumbo-jumbo, but nonsense like this
makes me wonder.

~~~
callahad
> _The notion that this was a deliberate ploy to get more people to turn on
> Studies is surely conspiracy-theorist mumbo-jumbo, but nonsense like this
> makes me wonder._

This is not the case. Please see my response downthread:
[https://news.ycombinator.com/item?id=19872490](https://news.ycombinator.com/item?id=19872490)

~~~
yellowapple
I know full well it's not _actually_ the case, but that doesn't make it not
_feel_ like it could be the case. It feels scummy, and I'd expect Mozilla to
be above that scumminess.

Like, just link to the XPI. Not that hard. The unexplained reluctance to do so
is suspicious.

~~~
callahad
Gotcha. Manually installing the hotfix XPI makes cleanup a bit harder now that
we have a proper fix. E.g., without coming from Studies, there's no study to
ever _end_. Direct installation also makes it harder to quickly respond to any
bugs we might discover in the initial revision of the hotfix.

Now that we have a stable fix, we _will_ publish an XPI with the option of
direct installation for users of older, unsupported versions of Firefox (all
the way back to 52) who have opted out of automatic updates.

~~~
yellowapple
I see. Some follow up questions:

> Manually installing the hotfix XPI makes cleanup a bit harder now that we
> have a proper fix. E.g., without coming from Studies, there's no study to
> ever end.

The language around enabling Studies for the hotfix also claimed that once the
hotfix installed, one can feel free to turn off Studies. Could similar
language not have been included for the XPI approach (e.g. "once the fix is
applied, you can uninstall this add-on")? Or is this a case where the
extension does have to be installed (at least until the user upgrades to a
point release with a fixed certificate)?

Alternately, do extensions have the ability to uninstall themselves? If so,
then perhaps the extension could install the new certificate and immediately
uninstall itself (or, in the "extension has to be installed for the fix to
exist" scenario above, uninstall itself if it detects itself running on an
updated Firefox and/or flag itself as incompatible with Firefoxen newer than
the latest affected version)?

Alternately, is there no way for Firefox itself (e.g. in a point release) to
explicitly blacklist an extension?

Alternately, is it possible to revoke the certificate/signature for that
extension such that Firefox deems it invalid and disables it (using,
presumably, the same mechanism and rationale as what caused this particular
bug)?

Seems like this is a problem with multiple potential solutions besides "just
do it as a Study". Even if it really is/was unsolvable, I feel like power
users would be perfectly happy with getting the quick fix in exchange for
subsequent cleanup being on them; ain't ideal, but it's better than waiting
for multiple hours for Studies to work its magic.

> Direct installation also makes it harder to quickly respond to any bugs we
> might discover in the initial revision of the hotfix.

I'm sure there are some people out there who would be happy to test the XPI
while having Telemetry enabled so y'all can get all that juicy fresh debugging
data :)

------
toyg
_> users should be able to opt-in to updates (including hot-fixes) but opt out
of everything else_

Finally some good news. This is what I suggested in one of the previous
threads: there should be a delivery channel for important updates, and a
channel for experiments/telemetry/whatnot. Some other HNer said it was an
unrealistic expectation "because manpower". Guess what, it isn't. This is how
things should _always_ be.

~~~
mook
They did have that capability (see
[https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-
ons/Pr...](https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Process)
), at least the linked repository had commits in 2016. I can't tell from a 30
second search why that no longer works, just that it is a replacement of a
similar previous capability.

~~~
sciurus
This started before my time at Mozilla, but I think several different systems
grew out of the "go faster" initiative. Of them I believe the fastest and most
flexible means of shipping changes is Normandy, which is primarily used for
our shield studies. Like the blog post says, we need to revisit this.

(Disclosure: I work for Mozilla)

------
drtillberg
The Firefox update required an administrative login in my windows system at
work, which I don't have have. Normal updates haven't required that. So far
I've just left it broken and it keeps prompting me for an administrative login
on launch.

The article doesn't explain why elevated privileges are required to apply the
update.

~~~
kbrosnan
Does the current user have rights to write to the Firefox install location? If
not then elevation is required to overwrite the files.

~~~
hcs
The current user may not need to have those rights if the maintenance service
is installed, which allows the updater to run with higher privileges. I don't
know of any reason why the recent updates would be any different, though.
There may be an unrelated issue that broke "silent" updates for drtillberg.

(I work on the Firefox updater and am a Mozilla employee)

------
Tsubasachan
My heartfelt condolences to everyone who has had to browse the internet
without an ad blocker.

Nobody deserves that.

~~~
driverdan
This is why you should have multi-tier defenses. An ad blocker in the browser,
blackholed domains in your hosts file, and use DNS servers that also
blackholes ad domains.

~~~
BuckRogers
I used to do all 3 in the past, 4 with domain blocking done in router
configuration, and am no longer a fan. It made troubleshooting any issue that
may pop up too much of a hassle. Now I just use single-source blocking at the
application layer. When Firefox disabled extensions, I simply used Edgium
until they had it fixed.

------
Faark
> In theory, fixing a problem like this looks simple: make a new, valid
> certificate and republish every add-on with that certificate. Unfortunately,
> [..]

I'd expect addon usage to follow a pareto distribution. Thus resigning the
most important ones would have helped a lot of users. Why didn't they start
going this route anyway? Not enough manpower for this to not diverted
resources from the other more important fixes?

~~~
sciurus
I think this is covered in the blog post, but I'll take another stab at
explaining it from my perspective:

Republishing was one of the options we were investigating early on. However,
the problem is that it only fixes things once you check for and install the
updated version signed with the new certificate. Firefox would still have
disabled your installed version that had an expired certificate.

Firefox checks for addon updates every 24 hours, but it checks in with
Normandy every 6 hours. Thus once we had stopgap fixes shipping to users via
Normandy, and a proper fix of a new Firefox release in progress, republishing
addons wasn't necessary.

(Disclaimer: I work for Mozilla)

~~~
Faark
Thanks for your response.

An update being deactivated doesn't trigger an update check for said addon?
Well, complexity of a graceful update attempt for a not loadable addon
probably outweighs the benefit of such rare cases.

But the main reason I had a hard time with you guys discarding this path is
probably addon stats like [0]. The spike in downloads made me think it could
have helped at least some users. But on second thought, it might also have
been caused by extensions getting re-enabled post fix. 2m downloads at 4m DAU
makes me guess this was caused by something automatic.

[0] [https://addons.mozilla.org/en-US/firefox/addon/ublock-
origin...](https://addons.mozilla.org/en-US/firefox/addon/ublock-
origin/statistics/?last=30)

~~~
sciurus
This is a good point; I'll try to make sure it's raised when we conduct our
postmortem.

(Disclosure: I work for Mozilla)

------
parliament32
>An important feature here is that the new certificate has the same subject
name and public key as the old certificate, so that its signature on the End-
Entity certificate is valid.

Shouldn't it be impossible to generate a new cert (with a different expiry
date) that ends up having the same public key as an existing cert?

~~~
bityard
No, one public/private key pair can be used to generate or sign as many
certificates as you like, as often as you like.

~~~
parliament32
No I'm not talking about the root, I mean they generated a new certificate
(the intermediate) (with a new private key) that had a public key identical to
an existing certificate -- you shouldn't be able to do this, public keys can't
be "specified" afaik, they're derived from your private key and the signer's
public key.

~~~
rossdavidh
Isn't that the point of having the root cert and the intermediate? Otherwise,
why not just have everything keyed to the root?

~~~
rndgermandude
Because a root typically has a longer expire date and thus has to be stored
extra secure and handled extra safely. You'd usually store it in some hardware
security module that is operated by an airgapped system or something like that
that requires physical access at the very least. Replacing the root often
requires shipping new versions of your software that disables the old root and
bakes in a new root.

If an intermediate certificate becomes compromised, you can revoke it and
issue a new intermediate certificate with your still secure root without the
need to push out new binaries.

------
silversconfused
For anyone who misses the old days when a browser only did what you told it
to, here is a solution: GNU icecat is a nice firefox esr fork with the mozilla
call-home bits all turned off by default. It's very pleasant to use.

~~~
djsumdog
It doesn't have the new Quantum rendering engine though, right?

~~~
silversconfused
I'm pretty sure it does.
[https://en.wikipedia.org/wiki/Quantum_(Mozilla)](https://en.wikipedia.org/wiki/Quantum_\(Mozilla\))
says quantum shipped in 57.

------
sciurus
There is a related blog post about what Mozilla is doing with the data
collected from users who enabled Studies in order to get the hot fix.

[https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-
thi...](https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-things-go-
wrong/)

TL;DR is "In order to respect our users’ potential intentions as much as
possible, based on our current set up, we will be deleting all of our source
Telemetry and Studies data for our entire user population collected between
2019-05-04T11:00:00Z and 2019-05-11T11:00:00Z."

(Disclaimer: I work for Mozilla)

~~~
jzl
As someone who reluctantly re-enabled Studies to get the fix, I appreciate
this.

------
hartator
> Second, we immediately pushed a hotfix which suppressed re-validating the
> signatures on add-ons.

What’s the point of re-validating of installed add-on in the first place then?

------
PudgePacket
> Note that this is the same logic we use for validating TLS certificates, so
> it’s relatively well understood code that we were able to leverage.

Don't know whether to be happy or scared that the TLS validation code is
"relatively well understood" :D ! I assume it's just a sub-optimal choice of
phrasing.

~~~
josteink
I’m scared that they (still?) confuse regular HTTPS TLS-validation with code-
signing.

These are two entirely different domains which follows entirely different
rules.

Current Firefox behaviour is still broken, even after the “fix”.

------
shanxS
Certificate expiration, if only we can figure way to see it coming. Oh wait,
metrics!

------
josteink
This is basic code-signing and “time-stamping” is not mentioned once in the
post-mortem, despite lack of that was exactly what caused this issue in the
first place.

That’s not inspiring a lot of confidence, to be honest.

------
koliber
I don't understand one thing. How were they able to generate a new certificate
with the same subject, a different expiration date, and the same signature.

I assume I missed something in reading this.

~~~
cyborgx7
It's a new Date but still the same key. So the signature of the root
certificate for the intermediate certificate is different, but the signature
the intermediate key generates are the same.

------
pcvarmint
Why is no-one talking about the Dissenter ban? Isn't that what prompted this
whole fiasco? [0][1][2]

0\. [https://elgan.com/blog/why-mozilla-and-google-are-wrong-
to-b...](https://elgan.com/blog/why-mozilla-and-google-are-wrong-to-ban-gabs-
dissenter-extension)

1\. [https://discourse.mozilla.org/t/the-removal-of-the-
dissenter...](https://discourse.mozilla.org/t/the-removal-of-the-dissenter-
extention/38140/13)

2\.
[https://www.youtube.com/watch?v=f0Cc8RpqH1g](https://www.youtube.com/watch?v=f0Cc8RpqH1g)

------
johnchristopher
> Even on Monday, we still had some users who hadn’t picked up either the
> hotfix or the dot release, which clearly isn’t ideal.

My kubuntu boxes had the patch wednesday, not before.

------
gpm
THIS IS WRONG: Preserved to make below reply not seem out of context.

> Second, we immediately pushed a hotfix which suppressed re-validating the
> signatures on add-ons.

Wait, that's not right is it!?

The only hotfix I'm aware of (and I was following this pretty closely, as you
might guess by my dozens of comments on the topic) was the addon installed via
the studies system.

That addon didn't suppress the re-validation of signatures, it installed a new
certificate and then _triggered_ the re-validation of signatures immediately.
It left the validation that happens on a 24 hour cycle alone.

~~~
KwanEsq
There was a hotfix before the one that installed a new cert:

> hotfix-reset-xpi-verification-timestamp-1548973•Complete

> This study sets app.update.lastUpdateTime.xpi-signature-verification to
> 1556945257.

~~~
gpm
Oh, you're completely right. Oops.

------
jsjohnst
What I don’t see mentioned anywhere is why the heck was this a last minute
scramble? Don’t they know when their certificates are going to expire!?

~~~
cheeze
Probably not. They probably know when most of their certs schedule but likely
weren't monitoring this intermediate code signing cert (which is different
than a TLS/webserver leaf/intermediate cert).

It seems super simple to do, but in practice IMO is harder than it seems. Most
major cloud providers have been hit by at least one cert expiry causing an
outage in the past year... Hell, likely in the past month.

This doesn't surprise me at all, certs are hard.

------
throw7
i lost all my custom multi-account containers after getting a version that has
the cert fixes (firefox is still broken on fedora; i had to download the
testing version today to get it back working).

either today's engineers are sub-standard or the foxes rule the henhouse.

~~~
floatingatoll
If you disable the addon, restart Firefox, and enable the addon, do your
custom containers get erased again?

~~~
lysp
It's my understand that they'd be deleted.

Current bug for issue:
[https://bugzilla.mozilla.org/show_bug.cgi?id=1549204](https://bugzilla.mozilla.org/show_bug.cgi?id=1549204)

This was discovered after the expiring caused a similar process to kill data.

------
jokoon
I hope this was not a government attempt to snoop and that mozilla did not
miss any detail...

------
keyle
So my first thought was that a lot of ads would have been printed during that
24 hrs? Anyone notice a blip in revenues of sorts?

I mean that's the most crucial add-on to ever work: ad blockers.

When I visit a site with ad-blocker off I feel compelled to clear all caches
and cookies and go have a shower.

~~~
djsumdog
None of the ad companies are going to publicly talk about that data. They
might have seen a blip, but I doubt it was anything major. Firefox doesn't
have the market share it once did either.

------
known
Catalog of classic Firefox add-ons created before WebExtensions apocalypse
[https://github.com/JustOff/ca-archive](https://github.com/JustOff/ca-archive)

------
zelpo
No problem for me. The only add-on I use and recommend is Pocket. Pocket helps
me keep my stuff organized, when im on the Go.

------
some13
I switched to falkon and hope to never go back to firefox.

------
ekianjo
I love how they paint the picture that certificates "unfortunately expired" as
if it were an act of god or something. Surely one cannot see it coming! No
mention at all why nobody was checking certificates expiry.

~~~
lifthrasiir
The post seems to imply that it was a simple overlook (which is frequent when
every such thing is not formally tracked). I agree that it is hard.

> We’ll be running a formal post-mortem next week and will publish the list of
> changes we intend to make, but in the meantime here are my initial thoughts
> about what we need to do. First, we should have a much better way of
> tracking the status of everything in Firefox that is a potential time bomb
> and making sure that we don’t find ourselves in a situation where one goes
> off unexpectedly. We’re still working out the details here, but at minimum
> we need to inventory everything of this nature.

~~~
ekianjo
> a simple overlook

I'm sorry but when your business is to enable 30 000 extensions to work your
business is also to check that certificates that enable such extensions don't
fail tomorrow. That's the core of one's job, not even a side project or
something.

~~~
lifthrasiir
I don't claim to know what actually happened, but one possible cause is that
the HSM (required for issuing and renewing intermediate certificates,
mentioned in the post) might require a right person and right schedule to
operate. The use of an HSM means that you don't normally use it to issue
certificates and you _should_ only touch it a few times a year. As a result
there may have been only a few people with an effective knowledge of the HSM;
when they can't be scheduled for some various reasons, well, it may be
forgotten. While it of course sucks, something like this happens anywhere
anytime.

------
acqq
> Second, we need a mechanism to be able to quickly push updates to our users
> even when — especially when — everything else is down.

They should simply streamline the "normal" updates for that purpose, not
invent the new "channels."

Specifically, not all binaries in the directory should be changed just to push
an update where only few lines of the code are different.

------
cyborgx7
> We clearly need to adjust our processes both to make this and similar
> incidents it less likely to happen and to make them easier to fix.

I knew it. They are going to use THEIR fuck-up to justify why they need even
more remote control and access to firefox installed on users machines.

------
hartator
> because they want to run old-style add-ons, but many of these now work with
> newer versions of Firefox.

Except it wasn’t working for this few days. Anyway older versions of Firefox
wasn’t signing add-on, so they are not concerned by this fix.

~~~
silversconfused
Wrong. I lost my addons in firefox 60 esr, which I got from debian. I'm done
with mozilla disrespecting my preferences, and have switched to waterfox and
icecat. Remote control of the browser is simply the last straw.

------
kwccoin
Do they follow basic PKI best practice. Do they actually know (not after the
fact) the certification path validation algorithm. It shall be auto.

Is Firefox use the normal PKI authentication mechanism. Their reaction is like
this is a surprise and even signing intermediate cert as the first step and
instead of talking about bypass or hack the whole PKI trust chain.

Based on some of the comments here, I think one has to understand that it is
not just timestamp and validity. The checking of PKI is per transaction and on
a continuous basis. It is NOT just based on signing but also based on CRL
(cert. revocation list) which is also key.

I read the blog a few time. I feel frightened not enlightened. It seems they
are not on the ball. A minor mistake (forget to renewal cert. like O2 (not
sure but heard same issue)) gave a lot of lights on issues.

Do they have CPS even ... :-) or :-(((

------
chrisseaton
After the last PR disaster - the Mr Robot tie-in - one of the ways Mozilla
tried to make it right was that they _promised_ that the survey system would
never again be used for something that was not an experiment.

[https://blog.mozilla.org/firefox/retrospective-looking-
glass...](https://blog.mozilla.org/firefox/retrospective-looking-glass/)

> A SHIELD study must be designed to answer a specific question.

Why have they abused it _again_ here to deploy a hot fix, breaking their
promise and policy that they put in place last time they messed up?

Or am I ignorant or some part of the story or technical details.

~~~
mmanfrin
I think using the system they used to provide a hotfix for a browser-breaking
issue does not clash with the spirit of their prior pledge.

I feel a complaint like this verges unhelpfully in to the pedantic.

~~~
gilrain
As someone who opted out of Studies after the last abuse, I felt betrayed that
my addons were, effectively, held hostage behind enabling both telemetry and
Studies. I decided to wait.

It’s bad optics at the very least. Users who opted in for the update were in
fact entered into studies they explicitly wouldn’t have wanted to be in
without the lure of an earlier update.

~~~
callahad
I'm sorry that you felt like you were held hostage by Telemetry/Studies. With
the exception of the hotfix, we disabled rolling out new Studies during the
incident, and will not be re-enabling them until some time after Monday next
week.

We are also completely deleting _all_ Telemetry and Studies data received in
the week following the incident to ensure we respect people who had concerns
like yours, but enabled Studies in order to receive the hotfix.

Specific details and timestamps are in the post at
[https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-
thi...](https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-things-go-
wrong/)

~~~
username223
"I'm sorry that you felt like you were..." is the worst form of apology,
because it admits no guilt or responsibility. "I'm sorry that you were..." or
"I'm sorry that we..." would be a legitimate apology.

That said, nuking this data is the first good thing Mozilla has done in this
whole fiasco. It's a small but real act of contrition, so kudos for that.

~~~
cyphar
> "I'm sorry that you felt like you were..." is the worst form of apology,
> because it admits no guilt or responsibility. "I'm sorry that you were..."
> or "I'm sorry that we..." would be a legitimate apology.

GP used the word "felt" and was expressing that he felt a certain way about
enabling Studies. You're nit-picking a conversation and it has gone like this:

A: I felt that $x.

B: I'm sorry that you felt that $x.

C: "I'm sorry that you felt ..." is an insincere apology.

Yes, some people use this trick to get out of admitting guilt or
responsibility but this is not an example of that.

~~~
cyborgx7
It is an insincere apology though, because it apologizes for something "you"
are doing and not something "I" am doing. An acceptable way to apologize to "I
felt that..." is "I'm sorry I made you feel like...".

And that's the minimum. Anything less than that is shifting the blame.

------
ziddoap
>First, I want to say that the team here did amazing work: they built and
shipped a fix in less than 12 hours from the initial report. As someone who
sat in the meeting where it happened, I can say that people were working
incredibly hard in a tough situation and that very little time was wasted.

It's a bit disheartening to see the "Lessons" section opened with this. I
understand that everyone worked very hard to get things up and running - but
that is not a lesson learned. That is self-back-patting.

~~~
fwip
It's context for the next paragraph.

> With that said, obviously this isn’t an ideal situation and it shouldn’t
> have happened in the first place. We clearly need to adjust our processes
> both to make this and similar incidents it less likely to happen and to make
> them easier to fix.

~~~
ziddoap
I did gather that much.

In my _personal opinion_, it would have been much more palatable to start the
lessons learned section with

>We’ll be running a formal post-mortem next week and will publish the list of
changes we intend to make, but in the meantime here are my initial thoughts
about what we need to do. First, we should have a much better way of tracking

And shoe-horn the back-patting in earlier or later.

Perhaps I'm being pedantic. Fair enough. But for me, I would rather see this
issue addressed in a way that the severity of the situation requires. Back-
patting is not on the plate for me.

