
Add-Ons Outage Post-Mortem Result - bzbarsky
https://hacks.mozilla.org/2019/07/add-ons-outage-post-mortem-result/
======
dessant
I believe it is essential for Firefox to allow installing extensions in a
private way that does not force us to upload our source code to Mozilla,
especially considering their renewed push for user privacy.

It could be a feature that can only be enabled from the browser UI, with
appropriate warnings, but the choice must be offered for those who need it.

Until that happens, those with private extensions must resort to compiling
their own browser, use unbranded or developer builds, or patch Firefox at
runtime to disable signature checks.

This is how you allow unsigned extensions in Firefox on Arch Linux, the same
files can be edited on Windows and macOS, restart the browser after changes:

    
    
      sudo tee /usr/lib/firefox/defaults/pref/config-prefs.js &>/dev/null <<EOF
      pref("general.config.obscure_value", 0);
      pref("general.config.filename", "config.js");
      pref("general.config.sandbox_enabled", false);
      EOF
    
      sudo tee /usr/lib/firefox/config.js &>/dev/null <<EOF
      // keep this comment
      try {
        Components.utils
          .import('resource://gre/modules/addons/XPIDatabase.jsm', {})
          .XPIDatabase['SIGNED_TYPES'].clear();
      } catch (ex) {
        Components.utils.reportError(ex.message);
      }
      EOF

~~~
cesarb
> It could be a feature that can only be enabled from the browser UI, with
> appropriate warnings

That's not the issue. The problem is: where do you record the fact that the
user has seen and acknowledged these warnings? Wherever it is, malware can go
there and write "yes, the user has already seen the warnings, no need to ask
them, you can run the extension".

~~~
user17843
I don't understand this logic at all, it baffles me.

Malware can do a lot of things, but I never heard of common software being
designed under the assumption that malware has taken over the PC. The software
I use is designed under the assumption that the user is intelligent enough to
keep the computer malware free. Only the OS or explicit security software is
designed to keep me from malware, but not my text editor, office software or
browser.

It is a terrible excuse for ignoring user intent.

What's next? Disallowing to change the startpage from google.com because
malware can change it? Disallowing downloads because users could download
malware? Going that route would essentially take away the entire software in
the end. Maybe at one point Firefox is only allowed to operate from within the
cloud, where employees make sure it is 100% safe?

Also others have already said that one could add admin privilege to certain
settings.

~~~
Spivak
I mean we're not necessarily talking about malware that has taken over the
computer. More like software companies whose installers 'helpfully' install
their browser extension behind the users back.

The same is true for link handlers, file extension associations, context menu
entries, browser plugins. None of these should be able to be changed by
applications, only by the user themselves.

We're entering an age where applications running in a user session are no
longer user agents but users of your computer unto themselves that need have
their own permissions boundary. Applications running as the user != the user
anymore.

~~~
user17843
> More like software companies whose installers 'helpfully' install their
> browser extension behind the users back.

When there's a will, there is a way...

------
quanticle
The thing I found somewhat baffling is that they didn't renew the certificate
because they were certain that Firefox wasn't checking the certificate
expiration. I'm wondering why they didn't renew the certificate and then
implement a certificate expiration check.

I don't mean to be too harsh, but that part of the post-mortem reads like,
"Oh, we don't need to renew our intermediate signing certificate because
there's a known security vulnerability in Firefox that allows us to get around
it."

~~~
Yoric
Well, it's the kind of errors that happens when two teams forget to
communicate.

~~~
luckylion
But the communication is just the "why they thought it wouldn't be a big
deal". The "why would you let a certificate expire that is actively used in
code signing" is a different issue.

------
groovybits
Looking at the Technical Report [1], I am confused...

First bullet under Root Causes:

"The server-side teams whose systems (e.g., Autograph) generate signatures
knew the certificate was expiring, but did not see a problem because they had
reason to believe that signing certificate dates were not checked by the
client."

So, if I understand correctly, the assumed process was: Root signs
Intermediate --> Intermediate signs Add-on (whether Intermediate is valid or
not) --> FF only validates the End-entity is within date range.

Is this the correct procedure? You're telling me that the intended behavior is
that FF allows add-ons signed by an expired cert? If that is the case, then
what is the point of the signing system in the first place?

Additionally, third paragraph under Recommendations:

"Second, if we are committed to the current PKI and add-on signing approach,
[...]"

'_If_ we are committed' is non-absolute. Is the FF team actually set on this
approach?

1: [https://wiki.mozilla.org/Add-ons/Expired-Certificate-
Technic...](https://wiki.mozilla.org/Add-ons/Expired-Certificate-Technical-
Report)

Edit: clarifications

~~~
sciurus
"FF only validates the End-entity is within date range"

No, this was disabled back in 2016. The technical report briefly had this
worded wrong, sorry if you read it before it was corrected.

"what is the point of the signing system in the first place?"

Like the technical report describes, things like "monitoring add-ons not
hosted on AMO, blocklisting malicious add-ons, providing cryptographic
assurance by chaining add-ons to the Mozilla root."

(Disclosure: I work for Mozilla)

~~~
groovybits
So, the intended behavior is that FF does not check any Add-on related cert
for a valid date range? In that case, what prevents an insider threat of a
Mozilla employee signing malicious add-ons with an out-of-date cert?

~~~
michaelt
When you sign software with certificates that can expire, you run into the
problem that software can stop working with no change on the user's end
whatsoever. And of course, the shorter the expiry on code-signing
certificates, the worse this problem becomes.

The solution for systems like Java and Authenticode is for Certificate
Authorities to also offer a 'Trusted Timestamping' service, which certifies
that the software existed at a time when the code-signing certificate was
valid.

In Java's case, as the CA providing the timestamp is already trusted issue
code-signing certificates with arbitrary dates, this doesn't add any new
trusted parties.

------
mook
From the linked incident report:

> Now that we understand the importance of add-on support for mobile (ratings
> plummeted during the incident), we might want to reconsider that
> strategy/priority for Fenix.

Well that's promising at least. The listed Next Steps is still full of
uncertainty and there isn't a firm decision (presumably the people involved in
this post-mortem are not the relevant decision makers), but at least somebody
is bringing it up.

Given the Mozilla track history, though, I'm not optimistic.

------
sebazzz
I wonder why there is no timestamping used on the binaries.

If I sign a binary with Authenticode on Windows and I use a timestamp server,
Windows still considers the binary signature valid, even if the certificate
expired because it can verify that at the time of signing the certificate was
valid.

If you don't use a timestamp server when signing the binary, after certificate
expiration the binary is considered invalid.

------
gilrain
I'm very disappointed that although using Studies (Normandy) is briefly
mentioned, it is treated as just an unusual option rather than a breach of
policy. I was really hoping for an explanation of why they felt they could
break their promise to users that Studies would only ever be used for, well,
studies.

* All Shield Studies must be designed to answer a specific question - Shield is a tool for evaluating ideas and features in the product. If you are not trying to evaluate a proposed feature or idea there are other means of shipping your code. Shield studies will always respect user privacy in accordance with our data collection policies

* All Shield studies will adhere to the scientific method for answering complex questions - Generate a hypothesis, test, collect and analyze data, validate or refute hypothesis, refine, repeat..

* All Shield studies require a PHD (Product Hypothesis Doc) - A PHD or Product Hypothesis Doc outlines the overarching research question the study is trying to answer. It requires the study author to think critically about the problem and the outcomes long before the study ships.

These guidelines
([https://wiki.mozilla.org/Firefox/Shield/Shield_Studies#Guidi...](https://wiki.mozilla.org/Firefox/Shield/Shield_Studies#Guiding_Principles))
were created after the beach of trust represented by the Mr Robot cross-
promotion as a means to restore confidence. Yet now they have been ignored,
and again "Studies" has been used as a convenient backdoor.

~~~
calcifer
> I was really hoping for an explanation of why they felt they could break
> their promise to users that Studies would only ever be used for, well,
> studies.

It's literally there:

> Because we had a lot of users who were affected, getting them fixed was a
> very high priority, which made Studies the best technical choice.

Which makes perfect sense to me. When you have a worldwide critical
degradation of service, you fix that as soon as humanly possible, ideological
pearl clutching be damned.

~~~
SilasX
"Honoring the promise you depended upon to get users to install the feature in
the first place" isn't ideological, or pearl-clutching.

Imagine your point in other contexts:

"If your company is about to go bankrupt unless it gets some serious revenue,
then sure, you start selling the user data you promised not to, ideological
pearl-clutching be damned."

"If a priest receives the confession of the killer in an unsolved mystery, he
should report it to the police, religious pearl-clutching be damned."

And "critical degradation of service" is overstated; users always had the
option to use another browser (like Mozilla seems to want these days). When
they shut off everyone's unsigned, whitelisted addons in 2016, that wasn't
regarded as a critical degradation.

~~~
Faark
> "If a priest receives the confession of the killer in an unsolved mystery,
> he should report it to the police, religious pearl-clutching be damned."

Confidentiality in such areas (also med/law) is usually restricted to past
actions. If you confess you're going to kill someone, that does indeed have to
be reported to the police. I doubt your local laws are different here.

Thus to construct a better analogy we need ongoing harm that was initiated in
the past but can only be shortened by taking action. Maybe a confession like
_" Just had an argument with my kid and imprisoned it in the basement without
food and water. Will release it after two days"_. Now should this have to be
reported to CPS? I'm not sure, but think I would favor yes.

~~~
SilasX
I was making a moral (and somewhat practical) argument, not a legal one about
a particular jurisdiction.

------
silversconfused
There's no reason for local software to stop working when a remote server is
unavailable, reconfigured, neglected, or your clock is wrong. They are doomed
if they think this is sustainable.

~~~
dralley
That is not what happened. A certificate built into the Firefox installation
expired. It had nothing to do with "remote servers".

~~~
silversconfused
It was a time bomb to disable functionality after a certain date and should
never have been there in the first place. The decision to silently disable the
override in about:config is just more evidence of how sick this it. This is
not software empowering the user. This is software assuming the user is a
moron who may break it.

------
ars
The Post-Mortem result for me is that I have xpinstall.signatures.required
permanently disabled.

I'm stuck running an old version of Firefox because I have too much data in
Tab Groups, and I haven't be able to migrate it yet.

I'll have to find the time someday, since Google Sheets no longer works.

~~~
JoshTriplett
> I'm stuck running an old version of Firefox because I have too much data in
> Tab Groups

Yikes! You're running a years-old version of Firefox without security updates?
And you're browsing public websites?

I think there are some supported add-ons for tab groups that work with current
Firefox. Please, please don't run an insecure browser on the public internet.

~~~
CamJN
There are no working tab group addons post XUL, I've tried them all and they
are all completely broken. While I agree that you shouldn't run without
updates, please don't tell people that there is a working substitute for this
functionality.

~~~
Vinnl
I used to use the XUL version and now use a replacement, and it fully fulfils
the function the XUL version did for me.

------
michaelt

      the Firefox QA plan didn’t incorporate
      testing for certificate expiration (or
      generalized testing of how the browser
      will behave at future dates) and
      therefore the problem wasn’t detected
    

A few years ago I tried to find a VM or something similar that would let me
test software with clock offsets applied. At the time, most of the VMs I
looked at had prioritized the much more common requirement of _not_ having the
guest OS clock become inaccurate.

Does anyone know the state of the art in integration testing software for
future dates?

~~~
donio
One way is to override the libc time functions using LD_PRELOAD using
something like
[https://github.com/wolfcw/libfaketime](https://github.com/wolfcw/libfaketime)

This doesn't work with everything (for example doesn't work with Go because Go
doesn't use libc for such things) but if it does then it's a very simple to
use:

    
    
      % LD_PRELOAD=src/libfaketime.so.1 FAKETIME='1999-12-31 23:59:59' date
      Fri Dec 31 23:59:59 PST 1999
    
      % LD_PRELOAD=src/libfaketime.so.1 FAKETIME='1999-12-31 23:59:59' python  -c 'import time;print(time.ctime())'
      Fri Dec 31 23:59:59 1999
    

or to have the fake clock ticking starting from the specified time:

    
    
      % LD_PRELOAD=src/libfaketime.so.1 FAKETIME='@1999-12-31 23:59:59' python  -c 'import time;print(time.ctime());time.sleep(1);print(time.ctime())'
      Fri Dec 31 23:59:59 1999
      Sat Jan  1 00:00:00 2000

------
mrosett
The HN frontpage is just a long list of postmortems right now.

