
Update Regarding Add-Ons in Firefox - akyuu
https://blog.mozilla.org/addons/2019/05/04/update-regarding-add-ons-in-firefox/
======
Tharkun
I have a bunch of privacy-enhancing addons installed, which have now all been
disabled. If I hadn't read HN this morning, I wouldn't even have known why.
Until now, I had no idea that it was even possible to remotely disable my
addons.

And now Mozilla are saying that the "fix" is to allow them to install & run
"studies" on my machine? What are they smoking? I'm having a hard time
trusting a company that randomly & remotely disabled all my addons, regardless
of the cause.

~~~
yborg
I enjoy a nice cup of outrage in the morning just like the next guy, but this
one is really weak and lacks that fresh taste of evil conspiracy that I really
crave.

You use a browser that has remote update capability, which allows them to
install and run new software on your machine all the time. There is a whole
separate section of the Preferences that says "Privacy" in large print that
has a section that clearly identifies the Studies feature and lets you turn it
off. And you use a browser that lets you install privacy-enhancing add-ons in
the first place, and in fact which invented the whole concept of add-ons. When
the browser discovered that it couldn't verify the add-on integrity with a
valid cert, it did what it's supposed to do, it disabled them to protect you
from someone backdooring these add-ons.

Someone at Mozilla fucked up, and they're trying in good faith to fix it. I
don't know what else people are expecting them to do, putting on sackcloth and
ashes won't resolve the problem.

~~~
overgard
Here's the thing, though: yes, we most certainly are giving them a lot of
trust by allowing them to install software on our machines. Which means
outrage when they screw up is _totally_ justified, because they broke that
trust.

Here's a metaphor: Let's say you let someone seemingly trustworthy watch your
kid. (In this metaphor you have a kid). And they let your kid get a broken arm
through gross negligence (let's say they passed out drinking beer), and then
someone said "well, obviously, you should have never trusted that person,
after all, they can do anything with your kid while you're gone, so why are
you outraged?" You probably would still be pretty outraged right? You would
certainly question your decision to trust them, but at the end of the day you
have to trust someone, you'd be a complete shut-in if you could never hire a
baby-sitter.

------
the8472
At the minimum they should add a testsuite that runs at least a month into the
future to catch these kinds of things.

There was a similar issue[0] a few years ago that was only caught a month in
advance.

Even better would be to set things up to only do a verify on install instead
on every startup.

[0]
[https://bugzilla.mozilla.org/show_bug.cgi?id=1267318](https://bugzilla.mozilla.org/show_bug.cgi?id=1267318)

~~~
Mindless2112
> _Even better would be to set things up to only do a verify on install
> instead on every startup._

That would defeat the purpose of verification: "Add-on signing in Firefox
helps protect against browser hijackers and other malware by making it harder
for them to be installed." [1]

And it's not just malware that was doing that. Microsoft force-installed the
".NET Framework Assistant" into Firefox on Windows, and you had to edit the
registry to remove it. [2] If I recall correctly, AVG and Logitech were also
among the list of offenders.

[1] [https://support.mozilla.org/en-US/kb/add-on-signing-in-
firef...](https://support.mozilla.org/en-US/kb/add-on-signing-in-firefox) [2]
[https://support.microsoft.com/en-us/help/963707/how-to-
remov...](https://support.microsoft.com/en-us/help/963707/how-to-remove-the-
net-framework-assistant-for-firefox)

~~~
deogeo
Could you explain why verifying on every startup, instead of just on install,
is necessary? The page you linked doesn't mention it.

Edit: Let me amend my question - why is it necessary for the certificates to
expire? If a plugin is signed by Mozilla, why wouldn't it be trusted once it
gets old?

~~~
tzs
> Let me amend my question - why is it necessary for the certificates to
> expire? If a plugin is signed by Mozilla, why wouldn't it be trusted once it
> gets old?

I asked essentially that question earlier, and received some good answers
explaining why [1].

Briefly, if something is signed by an expired certificate, whether or not you
can trust the signature depends on whether or not the signing took place while
the certificate was not expired.

If all you have is the thing and the signature from the code signer, you can't
tell for sure when it was signed. If a bad guy has obtained an old signing
certificate and its keys, that bad guy can generate new signatures that claim
to have been signed while the certificate was valid.

Some code signing systems, such as the one in Windows (and I think the one
Apple uses) also use another certificate, from a timestamp service, to prevent
this. The way a timestamp service works is you send them a hash of a document,
and they generate a certificate signed by them that essentially says "We were
shown this hash on this particular date/time".

When you include the timestamp certificate with the signature from the code
signer, then when you come across code that was signed by an expired
certificate but purports to have been signed while the certificate was still
valid, you can check the timestamp certificate to see if that is true. If it
is, you can still consider the code signing to be valid.

Forgetting to renew a signing certificate is still bad even if you do this,
but not as bad. If Microsoft or Apple forget, it doesn't stop existing
applications from working, so end users aren't immediately impacted. It does
stop developers from shipping updates or new applications, so still would be a
big deal. I could see a bad guy noticing that a company always updates
expiring certificates one month in advance, say, and then noticing that a
certificate is expiring in just a week, infer that the certificate renewal has
slipped through the cracks and is going to expire, and time the use of a
critical zero day exploit to fall in the window when updates are broken by the
expired certificate.

[1]
[https://news.ycombinator.com/item?id=19824017](https://news.ycombinator.com/item?id=19824017)

~~~
DoctorOetker
I checked the hash functions for the xpi, its broken MD5 and broken SHA...
which doesn't matter for a running instance if FF takes care to download it
from their own servers over HTTPS... but in the attack model you describe it
is retrieving the signatures from an untrusted disk.

[https://news.ycombinator.com/item?id=19830228](https://news.ycombinator.com/item?id=19830228)

------
watermelon0
IMHO it seems problematic, that they can remotely push code changes, including
replacement of trusted certificate, and bypass package managers.

I don't expect software to (significantly?) change during runtime, outside of
what was packaged, signed, distributed and installed as part of
apt/yum/pacman/etc.

I understand (not that I like or agree with) that some apps are just embedded
web browsers, and load everything externally, and that Firefox extensions are
in the end just some JS/CSS/HTML loaded outside of system's package manager.
However, extensions have limited API they can interact with, and you need to
allow permissions for each extension. Having Mozilla owned extension, that can
modify core functionality, seems a bit scary.

~~~
benbristow
If you didn't have browsers auto updating no-one would update them manually,
meaning bad news for web developers wanting to take advantage of newer
features.

~~~
HNthrow22
I understand the appeal of that for developers but it comes at the cost of
users agency and control of their own system, I've been very annoyed with even
simple UI changes in firefox updates as I simply didn't ask or want any such
change. Reading other comments here it's clear I'm a dying breed of old and
stubborn users that prefers full control and agency over my own system. Making
it easier for web developers to implement new features is absolutely not a
tradeoff I'd make willingly at the cost of my systems consistency and
reliability. Also the reason I use firefox is because of all the major
browsers vendors they seem the most aligned with those values although this
seems to be changing more and more every year.

~~~
benbristow
Mozilla/Firefox have the Extended Support Release (ESR) for you.

[https://www.mozilla.org/en-
US/firefox/organizations/](https://www.mozilla.org/en-
US/firefox/organizations/)

------
nullc
I wonder if there is someone out there in the middle of the ocean with a
browser extension based communication and navagation system which is dead in
the water?

It sounds to me that the real headline here is that every copy of firefox out
there was timebombed and we only noticed because someone forgot to elongate
the fuse.

~~~
zimmund
The browser itself continued working fine. Are you aware of any life-depending
extension? Leaving this particular issue aside, your hypothetical "browser
extension for people in the middle of the ocean" was doomed from its inception
if it was designed to run as a browser extension (though it opens the door for
an interesting discussion about similar scenarios that _are_ happenning, like
pilots relying on ipads)

~~~
Dylan16807
> your hypothetical "browser extension for people in the middle of the ocean"
> was doomed from its inception if it was designed to run as a browser
> extension

Why? You haven't backed up that statement at all. Especially before they
killed XUL it was easy to make a non-doomed app that runs as a browser
extension, and it's still plenty possible.

No (non-demo) program should brick itself if it can't connect home.

------
open-source-ux
I hate to say all these things because I use Firefox all the time, but...the
communication around the add-ons issue has been poorly handled by Mozilla. I
only learned of the problem by visiting HN. But what of the thousands of other
users who don't visit HN?

If you visit the Mozilla homepage, there is nothing to acknowledge the problem
(at least at the time of writing this message). Let's try the Support page.
Where is it? Scroll down to the bottom of the lengthy Mozilla homepage to the
page footer to find the link. (How many visitors will make it to the bottom?)

When you click through to the Support page, an easy-to-miss banner in tiny
text appears at the top of the page that mentions the problem - screenshot
here: [https://imgur.com/a/TAHZSWa](https://imgur.com/a/TAHZSWa)

Additionally, when the add-ons are disabled, Firefox misleading says: "These
extensions do not meet current Firefox standards so they have been
deactivated". This is probably a generic message but it's also an example when
a generic message is misleading.

Finally, poorly-named settings like "Normandy" and "studies" that give no hint
of their meaning only adds to the confusion.

~~~
notimetorelax
The way I see it, people might have gotten used to software break from time to
time. Once software breaks it is reasonable to expect it to get fix in a
couple days when it is updated. At least this was probably the experience for
the majority of users, those that noticed the issue.

~~~
orthecreedence
The sad reality is Mozilla has been losing mindshare to Chrome for a long time
and this will rapidly accelerate it. People don't expect things to break. They
expect things to work, and when things break they get angry.

I love Firefox. It's my daily driver. It will continue to be. But this is a
huge fuck-up and they're probably going to pay big in usership because of it.

------
Grollicus
I'm interested in the general writeup what went wrong that they missed this
certificate expiring. That's a structural problem.

Also why it took 6 hrs to assign P1 to the bug

~~~
pcwalton
> Also why it took 6 hrs to assign P1 to the bug

Because people were staying up until the wee hours of the morning working on
fixing it instead of toggling priorities in Bugzilla. This was treated as a
five-alarm fire.

~~~
SlowRobotAhead
I agree with you, it was more important to do the work than to signal.

However, I bet it’s likely they have procedures and policies for work that
first involve signaling like for example the priority level.

I’d be willing to bet lots of things surrounding this issue weren’t handled in
a by the book manner. So if you are always going to wing it, why have a book
(or a public priority level system) at all?

~~~
pcwalton
First, because priority is for things like major feature work, so that
engineers can find the bugs that are useful to work on. In this case, everyone
in the team responsible was already spending 100% of their time addressing the
issue.

Second, because we care about solving problems, not being bureaucrats.

~~~
yawaramin
Really? Because being the bottleneck (i.e. single point of failure)
responsible for approving all addons is exactly what bureaucrats would want to
do ;-)

The non-bureaucratic thing to do, as has been pointed out many times of
course, would be to give users the power to override the cert signing check as
an advanced option.

~~~
pcwalton
It is, in fact, an "advanced option", in about:config.

~~~
yawaramin
But that option only works in nightly builds, not Firefox release builds. If
an option doesn't honour what it claims to, imho it might as well not be
there.

------
overgard
I know Firefox isn't being malicious, but ugh, this seems like the worst
possible PR move for this, optics wise. "Hey so uh, we accidentally broke your
browser, so you need to opt-in to becoming a guinney pig. But don't worry! You
probably were already opted in anyway and just didn't realize it! Also it
might take six hours to work."

~~~
usecontainers
So that's pretty unfair. 1) They state they are working on a fix for normal,
release channel users who don't want to run studies 2) they tell you to
temporarily run studies to get the fix within _up to_ 6 six hours (could be
faster; set expectation) 3) You can explicitly install nightly or 66.4 before
it's pushed if you want a fix now

Yes, it's unfortunate, I'd expect them to meet it head on, push a tested fix
in a timely matter, admit a mistake was made, explain publicly how/why and
apply learning moving forward. Beyond that, what's your expectation?

~~~
overgard
Not saying that their current actions are _wrong_ , just that the optics of it
are terrible for them.

There was a chain of bad decisions that led them here though: 1) thinking it's
ok to disable software after its installed (using cert expiration -- I'm ok if
the cert was _revoked_ but that's a totally different discussion), 2) Taking
more control of people's local software than many people are comfortable with,
especially considering that their main market is tech savvy people that tend
to be more sensitive to this than most 3) Making some of these things opt-out
rather than opt-in, giving the perception that they may value data collection
and control more than their users privacy.

~~~
Vinnl
> their main market is tech savvy people that tend to be more sensitive to
> this than most

Is that so, though? Firefox is still being used by millions of users, and I
doubt those are only the tech savvy internet users.

(Then again, this mostly applies to Firefox users using add-ons, which
probably has a higher share of technical users.)

~~~
yawaramin
One of their biggest 'selling points' is that they protect your privacy. It's
really, really off brand for them to be distributing a critical bugfix through
a _telemetry collection_ channel.

~~~
DoctorOetker
it does feel like the normalization of deviance

it's also entirely predictable that a non-negligable fraction of users -after
enabling studies and verifying everything works again- will ... simply go on
with their lives and forget about disabling studies...

I also don't understand why the certificate graph is not exposed through a
user interface, so that the user can add and remove certificates, or enable
and disable certificates at their own discretion. This should have been
obvious when the certified add-ons were introduced. Then all they would have
to do is host the certificate file on their own domain and everyone could
follow the simple steps in the GUI to replace the expired certificate...

------
tempodox
An article that mentions a timeframe of “the next few hours”, but doesn't have
any timestamp besides a date without a timezone.

~~~
reubenmorais
You can hover the date to see the full timestamp, it's 2019-05-04 07:01:35
UTC-7.

~~~
TeMPOraL
It's an annoying UX antipattern, the only thing worse being not adding the on-
hover text at all. Displaying date and time should be the default.

HN has this problem too.

------
furgooswft13
Granted I'm using Nightly (and previously disabled extension signing in
about:config), but now all my themes are disabled, even the default one
apparently, though that is what it is using. Cannot be re-enabled. When do I
get my dark theme back?

Also...my default search engine is now Amazon.com?? WTF is going on.

EDIT: Also my only search engine. Heck of a job Mozilla.

~~~
lugg
Bezos and his strategy to name the company something starting with A so it
always comes first finally pays off.

~~~
furgooswft13
Now I understand why Google renamed itself Alphabet. One step ahead,
literally.

~~~
sydney6
Have you been able to restore your search providers? Really a kick in the
nuts, this one..

~~~
furgooswft13
Nope. Maybe I should restart Nightly again but...I just added Google back, and
it was not as easy as I'd expect. The Firefox "add search engines" page is a
mess and straight Google search was like 3 pages down.

Also I found another dark theme that is somewhat similar to what I had (but
not official Mozilla) and installed that. All the default ones are still
disabled.

~~~
sydney6
Thanks.. Same here on every machine.. The hotfix update has been installed and
add-on functionality is indeed recovered, but still i only have "amazon.com"
as a pre-installed search provider and have been unable to restore them to
their default settings without creating a completely new profile. Besides, i
would be rather hesitant to install search providers from that list, imho.

------
vesinisa
For me (repository firefox on ArchLinux), the temporary fix was setting
devtools.chrome.enabled = true in about:config, and running this small
JavaScript snippet in Chrome DevTools (Ctrl+Shift+J):

[https://wiki.archlinux.org/index.php/Firefox#Firefox_disable...](https://wiki.archlinux.org/index.php/Firefox#Firefox_disables_all_extensions_due_to_Mozilla_not_renewing_their_certificates)

AFAIK, this will enable all the disabled add-ons until the next check, which
is in 24 hours. This will be hopefully enough time for Mozilla to release a
stable channel update, instead of the "Studies hack".

At least for me, fiddling with the Studies settings had no effect; the
about:studies page remained empty regardless of what I did. I've also seen
multiple reports from people who got the Studies hack working that the fix
actually failed to address the issue properly.

~~~
eikenberry
On Linux you should also have the option of disabling the cert signature
check. Go into about:config and set `xpinstall.signatures.required` to false.
Just remember to set it back to true once this is fixed.

------
akersten
Very curious how the decision to use the Studies program happened. Why not
just roll the version early and include the fix in the new version - isn't
Firefox an evergreen browser now? Maybe there is extra bureaucracy to roll a
new version, or the hotfixers didn't have permission to do it. Either of which
I can understand, being that they made the fix late on a Friday night - so
huge kudos to those who worked hard to get this fix started. Seems like a good
case study for lessons learned here, I'm eagerly anticipating the postmortem
and follow-ups.

~~~
mccr8
Doing a full release takes a lot longer, so they presumably decided to use a
faster method where possible. [https://hacks.mozilla.org/2018/03/shipping-a-
security-update...](https://hacks.mozilla.org/2018/03/shipping-a-security-
update-of-firefox-in-less-than-a-day/)

------
argd678
Certificate expirations show up on causes of outage lists so frequently, yet
little has been done to address the underlying issues of how PKI works to
address it. The core issue of time limits on trust and no specs and
requirements on how to handle the most common case of saying “yes, this is
still trusted” is oddly absent.

Could we have for example a publicly verifiable ledger that can be used to
verify a cert chain with not only a defined workflow to answer if a cert is
still trusted but also a requirement for the workflow to be fully implemented?
Seems quite doable, vs sort of hacks of auto-renew which are hit and miss
depending on the CA.

In other words, when do we fix the sport rather than the players here?

------
jchw
Interesting. Sadly, I imagine many users will have studies disabled since the
Mr. Robot incident. I've re-enabled it but there does not appear to be a way
to force it to check for updates. Guess it will show up in the next 6 hours.

~~~
dralley
You can set app.normandy.first_run to true and then restart Firefox.

~~~
_0ffh
I've seen the same tip elsewhere and tried, but didn't work for me.

What did (seem to) help was setting app.normandy.run_interval_seconds to a
small value (21). At least just a couple of seconds after I did, all my addons
came back.

Edit: plugins -> addons

~~~
Dylan16807
Remember to put it back to normal after!

------
Chardok
The most frustrating part of this for me is there is no (relatively) easy way
to override this behavior. Its fine to disable the addons, but please allow me
to "understand the risks" and continue against Firefox's recommendations.

The feeling of no control over my web browser was why I left Chrome in the
first place.

------
pbhjpbhj
On Android I get this:

>We rolled out a hotfix that re-enables affected add-ons. The fix will be
automatically applied in the background within the next few hours. For more
details, please check out the update at [https://support.mozilla.org/en-
US/kb/add-ons-failing-install...](https://support.mozilla.org/en-US/kb/add-
ons-failing-install-firefox)

Which is like "we did something we shouldn't have causing unauthorised changes
to your computer, so we're going to make unauthorised changes to fix it".

Quite telling is that this is supposed to protect us from other developers. On
the add-on screen "Enable" is greyed out, there's no "Enable even though
Mozilla doesn't like it".

The UX is just like the "fuck you this computer isn't yours it belongs to
Microsoft and we'll do what we like with it" that I thought I'd left in the
past decades ago.

It's not your computer Mozilla, you fucked it up, you don't get to mess around
with it without asking the owner.

My understanding is that this is literally illegal in the UK.

Mozilla barely had any trust left to burn IMO but they sure went all out.

~~~
Vinnl
If Mozilla had barely any trust left than the industry as a whole is truly
fucked.

~~~
hu3
I could say that for reliable software like SQLite or curl.

But for Firefox? My expectations for browsers in general aren't anywhere high
enough to warrant raised eyebrows even in the face of monumental fuckups like
the one were seeing today. Specially because users tried to warn that this
could happen and Mozilla stubbornly said NO:
[https://news.ycombinator.com/item?id=10038999](https://news.ycombinator.com/item?id=10038999)

------
huhtenberg
Can we take a moment and consider the side effects?

This is a once in a lifetime chance for Google & Co. to get a glimpse of all
those sly fuckers hiding behind adblockers.

This effectively uncloaked a very specific subset of Internet users and
exposed them to the very companies that they've been actively trying to avoid.
Not just those who avoid Chrome, but those who take extra steps to explicitly
evade the tracking.

Surely Mozilla, the privacy advocate, must understand the impact of this fuck
up, and yet the offered "fix" doesn't even mention a one-click .xpi install,
but rather asks to enable a mechanism that, if left enabled, will grant
unnecessary control to Mozilla over people installs.

This ain't right.

~~~
Macha
For me at least, it was pleasantly surprising that Google either has no
inventory or just no clue who I am, as when ublock got disabled by this bug,
YouTube started presenting me with ads for cars in mostly Japanese, with
prices in Yen and "Singapore stock also available".

I guess because I watched anime videos?

~~~
RandomBacon
It would be interesting to see what they show for me, but I'm avoiding surfing
websites other than HN until it gets fixed (FF for Android).

~~~
hiisukun
You can disable the signature verification on Firefox for Android, using
about:config and searching for the flag named 'xpinstall.signatures.required'.

Perhaps leave yourself a note to change it back once an update ships : )

------
userbinator
Mozilla decided to make signing mandatory, then screwed up and now they're
trying to fix it by making use of a feature that basically allows them to
remotely execute code on all their users silently?

I checked Mozilla's main site again, and it still has this ironic statement in
its description tag (it's been there for many years):

[https://www.mozilla.org/en-US/firefox/new/](https://www.mozilla.org/en-
US/firefox/new/)

 _Firefox is created by a global non- profit dedicated to putting individuals
in control online._

...I guess it's more dedicated to putting _Mozilla_ in control now. Something
about this whole incident brings up a point that just feels very wrong to me
--- it's not a Google or Microsoft, but the fact that Mozilla also seems to
have this large amount of control over its users is unsettling.

------
gnud
I'm not gonna bother with 'studies' or manual workaround - I'm just going to
wait for an update.

In the meantime I'm enjoying trying out Vivaldi[1] - really reminds me of
opera 3/4, that I loved.

1: [https://vivaldi.com](https://vivaldi.com)

~~~
Causality1
Will Vivaldi let me put the tabs on bottom where they belong?

~~~
jobigoud
What? Tabs belong on the side. In a tree. ;-)

------
gpm
Instead of enabling studies just click on this link. It installs that specific
"study" (hotfix) without installing anything else.

[https://storage.googleapis.com/moz-fx-normandy-prod-
addons/e...](https://storage.googleapis.com/moz-fx-normandy-prod-
addons/extensions/hotfix-update-xpi-
intermediate%40mozilla.com-1.0.2-signed.xpi)

~~~
kosma
How do I uninstall this? It doesn't show up anywhere after installation.

~~~
tdurden
`about:studies` will show active studies and allow you to remove

~~~
ContrQ
I am not sure on others, but it does not show in "about:studies" on mine since
I manually added it.

~~~
gpm
Same.

------
pleasecalllater
This one will be emotional as this destroyed some of my today's work.

F __ __you Mozilla. I lost all my tabs opened in other containers. The
containers don 't work too, so I cannot reopen them.

This bug has been known for 3 years, and you did nothing to fix it. You get so
much money, and what you do is basically provide a pathetic software
(thunderbird) and a nice browser (which you just stopped from working) and you
show me banners asking for more money.

You should be ashamed. 3 years. And no, I'm not going to listen things like
"this is open source, you are free to fix it". I will just go and switch to
another browser. I need a browser which works, not a one which suddenly
decides that my stuff should be broken because all the developers and managers
have been ignoring a critical issue for a couple of years.

:( I know, this will be flagged, and removed. I don't care, I just need to get
all my tabs in containers back. I have never thought that a browser can just
close my tabs because a certificate expired.

~~~
dimator
I lose all tabs occasionally. Browsers aren't perfect, it does happen after a
weird crash, or something. It's exceedingly rare, like maybe twice a year.

With that said, I've always considered tabs to be volatile state. Browsers
make their best effort to e.g. restore the previous session after a crash, but
if you want non-volatile browser state, you should use bookmarks.

~~~
anonymousab
It's not just tabs, but for many users the containers themselves are gone - as
well as their cookies and other associated assumed-to-be-non-ephemeral state.

~~~
jzl
Yep. I just got hit by the bug. Enabled studies, got the fix, but all my
configured containers are gone. NOT happy right now.

------
erikpukinskis
This is why you always have one of your employees keep their clock 30 days in
the future.

Preferably someone who doesn't go to meetings and installd updates.

------
wool_gather
Another workaround if you don't want to enable "studies" is to manually re-
load the add-ons in Debug Mode. I don't know the full consequences of this,
but Firefox seems to be behaving normally having done it.

Go to about:debugging from the address bar. Right at the top is a button to
"Load Temporary Add-on", with a checkbox "Enable add-on debugging". (On a Mac,
the add-ons are in ~/Library/Application
Support/Firefox/Profiles/«ID».default/extensions (assuming that you have only
a single profile).) They should stay enabled until Firefox relaunches.

------
makecheck
From a UX point of view I don’t know why tools don’t have these two features:

1\. “Warning, a critical method for verifying authenticity is set to expire in
X days. Please visit <Y> to update now.”

2\. “A critical verification certificate has expired; while you should
immediately go to <Z> to obtain an update, you may defer authentication for up
to 5 more days.”

...or in other words, why can’t tools cut us some slack on either side of a
deadline? Security for most things is not going to fall apart just by giving
people a little room to deal with issues on their own schedule.

~~~
chronogram
That would be assuming this would ever happen, which clearly Mozilla didn’t
expect, wrongly. It’s a shame, but eh! Mistakes were made. Precautions will
hopefully be made.

------
coffekaesque
How long until heads roll?

There's something really wrong with the organization.

And I thought it was only their marketing/pr that was bad.

> We can't afford to lose Mozilla and Firefox.

[https://news.ycombinator.com/item?id=18800360](https://news.ycombinator.com/item?id=18800360)

~~~
user17843
nowadays they seem to make it a hobby to make negative headlines at least once
every quarter. I fear there will be no negative repercussions for the
leadership.

Basically, the management set their own salaries, the entire work force gets a
40% yearly bonus, and they have no one from the outside to report to.

On top of all of this, the money flows regardless of what anyone is doing.
(While there is a yearly loss of 10% of their users, the past deal with
Verizon made them very rich, so they can go like this for years). Revenue has
been only going up, despite a loss of absolute users. So this explains why
they continue to do bad things even though outside observes can not understand
- during the last 5 years losing users did not impact their financials in any
meaningful way. While people were complaining and users leaving the product,
revenue was increasing.

They do take care of their employees with lots of benefits and other stuff, so
as an employee you don't want to risk all that with speaking up against your
superior.

Over the years they have created a company culture where there are endless
number of small teams doing irrelevant stuff, with absurd hierarchies, with
some people doing no work at all. With 16 people in the upper management,
there's also fragmentetion of decision making going on. It's all a bit
headless.

Due to the complicated hierarchies in the company everyone is content with
doing just enough to not make life harder for anyone else - suggest to change
things fundamentally and actually work on delivering a great product and you
will not get very far.

~~~
jdashg
This is categorically untrue, and unhelpful.

~~~
user17843
some things I wrote I can not prove, that is right. I would love to revise my
negative opinion in light of better evidence.

------
lostmsu
Sadly, this removed my settings for multi-account containers extension :(

~~~
lostmsu
Just heads up here. I was able to restore it partially on Windows using
[https://www.shadowexplorer.com/downloads.html](https://www.shadowexplorer.com/downloads.html)
(which is, btw, a great tool!)

You'd be looking for a file
C:\Users\YOUR_USER_NAME\AppData\Roaming\Mozilla\Firefox\Profiles\YOUR_PROFILE\containers.json

and also ...\YOUR_PROFILE\browser-extension-data\@testpilot-containers

~~~
ruricolist
Thanks, you just saved me a headache.

------
aswan
(disclosure: I am a Mozilla employee but not commenting in any official
capacity)

"Give me control over what code I run on my computer" (meaning "provide a
switch to disable the requirement that extensions be signed") keeps coming up
over and over. And perhaps it hasn't been clearly stated but the problem is
this: if there's a switch that a user can flip, the browser has to record the
state of that switch somewhere (presumably on disk). If such a switch becomes
available, we'll quickly be flooded with malware that flips that switch
without users' consent. At that point, there's no way to tell the difference
between savvy users making an informed choice to enable unsigned extensions
and malware doing it behind their backs. The browser can do various things to
obscure the way that setting is stored, but ultimately any method the browser
uses to read and write the state of that switch is something that other
software can easily mimic.

This is not a theoretical concern, a modern web browser target is an
irresistible target for all sorts of get-rich-quick scammers -- if you don't
experience this day-to-day its due in no small part to the fact that browser
vendors among others are constantly working to keep the bad guys at bay. But
make no mistake: the bad guys are out there and they quickly find and exploit
any opportunities that are available to them.

So as to the problem of how to let users disable signing but ensure that they
have made a conscious decision to do so, there is a stark tradeoff here:
giving the most savvy users that switch necessarily makes other users less
safe. The solution that Firefox has opted for here is to handle this tradeoff
differently on different channels. The release channel (aka the stable
channel, or the thing you get by default when you download Firefox) is
intended for a very wide audience, and so it handles this tradeoff by favoring
safety for all users regardless of their level of technical knowledge. The
developer edition and nightly channels are intended for more technically savvy
users and they handle this tradeoff differently; specifically they do provide
a switch for disabling extension signing.

If there are other (practical and effective) ways to solve this problem of
determining true user intent, I (and I'm sure many many others) would be very
interested in hearing about them. In the mean time, using the mass-market
versus developer-focused channels as a signal for users' preferences on the
risk-configurability continuum seems like a reasonable way to handle this.

~~~
greendestiny_re
My gripes with the switch paradigm are that it:

a) isn't transparent

b) doesn't empower the user

c) isn't easily modifiable

a), b) and c) are the exact opposites of what open source software is meant to
stand for. Firefox is slowly losing its unique position of being an amazing
open source browser in favor of what seems to me a negligible increase in user
security. In my mind, Mozilla is wasting time on micromanaging user risk
instead of actually innovating.

To put it this way, every time I go out biking, I can get hit by a car. It is
a known and well understood risk, one that I have to consider whenever making
a turn. However, riding a bike also provides chances to go faster, meet new
people and so on. Should Firefox aim to reduce my risk of being hit by a car?
No, because I get to choose the level of risk in my life, not Mozilla.

~~~
aswan
You can choose the level of risk. If you want to run unsigned extensions, use
Developer Edition or Nightly.

------
Causality1
So the studies update has hilariously enabled my essential legacy extensions
while leaving my more modern WebExtensions still disabled. Way to go, Mozilla.

------
Metricon
From ghacks (comment section): [https://www.ghacks.net/2019/05/04/your-
firefox-extensions-ar...](https://www.ghacks.net/2019/05/04/your-firefox-
extensions-are-all-disabled-thats-a-bug/)

This should allow the extensions to work until the next check (Verified
locally):

1) Shut down Firefox

2) Open extensions.json (located by about:profile -> Root Directory)

3) Replace all instances of “appDisabled”:false to “appDisabled”:true

4) Replace all instances of “signedState”:-1 to “signedState”:2

5) Save and close extensions.json

6) Start Firefox

7) Close Firefox

8) Open extensions.json

9) Replace all instances of “appDisabled”:true to “appDisabled”:false

10) Start Firefox

11) Disable and re-enable all extensions in about:addons

~~~
svnpenn
it works!

------
pmontra
No Firefox Studies on Android. Maybe they'll release a whole new version.

~~~
AsyncAwait
You can click on [https://storage.googleapis.com/moz-fx-normandy-prod-
addons/e...](https://storage.googleapis.com/moz-fx-normandy-prod-
addons/extensions/hotfix-update-xpi-
intermediate%40mozilla.com-1.0.2-signed.xpi) to get the hotfix on Android.

~~~
classichasclass
I tested this earlier and it works great here.

------
dehrmann
The study pushed, but all my installed plugins are listed as "unsupported" and
says "<plugin> could not be verified for use in Firefox and has been
disabled."

------
mrb
It's incredibly strange that this blog post doesn't even link to the bug
description
[https://bugzilla.mozilla.org/show_bug.cgi?id=1548973](https://bugzilla.mozilla.org/show_bug.cgi?id=1548973)
Imagine someone stumbling upon this post and trying to find more technical
information... Poor communication.

------
wetpaws
I'm trying to be angry at Mozilla for messing up, but cant force myself to do
it. They ended up to be that adorkable kid that spilled the paint all over the
carpet and you just sigh and start cleaning up the mess.

------
gorbachev
This isn't even working reliably. The hotfix study is showing up completed,
not active, on my Firefox at home.

No idea why, there's no information about how to reactivate it. No, re-
installing it didn't help.

------
spevman
Well, I'm about to make a lot of people happy with this info. I was
researching this today as I'm using FF 56.0.2 and found the solution on this
discussion thread. Leave it to an end user to do the job the professionals
either failed or refused to due. It worked for me on 3 different machines. Go
to this link and follow the instructions detailed:

[http://bit.ly/2DUiOLN](http://bit.ly/2DUiOLN)

------
tianheyang
If this hadn't happened, I probably wouldn't have discovered Brave. Noticeably
faster and the ad-blocking is built in. Thanks Mozilla

------
clairity
hmm, i don't seem to have been affected by this bug somehow (my extensions are
all still working). i turn off as much phoning home as i can (including
turning studies off) and block connections to *.services.mozilla.com

any idea why i might not be affected? it may help others who might want to
retain control of their firefox browser (chromium-based browsers being non-
sequiturs).

~~~
Vinnl
I turn on mostly all phone home functionality in Firefox, and also wasn't
affected. Apparently the certificate check is only executed once every 24h, so
I guess our checks only occurred after a fix was already pushed.

~~~
clairity
yes, but i also don't have the fix, since i turned off studies. i'm guessing
the cert check was blocked. what url does the check attempt to connect to?

~~~
gpm
Cert check doesn't need to connect to any url to fail. It's just the cert's
only expired 20 hours ago, so this issue is currently only affecting
approximately 83% of users who happen to be checking at a time of day that has
already passed, over the next 4 hours that will go up to 100% (ignoring users
who receive a fix before it breaks).

~~~
clairity
ah, good point. i didn't think through beforehand how this should have worked.

~~~
clairity
i passed my `app.update.lastUpdateTime.xpi-signature-verification` time and
still had no issues with addons failing, so i dug into it a bit.

turns out i had `xpinstall.signatures.required` set to `false`, which i'd done
to get an older extension working in the past (and subsequently forgot to set
it back to `true` later). so i don't think my install of firefox has been
checking addon signatures for a while now, which is why my addons remained
functional (with other security implications of course).

i didn't immediately find where the cert was stored (to check it's expiration
date) however.

------
philipwhiuk
Patching security bugs using Shield is utterly stupid.

What would happen if they found a Zero Day - would they use the same method?

------
levlaz
If anyone is running firefox-esr on debian (I am), there is a separate
discussion happening here: [https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=928415](https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=928415)

------
NicoJuicy
Install [https://storage.googleapis.com/moz-fx-normandy-prod-
addons/e...](https://storage.googleapis.com/moz-fx-normandy-prod-
addons/extensions/hotfix-update-xpi-
intermediate%40mozilla.com-1.0.2-signed.xpi)

Restart browser

Fixed

------
Insanity
Funny, I only just noticed my extensions were gone because of advertisements
on youtube.

I just assumed I somehow messed up my browser and started looking around the
settings.

A banner displaying why they silently updated FF and disabled the addons would
have been nice as well :)

------
hartator
Isn’t the issue that they’ve forgotten to renew a SSL certificate? Why not
just renew it?

~~~
Xylakant
It’s not an SSL certificate. It’s a certificate that’s used by the browser to
validate the signature of an installed extension. It’s baked into the release.

------
firefox1234
It seems my mistrust in Mozilla is not misplaced. Every time, I install
Firefox, I open about:config and search for all http[s] / [s]ftp links and set
them all to random strings. My installed extensions still work.

------
gshdg
Huh, doing this re-enabled my add-ons but nothing is listed under
about:studies.

------
stubish
Good to see Mozilla thinking out of the box and getting a fix out as quick as
possible. It would take a lot of time to get an emergency build out of
standard channels, because there are so many of them.

------
lewiscollard
> Please note: The fix does not apply to Firefox ESR or Firefox for Android.

Damn it, Firefox, because it supports extensions, i.e. uBlock Origin, is the
only usable browser on Android :(

------
c64ec30650ea3c4
I'm shocked and suprised to find out that mozilla is using EXPIRING
certificates for this. It requires them to continuously take action to prevent
all addons from breaking, which will eventually fail (like it did).

Firefox has a pretty robust update system and everyone is used to frequent
updates. Why don't they instead have a revocation system built into updates?
That way they would have to take action to disable malicious addons, and the
good ones could go on working forever.

Is there something about this idea that is so much worse than what happened
today?

~~~
user17843
You are right. It doesn't make any sense to use certificates for this kind of
stuff.

If an extension turns out to be malicious, you simply deactivate it in the
store, and then proactively deactivate the existing installs. This is how
Chrome is doing it.

But having a certificate does offer Mozilla the feeling of absolute control,
which seems to be of primary importance for them nowadays.

This is probably the reason release and beta users are not even allowed to
deactivate signing in the about:config settings.

------
tiredwired
I got the update and containers are working again but, it seems like it forgot
my old container configuration and I had to set them all up again.

------
pmlnr
about:config

xpinstall.signatures.required => false

Yes, this will void your warranty.

------
martin_a
So, I left Chrome for all the b's they were doing with/to the web. Now Mozilla
is fcking it up, too. Which browser to choose now?

~~~
userbinator
IE6.

No, I'm only half-serious. ;-)

Remember when the Web was mostly about sharing information, browsers didn't
silently auto-update nor break in the process of doing so, organisations
didn't add invasive "telemetry" to everything, and things would mostly stay
working because the pace of change was generally much slower?

Now that the "keep pushing it forward and breaking things" trendchasers seem
to have gotten their way, instead we have the constant churn of web
development, increasingly bloated sites and JS annoyances, browsers becoming
more complex and fragile than OSs, dumbed-down UIs and taking control away
from the user --- yes, that includes Mozilla who got to where they are today
for their "user freedom respecting" position, and the repulsively ignorant
"newer is always better" mentality that's infected even search engines like
Google.

Maybe I'm just being overly nostalgic, but incidents like these really put
things into perspective.

~~~
martin_a
I understand what you mean and I think I feel quite the same.

"The web" has just become so... "strange" in the way everything works and we
take care of it or however you'd like to call it. Often it's just broken with
full intention to do just to push some new shiny technology on us. And I'd
really like if it wasn't that way.

------
rodorgas
I don't like the fact that I can't run an extension that wasn't signed by
Mozilla. This behaviour lacks freedom.

------
zwaps
I am a bit salty that this reset my default search to Google

of course, just an option away. Still :<

~~~
mk89
I am also extremely pissed.

Still, I was searching for alternatives on Android. It's ridiculous that no
other browser allows extensions, not even Chrome itself.

Firefox is way more advanced on this. That's one more reason that is very hard
to say goodbye to them...

------
kreetx
Would have disabling automatic updates have prevented this entire issue?

~~~
mfjordvald
Don't think so, from what I understand, the problem was the intermediate
certificate expired, it would have expired regardless if there were no
automatic updates.

~~~
Karunamon
Even on completely isolated distributions like Tor Browser or enterprise ESR
installs. The only way you avoided this is if you were running Nightly or
Developer or the normal one on Linux, _and_ you disabled signature checks.

~~~
kreetx
Yup. Am on macOS and Nightly and still got hit with the isdue (luckily the fix
was already out).

Guess we'll see a post-mortem soon and get to know how did this even came to
be.

------
chenshuiluke
Am I the only one whose addons weren't affected at all?

------
ncmncm
Can anybody confirm thst Mozilla is scrubbing replies on the linked page that
mention about:config and toggling "xpinstall.signatures.required" ? I find it
suspicious that no replies there mention it.

~~~
zamadatix
I found 5 replies mentioning it, the earliest on page 3. It's more likely it
wasn't mentioned as often as it only works for the minority of the install
base:

"The Nightly and Developer Edition versions of Firefox have a preference to
disable signature enforcement. There are also be special unbranded versions of
Release and Beta that have this preference, so that add-on developers can work
on their add-ons without having to sign every build. To disable signature
checks, you will need to set the xpinstall.signatures.required preference to
"false"."

[https://wiki.mozilla.org/Add-
ons/Extension_Signing](https://wiki.mozilla.org/Add-ons/Extension_Signing)

~~~
eikenberry
ESR release in Debian also has this option available. I think the option is
generally available with Firefox installed from any Linux distro.

------
dingo_bat
I asked this in the other thread but I guess there's too many comments there:
Is there a project for Firefox that is analogous to Chromium for Chrome? I
need a Firefox build with all the Mozilla shit ripped out. I don't trust the
org that decided their certificate expiration was more important than giving
users the choice to run what they want.

~~~
Karunamon
Librefox isn't exactly what you described, but it's close. It's a set of
configs that disable a bunch of telemetry and other unauthorized mothership
connectivity and settings pushing.

[https://github.com/intika/Librefox](https://github.com/intika/Librefox)

However, Librefox is only Firefox with some configuration changes. It is not a
whole new build, and it wouldn't have protected you from this problem since
the problematic addon cert checking is still there.

Note that this would have happened even if the browser never communicated back
home - this problem was triggered via an unwitting time bomb of sorts, not
because Mozilla actively took an action that inadvertently broke something.

------
LifeLiverTransp
Best chrome add there ever was.

------
dvanwag
Mozilla has been on the downward spiral over the last several years. They took
something (i.e. Firefox) that wasn't broken and "fixed" it until it was, first
by killing off XPCOM and then suffering through the misadventures of such
bastard products as Firefox OS. The folks at Mozilla should really stick to
what their good at and focus on an all around open source browser that people
will actually WANT to use.

~~~
asutekku
That doesn’t make money which kinda is required to work on Firefox.

~~~
chrisseaton
I thought they made money by sending searches to Google. A browser that people
want to use means more searches and so more money to keep working on the
browser. Childish stuff like TV show references, and this certificate issue,
means less users, less searches, less money.

~~~
ChrisSD
They've been attempting to diversify their income for ages. A lot of people
have issues with Firefox being substantially reliant on Google.

------
peshooo
Firefox has an interesting backdoor...

~~~
SquareWheel
They also have a frontdoor: built-in automatic updates.

------
RenRav
>Your Firefox extensions are all disabled? That's a bug!

Is this supposed to affect everything installed? I'm running Firefox 56.0.2
and this only affected addons which I had already disabled, all the other
addons are fine... still, am I forced to update to fix this bullshit?

------
felixfbecker
I think Firefox needs to stop this add-on signing and review madness. The web
is OPEN. It's not a walled-garden Apple App Store. Yes, extensions run
arbitrary JavaScript code. So does any webpage you go to, and nobody from
Mozilla reviewed all that JavaScript either. How are extensions any different?
Chrome is doing just fine without all this non-sense process and policy.

~~~
ripdog
Extensions have dramatically more access to powerful APIs to affect the
browser. They can be used to perform a great variety of annoying, intrusive or
downright malicious actions which a website is incapable of. Of course they
have much more stringent policy.

Chrome implements mandatory addon signatures as well, and only Google can sign
them.

~~~
user17843
I doubt google has certificates that run out automatically. Rather, the best
way is with each signing to include signing the date and not allow the
certificate to expire retroactively.

~~~
ripdog
All signing certificates expire. They must. It's a fundamental part of the
security model, because otherwise a malicious actor could take an old,
comprimised cert and inject it into Firefox, allowing them to run malicious
'signed' addons. This attack would work the exact same way in Chrome, so
Chrome will expire it's certificates too.

~~~
user17843
Thanks for the explanation. Can you provide me with a link to learn more about
how google manages extension certs? I am interested in learning how their
system differs from Firefox, if at all.

------
jimmaswell
Mobile's still broken. Which is a browser Mozilla has apparently abandoned the
userbase of in favor of focusing all their mobile effort on some silly
pointless separate browser that caters to millenials more somehow or some
other nonsense, instead of just working on the browser they already have.
Really fed up with Mozilla at this point and considering switching to a fork
or something.

~~~
firethief
Servo was a silly pointless separate browser once

~~~
gpm
I'm not sure I would call it "pointless", they've merged and are merging a lot
of the code they used servo to experiment with into firefox.

Of everything mozilla has done recently, Servo is one of the things I'm most
positive about.

~~~
sachdevap
I'm sure your parent comment was being sarcastic. The quotes were missing.

~~~
gpm
You're probably right, oops.

