
All extensions disabled due to expiration of intermediate signing cert - xvector
https://bugzilla.mozilla.org/show_bug.cgi?id=1548973
======
weavejester
There's a workaround that involves going to about:config and setting
xpinstall.signatures.required to false.

However, if you're running the Stable or Beta version, it will only work under
Linux. On Windows and MacOS you'll need to download Nightly or the Developer
Edition.

To fix this on MacOS I did the following:

1\. Downloaded and installed Firefox Nightly

2\. Ran /Applications/Firefox\ Nightly.app/Contents/MacOS/firefox-bin
--profilemanager

3\. Changed the profile to "default" so my normal Firefox profile would be
used

4\. Started up Firefox Nightly, opened about:config, then set
xpinstall.signatures.required to false

Not sure if it's a good idea to use my default profile in Nightly. It might be
a wiser idea to copy it instead.

~~~
floatingatoll
Upgrading your profile from Release to Nightly, which occurs automatically
when you open it with Nightly, is a one-way irreversible step. This could
prevent your profile from being used with Release without crashes, or lose
profile data such as bookmarks or saved passwords when later used with
Release, depending on what work is underway in Nightly and if it happens to be
backwards-compatible. Be sure to backup your profile if you choose to switch
channels.

Note: I am told that Developer channel uses a separate profile, but there are
instructions below showing people how to override that, at which point this
warning becomes relevant once again.

~~~
andreareina
Oof. Would you happen to know if it's the same with the developer edition as
well?

~~~
pygy_
The developer edition has its own user profile.

~~~
andreareina
And I told the developer edition to use my regular profile because that's the
one that has all my settings and add-ons and I didn't realize the risk was
there. Guess at this point all I can really do is hope and cross the bridge
when I get there.

~~~
obituary_latte
If you’re on Mac, you should be able to recover the old profile with time
machine. Or if you are on windows and have another backup setup.

------
rmbryan
Update: We have rolled out a partial fix for this issue. We generated a new
intermediate certificate with the same name/key but an updated validity window
and pushed it out to users via Normandy (this should be most users). Users who
have Normandy on should see their add-ons start working over the next few
hours. We are continuing to work on packaging up the new certificate for users
who have Normandy disabled.

~~~
neilv
I've been through all of Firefox `about:config` a few times in the past,
fixing preferences to, e.g., try to disable umpteen different services that
leak info or create potential vulnerabilities gratuitously, but this is the
first I recall hearing of Normandy.

Apparently I missed `app.normandy.enabled`, because I think I would've
remembered a name with connotations of a bloody massive surprise attack.

Incidentally, `app.normandy.enabled` defaults to `true` in the `firefox-esr`
Debian Stable package. Which seems wrong for an ESR.

For personal use (not development), I run 3 browsers (for
features/configurations and an extra bit of compartmentalization): Tor Browser
for most things, Firefox ESR with privacy tweaks for the small number of
things that require login, and Chromium without much privacy tweaks for the
rare occasion that a crucial site refuses to work with my TB or FF setup.

Today's crucial cert administration oops, plus learning of yet another very
questionable remote capability/vector, plus the questionable preferences-
changing being enabled even for ESR... is making me even less comfortable with
the Web browser standards "big moat" barrier to entry situation.

I know Mozilla has some very forthright people, but I'd really like to see a
conspicuous and _pervasive_ focus on privacy&security, throughout the
organization, which, at this point, would shake up a lot of things. Then, with
the high ground established unambiguously, I'd like to see actively reversing
some of the past surveillance&brochure tendencies in some standards. And also
see some more creative approaches to what a browser can be, despite a hostile
and exploitive environment. Or maybe Brave turns out to be a better vehicle
for that, but I still want to believe in Mozilla.

~~~
robolange
I too use Debian's Firefox ESR. I noticed the "Allow Firefox to install and
run studies" option in Privacy & Security Preferences a long time ago. It was
unchecked and greyed out (i.e., unclickable), and a label below it says "Data
reporting is disabled for this build configuration", so I gave it no further
thought. This morning I woke up and launched Firefox, noticed this headline,
and then noticed my extensions were still running. I looked in about:config
and lo and behold, app.normandy.enabled=default [true]. I'll be filing a bug
with debian to disable this in the build configuration.

Edit: There are some questions about whether Normandy is really enabled in
Debian Firefox ESR even if the about:config setting defaults to true. I've
filed a bug report, and I'm sure once a Debian maintainer has a chance to look
at it we'll find out the answer.

[https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=928433](https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=928433)

Edit2: It should go without saying, but please do not spam this bug report
with "me too" and its ilk.

~~~
bilbo0s
I had mine disabled. So let's think about this for a second. If I disable a
security hole that you can drive a semi-truck through, I remain foobar'd. If I
run my "secure" firefox configuration, with the security hole enabled, then
they un-foobar me first. Before anyone else. So I could effectively get
rewarded, for always keeping a security hole open. But I didn't keep it open,
so... yeah... they'll get around to me sometime.

?

>:-(

Grrr.

I'm just getting old and curmudgeonly maybe? I've decided though, I'm starting
an animated security blog to show people the ludicrousness of all this kind of
stuff in plain language. I'll be Statler, and I just need someone to be
Waldorf. Because this stuff really is getting Statler and Waldorf level
ridiculous.

~~~
taneq
> So I could effectively get rewarded, for always keeping a security hole
> open.

That's the way it always works, isn't it? Security and convenience are
opposing concerns.

~~~
hedora
Good security designs increase convenience (eg ssh, touch id, single sign on).

The three goals of computer security are integrity, confidentiality and
availability.

All three of those expand the usefulness of the system to the end user.

~~~
afiori
But the point here is not about integrity, confidentiality, or availability.
It is about whether you trust Mozilla, and how much trustworthy they are.

A configuration where Mozilla cannot push remote updates is neither more
secure nor less secure. Mozilla is often under fire for not allowing a privacy
conscious, minimal trust use case.

------
Chirael
Just discovered the same message in the Tor browser, and it seems that
NoScript got disabled. So people running Tor are a lot more vulnerable right
now.

Also, wow, the web has a ton of ads. I've been running uBlock origin so long I
forgot how bad it had gotten :(

~~~
mrep
> Also, wow, the web has a ton of ads. I've been running uBlock origin so long
> I forgot how bad it had gotten :(

Try turning it off. I got rid of ublock after arstechnica complained about a
lot of their users blocking ads years ago and it honestly isn't that bad.
Every once in a while I do back out of a page for maxing out one of my cpu
cores but otherwise, nothing ever bad happens. With ads: either it takes me
half a second to tell I'm not interested in an ad, or I actually am interested
and i follow the ad because I am interested and I want to support the website.

The alternative is websites charging insane amounts of money with paywalls
(Wall street journal has their "best" price for 12 months at $360 a year).
That is horrible because it means only rich people can pay for high quality
news as ads are one of the most progressive forms of payment (rich people ads
are way more valuable than poor peoples and yet everyone gets the same quality
services/news with the ad model despite their income/net worth).

~~~
tripzilch
The alternative is those websites not using third party ads with third party
trackers on it. Adblockers _already_ do not block those (cause they're
indistinguishable from image links). If they really just want my eyeballs they
know how they can get them.

But they really want to _track_ me. And I'm not having that. The moment they
stop tracking their users through third party ad networks, most adblockers
stop blocking (because there's no AI involved and they wouldn't know what to
block except images in general).

It's in their hands, really. If they want to show me ads they can do it in a
normal and decent manner.

News websites should in fact be the first to adapt this model, because it's
_exactly_ the same thing as ads in print media. But they _chose_ to get those
disgusting third party tracking networks involved. And not just one or two.

I don't have to put up with that, but I really don't see why there would be an
action required on my site to stop blocking those tracking ads.

~~~
onli
Just FYI, that's not really true. Adblocker use mostly all the same filter
lists and those do regularly block ads that just are regular images, and even
text notes. [https://www.troyhunt.com/ad-blockers-are-part-of-the-
problem...](https://www.troyhunt.com/ad-blockers-are-part-of-the-problem/) is
an example, even if that specific one got resolved

Adblock Plus has the ability to not block ads that conform to a certain
standard, but in addition to conform to standards ad publishers need to pay
for that. At least that's what they claim.

~~~
galangalalgol
If the images are hosted on the site instead of a third party the list won't
matter.

~~~
mrep
Not true, here is an example of easylist blocking OpenStreetMap advertising
OpenStreetMap events on openstreetmap.org [0].

[0]:
[https://github.com/easylist/easylist/pull/900](https://github.com/easylist/easylist/pull/900)

------
gpm
To re-enable all disabled non-system addons you can do the following. I am not
responsible if this fucks up your install:

Open the browser console by hitting ctrl-shift-j

Copy and paste the following code, hit enter. Until mozilla fixes the problem
you will need to redo this once every 24 hours:

    
    
        // Re-enable *all* extensions
    
        async function set_addons_as_signed() {
            Components.utils.import("resource://gre/modules/addons/XPIDatabase.jsm");
            Components.utils.import("resource://gre/modules/AddonManager.jsm");
            let addons = await XPIDatabase.getAddonList(a => true);
    
            for (let addon of addons) {
                // The add-on might have vanished, we'll catch that on the next startup
                if (!addon._sourceBundle.exists())
                    continue;
    
                if( addon.signedState != AddonManager.SIGNEDSTATE_UNKNOWN )
                    continue;
    
                addon.signedState = AddonManager.SIGNEDSTATE_NOT_REQUIRED;
                AddonManagerPrivate.callAddonListeners("onPropertyChanged",
                                                        addon.wrapper,
                                                        ["signedState"]);
    
                await XPIDatabase.updateAddonDisabledState(addon);
    
            }
            XPIDatabase.saveChanges();
        }
    
        set_addons_as_signed();
    

Edit: Cleanup up code slightly...

~~~
pepa65
I just set xpinstall.signatures.required to True in about:config and that
fixed it after a restart.

~~~
pfeerick
I can confirm this on Ubuntu 19.04 with Firefox 66.0.3. Changing
xpinstall.signatures.required from the default 'True' to 'False' resulted in
addons working again.

~~~
kgwxd
Android too.

------
pritambaral
Run this in your Browser Console[1] to delay signature checking for a day:

    
    
        function set_xpi_sign_time_now() {
            const {Services} =  ChromeUtils.import("resource://gre/modules/Services.jsm");
            const now = (new Date()).getTime() / 1000;
            Services.prefs.setIntPref('app.update.lastUpdateTime.xpi-signature-verification', now);
        }
        
        set_xpi_sign_time_now();
    
    

EDIT: Changed `Components.utils.import` to `ChromeUtils.import` because
apparently Beta and Nightly versions have removed the former, while the latter
was introduced in 60.

This does the equivalent of setting in about:config the time of last signature
verification to the current time. By default, Firefox re-checks signatures in
24 hours (or so I read somewhere here). I like the temporary effect of this,
compared to the permanent disabling of signature verification suggested
elsewhere.

\----

1: [https://developer.mozilla.org/en-
US/docs/Tools/Browser_Conso...](https://developer.mozilla.org/en-
US/docs/Tools/Browser_Console)

~~~
l0b0
Outside of about:addons I get this in Firefox 66.0.3 in Linux:

> ReferenceError: ChromeUtils is not defined

Also, this doesn't seem to help with currently disabled add-ons, unless I'm
missing something. Trying to reinstall Adblock Plus, for example, still
results in

> Download failed. Please check your connection.

~~~
pritambaral
> Outside of about:addons ...

You seem to be trying in a standard Web Console, not the Browser Console.

------
tzs
I'm a bit confused.

I thought that the way signing works in general is that the signer issues a
certificate for the thing being signed (domain, code, whatever) that contains
identifying information for the thing signed (host name for an SSL
certificate, checksum of the code for a code signing certificate), the valid
from and valid to dates of that certificate, and assorted other information,
and either a reference to or a copy of the signer's certificate, and it signs
the whole issued certificate with the signer's certificate.

Someone checking the signed thing is supposed to consider it validly signed
if:

1\. The date is in the valid range for the signed thing's certificate,

2\. A check of the signature of that certificate against the signing
certificate passes,

3\. The signing certificate is recognized as being from an issuer considered
trusted by the checker,

4\. Neither the signed thing's certificate nor the signing certificate have
been revoked, and

5\. The signing took place during the valid date range of the signing
certificate.

Note there is no "the date of the check is in the valid date range of the
signing certificate". A signing certificate expiring should not invalidate
things signed by it. It should just prevent signing anything else with it.

So why is a signing certificate expiring for Firefox breaking already signed
extensions? Shouldn't it just be stopping new versions of extensions from
being signed?

~~~
tedunangst
As noted, in practice, without additional info, there's no way to tell when a
signature was created and so all signatures die when the signing cert expires.

That said, existing signatures that have already been verified, for existing
extensions, should still be trusted.

~~~
bo1024
Yeah, this response is a bit strange because if I trust my own system's clock,
I just have to remember for each signature when I first saw it.

------
mukundmr
They have acknowledged the defect and are working on a fix. While this is a
severe impact, I am still with Firefox. The are enough alternative browsers to
tide over the problem for now. The fact that alternatives exist is the reason
why we should support projects like Firefox.

~~~
notatoad
>I am still with Firefox

that's kinda the problem. there's plenty of reasons to be "with" firefox
still, but you shouldn't need reasons other than it's the best browser. when
it starts requiring loyalty to be a user, that's a big problem.

~~~
ShinTakuya
For me, it is the best browser. Yes, this is a big fuck up, but it's not like
this has caused me material harm. It's easy for me to switch over to Chrome
until this is fixed, and I doubt the same mistake will be repeated in Mozilla.

I expect perfection from plane and car manufacturers, and I pay for that. My
browser, I can live with an occasional hiccup.

------
xvector
Looks like all extensions have been disabled for all Firefox users.

I think this fail-closed behavior is more of a security issue than the one it
is trying to solve. All of my security add-ons - Privacy Badger, NoScript,
Decentraleyes, and many more were disabled. Even worse, it happened without
notice to the user.

One moment I was browsing the internet (just barely) secured by these add-ons,
and the next moment, all of them disappeared (without warning) and I only
noticed when I saw my password manager was missing.

~~~
stevenwliao
If it failed open, anyone unlucky enough to update their extensions could end
up having a malicious version installed. It also would have taken longer to
notice.

~~~
Causality1
So why not just disable extension updates instead of disabling the extensions
themselves?

~~~
ssadler
Presumably because how would it differentiate between a legit "already
installed" extension with a signature that cannot be verified, and an
extension installed by malware that also cannot be verified?

~~~
Causality1
Personally I despise the idea of the software already on my pc being dependent
on signatures stored on a remote server. I installed it and Mozilla can fuck
right off. It's my responsibility to police what software is on my computer,
not theirs.

~~~
cesarb
According to
[https://news.ycombinator.com/item?id=19824520](https://news.ycombinator.com/item?id=19824520)
the signatures are on the extensions themselves, not on a remote server.

------
MrEldritch
As a temporary fix, go to about:debugging, and click "load temporary addon",
then paste in the download link of the missing add-on. Then just try and not
restart Firefox until they fix the broken cert.

~~~
hoekit
This works. On a desktop, you can reload installed addons from the firefox
profile folder >> extensions.

~~~
Animats
Yes, that seems to work on desktop. Thanks.

------
needle0
I’ll still keep using Firefox since I recognize the importance of browser
diversity and the hazards of a Chrome monoculture (that and vertical tabs),
but, yikes.

Still, this type of oversight seems all too common even in large companies. I
remember several cases from Fortune 500 companies in the past few years alone.
What would be a good way to automate checking for them? Has anyone developed a
tool designed specifically to avoid certificate expiry disasters?

~~~
revvx
> Still, this type of oversight seems all too common even in large companies.
> (...) Has anyone developed a tool designed specifically to avoid certificate
> expiry disasters?

LetsEncrypt renewal is supposed to be automated. [1]

I know of a company that hosted blogs for thousands of customers. They used
LetsEncrypt, but the CTO considered automatic renewals a possible security
risk, so they did it manually. Problem is, the expiration happened in a
weekend and they "forgot" to update the certificates before that. Suffice to
say that the next Monday wasn't pleasant. They automated after that.

[1] [https://letsencrypt.org/about/](https://letsencrypt.org/about/)

~~~
n42
Just curious, are you talking about Webflow? Because I had to hunt down and
make sure our Let's Encrypt auto renewal was working until I realized the
certificate was served by them. They wait until the last 12 hours to renew the
certificate. I have no idea what type of rationalization would lead to that
decision.

~~~
tass
90 days is 4 times a year. 60 is 6 times, 50% more expensive when you’re
paying someone to perform the task.

~~~
n42
I had the same thought, but I still find that absurd. Say they host 500,000
websites with HTTPS. 1,000,000 renewals they save spread across the year,
roughly 2 renewals a minute. That is pennies. A t2.medium could handle that
type of load increase

~~~
albru123
A bit OT, but what's up with this usage of Amazon EC2 tiers as a unit of
computational power?

~~~
rickycook
i think it’s a combined “fixed cost” rather than just computational power...
like you could do it with x, thus it should cost at most y

similar to saying that you could do it with a raspberry pi

------
Pxtl
I don't get why an expiring cert disables the extensions. Shouldn't the
browser be checking the cert expiry date against the date the extension was
installed, not against current time? As long as there's no way to manipulate
the extension installation date that would be fine, wouldn't it?

edit: or even why the browser is checking this at run-time. As long as it
checked the cert when the extension was installed, isn't that enough?

~~~
Grollicus
If Mozilla somehow lost control of one of these signing certs (as in: it got
stolen) they would put in on a revocation list. If certificates don't get re-
checked, all installations between "cert got stolen" and "noticed that the
cert got stolen" would keep installed & running.

------
dessant
I have disabled signature checks in Firefox because otherwise it is impossible
to install a private extension without uploading the source code to Mozilla.

This is how you allow unsigned extensions in Firefox on Arch Linux, the same
files can be edited on Windows and macOS, restart the browser after changes:

    
    
      sudo tee /usr/lib/firefox/defaults/pref/config-prefs.js &>/dev/null <<EOF
      pref("general.config.obscure_value", 0);
      pref("general.config.filename", "config.js");
      pref("general.config.sandbox_enabled", false);
      EOF
    
      sudo tee /usr/lib/firefox/config.js &>/dev/null <<EOF
      // keep this comment
      try {
        Components.utils
          .import('resource://gre/modules/addons/XPIDatabase.jsm', {})
          .XPIDatabase['SIGNED_TYPES'].clear();
      } catch (ex) {
        Components.utils.reportError(ex.message);
      }
      EOF
    
    

This method also works for the stable version of Firefox.

~~~
dvfjsdhgfv
It's very good that it's possible, but it's literally the worst solution for
most people.

------
former_mozzer
There have been major organizational problems at Mozilla for a long time that
precipitated this. Many of us saw something like this coming, saw gaps and
unclear responsibilities, reported these gaps and confusions up the chain, and
were reprimanded and financially penalized for asking the tough questions. The
questions were never answered, and we all quit, were fired, or lost motivation
as a result.

This is a tech problem, yes. Cert renewal has bitten everyone in a high
profile way (apple, google, and ms have all had renewal-related outages in
recent years). But this was preventable at Mozilla. Ask a Mozillian about IT
and Cloud Sevices, and what their respective responsibilities are. Ask
Mozilla’s VP of IT- who is responsible for cert renewal? Ask Mozilla
leadership- why are people afraid to ask questions?

~~~
dajonker
You basically just described any sufficiently large organization. Complaining
is not helping anyone in these situations, the only thing you can do to change
things is to go ahead and try to change things. Reporting things up the chain
hardly ever works because the chain is too busy with their own issues and
politics. You have to make it worth their while.

~~~
rat9988
>the only thing you can do to change things is to go ahead and try to change
things

how? Reporting up the chain is THE way to change things. When it doesn't work
what are you supposed to do?

~~~
armada651
Take on the responsibility for the gap yourself. If you get assigned something
from up the chain that gets in the way of that new found responsibility report
up the chain that they need to assign it to someone else first.

~~~
xvector
If I am slowed down in my day-to-day sprint because I decided to take on a
task management wasn't willing to fund/approve, I would be in much deeper
trouble

------
lucb1e
If everyone's add-ons are disabled, I wonder why mine are not. My computer has
been running over night (coincidentally, first time in years) and my add-ons
are intact. Does it take a browser restart? Or might I have a setting that
prevents this from happening? My system time is correct.

Edit: am on Firefox 66, Linux (Debian Buster/testing), using Firefox from
Mozilla directly (not through repositories), and my internet/wifi should not
have disconnected. System has been up since 2019-05-03T17:30:00Z, suspended
before that.

~~~
Svip
I think it only checks at start up, or something? Mine has been running for
days, and all my add-ons are working. As such, I've decided not to restart
Firefox until news arrive that it's working again.

~~~
blahyawnblah
Mines been running for days and everything was disabled.

~~~
lucb1e
Oh, darn. Thought I found a way to avoid this. Thanks for reporting!

Just a thought, might it be that you suspend (sleep mode in Windows) your
system? So not restart the browser, technically, but iirc applications still
receive some event when this happens. At the very least, open connections
would break. Any TLS connection would re-validate the certificate. In that
case, going offline would also trigger it. Have you done any of those
(suspend, be offline / switch networks)?

------
gtdawg
Interesting coincidence, my Ubuntu 12.04 Let's Encrypt certbox-auto updated in
March 2019 to a version which no longer runs on Ubuntu 12.04 without major
surgery.

[https://community.letsencrypt.org/t/pip-error-with-
certbot-a...](https://community.letsencrypt.org/t/pip-error-with-certbot-
auto/88200/4)

I wonder if this bit Mozilla and caused this issue.

~~~
jacquesm
I can beat that. I had a Let's Encrypt cert renewal that upgraded python and
itself and borked the system it was running on.

------
yardstick
In addition to the immediate fix, what needs to happen here and in general
anywhere a certificate is used, is the browser should display an informational
banner that the certificate is due to expire soon. I’d suggest start warning
at T-7 days left.

That way even if the business messed up, they would have a heads up from users
to fix it before d-day when everything stops working. This includes website
certs and addon signing certs and any intermediaries.

------
ww520
Not sure what kind business processes are practiced in Mozilla. Some
organizations have the notation of recurring tasks as part of their business
processes. Recurring tasks are just like bug reports except they are created
and assigned automatically to task owners on a schedule, such as every month,
every quarter, and every year.

The goal of recurring task is to get people's attention to review and perform
tasks the happen periodically. It could be as simple as reviewing it and
marking it done. They will show up as part of the bug report to the assignees,
so they can at one place see all the bugs, feature requests, tasks, and
recurring tasks.

Cert renewing would fall under the recurring task category.

~~~
former_mozzer
Who at Mozilla is responsible for cert renewal? Is it Mozilla IT or is it the
Firefox org? That question has never been answered, and those who asked were
often reprimanded. And this is far from the first certificate renewal problem.

~~~
ww520
Reprimanded for raising the question? That's quite dysfunctional. Sounds like
that's an ownership problem.

------
userbinator
I'm not familiar with Firefox extensions (and have pretty much stayed away
from the stuff ever since they started making it "mandatory"...) but shouldn't
the expiration only mean _new_ signatures won't be valid, yet signatures made
_before_ expiration should remain so? At least that's how I understand things
like Windows' driver signing works (when that was first introduced, I was
quite scared that it would mean perfectly working drivers could just stop
working due to the expiration, and asked... but apparently no one at Mozilla
asked this question.)

Edit: wow, downvotes? Care to explain what I'm missing?

~~~
MrEldritch
This same behavior is how certs _usually_ work. Stuff with expired certs just
_does not run_ after the expiration date; that's because the cert tells you
what server to ask for authentication, and if you have an old cert, there's no
way to be sure that the original issuer is still the one in control of that
domain.

~~~
userbinator
You're referring to things like HTTPS and contacting a _remote server_ , and
thus your reasoning there makes sense.

I'm referring to traditional code signing, which I assume Firefox extensions
are more similar to --- the goal being to ensure that some data has not
changed since it was signed, and only the validity of the certificate at the
time the data was signed is meaningful; even after the certificate expires, a
signature created when it was valid still asserts that the data it signed has
not changed.

------
electrotype
Newbie question: why can't they just renew the certificate, like in 5 minutes?

~~~
rndgermandude
A new certificate can be generated if you have to in a couple of minutes,
sure. Of course, you probably defined some procedures to do it properly and
securely that require more time.

Then the issue becomes: how to get the new certificate to a few hundreds
million users?

If it was a certificate on some server, just replace it there, done. Client
software will just pick it up. But not here. A copy of the certificate is
shipped in every add-on package file. Oops. Now you have to re-sign all add-
ons with the new certificate. And get those resigned files to the users.

Essentially it works like this (which is a slightly modified jar/apk signing
mechanism):

\- An add-on package is a zip file and other than the actual files there is
also a list of known-good hashes of those files in a file called "manifest.mf"
in the META-INF folder

\- Then there is a file "META-INF/mozilla.sf" giving hashes of "manifest.sf"

\- And finally, there is "META-INF/mozilla.rsa", which is a DER-encoded pkcs7
signature and two certificates. The signature verifies "mozilla.sf" was not
tampered with and still is the same as when it was signed by mozilla. Which in
turn verifies the known-good hashes are still proper.

\- The signature is made with a generated certificate, the first one included
in "mozilla.rsa". E.g. "CN=uBlock0@raymondhill.net" in case of uBlock.

\- The "CN=uBlock0@raymondhill.net" certificate was issued by an intermediate
certificate "CN=signingca1.addons.mozilla.org". This
"CN=signingca1.addons.mozilla.org" is the second certificate in mozilla.rsa.
It says "Validity Not After : May 4 00:09:46 2019 GMT". Oops. This is where
the chain breaks now!

\- "CN:signingca1.addons.mozilla.org" was issued by "CN=root-ca-production-
amo". This root certificate is baked straight into the browser and not part of
mozilla.rsa.

Therefore, it is not enough to issue another intermediate certificate (e.g.
"CN=signingca-number-two.addons.mozilla.org"), but you have to actually
generate a new "CN=uBlock0@raymondhill.net" (or whatever) signed by this new
certificate, put those two certificates and a new signature of "mozilla.sf"
based on those new certificates into a new mozilla.rsa FOR EACH add-on and
ship updated add-on files.

PS:

Try it yourself... Extract some addon package (it's a zip file). Then:

    
    
        openssl pkcs7 -in META-INF/mozilla.rsa -inform DER -print

~~~
Hakken
The root certificate is also part of the mozilla.rsa. Addons have it included,
it can be extracted and then imported into the browser Certificates >
Authorities and it will be used to validate addons, including those which are
not updated.

A sample procedure doing exactly this, and a fix for Firefox <= 56.0.2 can be
found here:
[https://www.velvetbug.com/benb/icfix/](https://www.velvetbug.com/benb/icfix/)

The same procedure can be used on newer versions, but the syntax is a bit
different (import cert, go to about:addons, open console):

    
    
      Components.utils.import("resource://gre/modules/addons/XPIDatabase.jsm");
      XPIDatabase.verifySignatures();
    

This only makes sense if you really need to fix it while the browser is
running, otherwise you can simply restart it after the new certificate is
imported.

------
bitbang
I like the alias name assigned to it: armagadd-on-2.0

Temporary work around till the cert gets fixed: set
"xpinstall.signatures.required" to false

~~~
driverdan
That doesn't work unless you're on dev or nightly.

------
vatueil
Active discussion here:
[https://news.ycombinator.com/item?id=19823465](https://news.ycombinator.com/item?id=19823465)

~~~
Deimorz
Even though that's active and has more votes, it's ranked very low (#39 as of
right now, and this post is now #1 on the site).

I think HN penalizes non-link posts (or people are flagging it because they
think it's just someone asking for tech support).

~~~
vatueil
Looks like you're right. I saw the other discussion first and it still has
more comments for now, but this one is ranked higher. Perhaps the threads
could be merged or something.

------
HaoZeke
Honestly, I just have to wonder what the Mozilla devs do. Don't they use
extensions? Didn't they notice what was going on? Why does it increasingly
feel like a more random Google Chrome-Lite paired with Apple's 'best
intentions'?

~~~
minitech
This is a certificate that expired, not some insufficiently-tested change
someone deployed. Mozilla devs had their extensions disabled at the same time
as everyone else.

~~~
teddyh
Well, most of the actual developers probably run Daily and have
xpinstall.signatures.required = false, and are therefore not impacted quite as
severely.

------
MrEldritch
This is a goddamned disaster. I'm just thankful that I use an offline password
manager, but even still ...

I _like_ FF, don't get me wrong, but this is going to absolutely fucking
destroy user trust in Mozilla. This kind of incompetence, on a _browser scale_
, is breathtaking.

~~~
SimeVidas
I dunno. I’m a typical Firefox user, and I’d rather jump off a bridge than
switch to a different browser because of a fuckup like this. People make
mistakes, but Mozilla still stands for things that certain other browser
vendors don’t, last time I checked.

~~~
mevile
That's my thinking too. I went back to Firefox a few months ago and it is back
to being a fantastic browser now, and it feels good to use something that is
also a force for good. I'm hoping they resolve this quickly and that it all
turns out ok.

~~~
drewmol
>and that it all turns out ok.

While I agree with you two assuming the bridge is over water and not too high,
there are real consequences that cannot be reversed. I cannot unsee the ads I
saw in the past few minutes before switching to nightly.

------
koolba
I'd be curious to see if there's a marked increase in net sales during the
time that this snafu is ongoing just because uBlock Origin is disabled and
users are forced to see more ads.

~~~
kgwxd
Maybe, if FF had a bigger market share. On top of that, I think many FF users
know not to browse without it.

------
indygreg2
Patches to workaround the issue were just committed. Those patches were
tracked at
[https://bugzilla.mozilla.org/show_bug.cgi?id=1549010](https://bugzilla.mozilla.org/show_bug.cgi?id=1549010).
At the time of writing this comment, that bug is not chained up to the bug
linked by this post.

------
interfixus
This is highly ungood, of course. The failure to notice an upcoming expiration
is terrible in itself, but I suppose we can sort of almost empathize with how
such shit may happen. The failure _mode_ is inexcusable. Someone somewhere
sometime must once have made a decision, "this will be the right way to fail",
or - worse - developed the addon certification system and deciding it _didn
't_ need no stinkin' graceful degradation route in case of disaster.

Still, it's no worse a calamity than most everyone else presents to the world
from time to time. I shall stick to my Firefox, partly influenced by the
complete absense of any viable alternative out there.

By the way: Writing this a few hours after midnight, May the fourth, local
time. Every single one of my 33 Firefox addons is active and working just
fine. Wating to see. Breaking out my Waterfox, just in case.

[Edit: May fifth -> fourth]

------
SimeVidas
My extensions are still running. I even restarted Firefox a few moments ago.
So it’s not everyone?

~~~
gpm
My extensions are also still running. Comments on the bugzilla bug restricted
so adding details here and figuring that if this is useful someone can forward
it to the right people

    
    
        $ date
        Fri May  3 22:45:22 EDT 2019
    
        $ date --utc
        Sat May  4 02:41:47 UTC 2019
    
        $ firefox --version # Installed from arch repositories
        Mozilla Firefox 66.0.3
    

about:config

    
    
        xpi.signatures.required true
        app.update.lastUpdateTime.xpi-signature-verification 1556920447
        extension.update.enabled true
    

1556920447 is unix timestamp Fri May 3 21:54:07 UTC 2019.

Edit: I think I know why. It checks the signatures daily, and the timing works
out so it hasn't checked since the cert expired for me. Just luck, it will
break within the next 21 hours for everyone. From the source code:

    
    
        const XPI_SIGNATURE_CHECK_PERIOD      = 24 * 60 * 60;
    
        [...]
    
        timerManager.registerTimer("xpi-signature-verification", () => {
          XPIDatabase.verifySignatures();
        }, XPI_SIGNATURE_CHECK_PERIOD);

~~~
gpm
Copying a potential workaround here from my lobste.rs comment, not really
tested obviously

If it hasn’t broken yet for you, I think (but I’m not very much not sure)
setting that preference to 1556940100 should keep it working until 24 hours
from now. And if you keep updating that value every 23 hours to the output of
date '+%s' until it is fixed via a firefox update it should keep working
forever.

I think you need to restart the browser as well after updating the preference
for the above idea to work.

------
fnrslvr
Mozilla doesn't seem to have communicated the issue well. I could imagine a
lot of unsavvy users have tried some wild things in an attempt to fix the
problem, and maybe made a mess in the process. Doesn't Mozilla have a
mechanism for blasting out a message to all Firefox browsers? Also I have a
Firefox account, why haven't I been inboxed about this?

Otherwise I'm not bothered. I won't be switching as long as this gets resolved
within the next few days.

~~~
saulrh
> Doesn't Mozilla have a mechanism for blasting out a message to all Firefox
> browsers?

The cynical side of me says that it must not have this feature because if it
did I'd have seen someone complaining about the browser "phoning home" or
"forcing Mozilla's opinions into my eyeballs".

~~~
Karunamon
...unlike the mothership breaking all addons, and the browser being designed
intentionally in a way that prevents me from working around the breakage?

This isn't just a petty snipe borne out of annoyance. Ads being the malware
vector that they are, and the degree of tracking and data mining out there,
all of those countermeasures being turned off overnight is an exposure that
should be treated with the same degree of seriousness as PII breach at a
company you have an account with.

God forbid you use FoxyProxy or Tor Browser or something else that masks your
connection source - this could have legitimate, real-life consequences if you
don't notice the change.

~~~
saulrh
On the one hand, I understand the point you're trying to make.

On the other hand, I'm going to be honest, I have trouble reading your post
without thinking things like "If you are literally trusting your life to
FoxyProxy, you might want to rethink your entire internet safety strategy."
Another favorite was "Defense in depth."

I've had _dozens_ of different experiences where my extensions silently and
unexpectedly malfunctioned. Configurations getting erased, new extension
releases with breakage or different behaviors, maximum compatible versions in
the manifest, new permissions, botched keystrokes in the extensions page,
profile corruption, incompatibilities between extensions, internal bugs that
cause them to crash-loop without doing anything, the works. Like, I run a ton
of addons, some of which are a bit esoteric and a couple that I compile from
head every few days, but even taking into account my outlier-sized surface
area it's a bit silly.

I'm not going to claim that this isn't a problem. That said, blaming Mozilla
for a life-threatening failure of _FoxyProxy_ , of all things, is like blaming
Cessna because a journalist flew one of their planes into a combat zone. There
just wasn't any way it was going to end well.

~~~
Karunamon
If the nature of this problem didn't also break turnkey distributions like Tor
Browser (this kills Noscript, which means your identity can be leaked), I'd
agree with you.

There's only so much defending you can do against a failure like this (running
Tor Browser is already pretty uncommon) and the blame for it has to be laid
squarely at the feet of Mozilla for the way they chose to centralize their
plugin architecture.

This is one of those low likelihood/high impact events that tend to catch
everyone by surprise.. if you spend all your time as a user thinking about
these failure modes (you don't.. nobody does), you'd be unable to get much
else done. I'd wager the fact that the browser would suddenly gimp itself is
not something the average user (even the average Tor Browser user) thinks
about or plans for.

------
whatshisface
Tomorrow (or whenever this gets fixed), ad companies are going to have some
great data about what the world would look like if adblock didn't exist. I
really home someome does a blog post about it. Yikes, I hope it doesn't play
out like a shark smelling chum.

~~~
kelnos
Given that Firefox's market share has dropped below 10%, I doubt it will make
much of a difference.

~~~
whatshisface
Yes but there's enough sample size for them to multiply the change by ten to
get what would happen if Chrome lost adblock as well.

------
Silhouette
I just ran into this. All my extensions were immediately turned off in the
middle of a browsing session, from UI conveniences to important safeguards,
with no apparent option to turn them back on again.

Warning about something that can't be verified is one thing, but automatically
shutting things off -- particularly things that could affect security and
privacy -- is a Windows 10 level of unacceptable interference.

------
Wowfunhappy
This is why users need to be in control of their own computers. Why can't I
tell my copy of Firefox to ignore the certificate? Why can't I sign my own
extensions?

Mistakes happen, it's okay. But users should be empowered to work around them.

~~~
ehsankia
> Why can't I tell my copy of Firefox to ignore the certificate? Why can't I
> sign my own extensions?

The issue is that if you leave _any_ sort of lever that reduces security, it
_will_ be abused by bad actors. This is why browsers are having ever
decreasing ways to bypass security and have full access. It is annoying, but
at the end of the day, protecting 99.999% of the users trumps what us power
users want.

~~~
userbinator
_protecting 99.999% of the users_

It is horribly paternalistic to advocate for keeping users ignorant,
unlearning, and --- dare I say it --- easily manipulated.

I will refrain from mentioning again that infamous Franklin quote. I am
frankly _very fucking pissed off_ by this authoritarian walled-garden trend,
and vehemently oppose anyone who helps this industry put the nooses around the
necks of others as well as their own.

~~~
macintux
I’ve been in software development and operations for 25 years.

I still don’t want to have to understand everything I ever touch, even if I
could.

~~~
mrob
>I still don’t want to have to understand everything I ever touch

If you don't understand it, don't touch it. The default settings should work
for most users. There can even be a warning against touching without
understanding, like with Firefox's about:config. The offensive thing is
preventing users from touching even if they do understand.

~~~
sfink
The difficulty is in how to keep them available to end users while keeping
them unavailable to malware and bad actors who post "helpful" advice or
publish temporarily useful addons that get updated to malware.

I'm not disagreeing with you, but the right mechanism is not straightforward
to figure out, and you'll always be in a game of cat and mouse. One that sucks
resources from whatever other useful stuff you might be spending your (or
Mozilla's) time on.

------
crehn
On a slightly different note, is there some curated collection of serious
incidents like this somewhere?

Something we could refer to when discussing possible pitfalls?

~~~
cesarb
I believe [https://github.com/danluu/post-
mortems](https://github.com/danluu/post-mortems) is close to what you want.

------
Animats
It's pathetic to see the attitude demonstrated by Mozilla support on this.

 _diox commented 4 hours ago_

 _I 'm locking this like I did in #851 because no new information is being
added. We're aware and we're working on it. This conversation has been locked
as spam and limited to collaborators._[1]

 _Bug 1548973 (armagadd-on-2.0)_ _All extensions disabled due to expiration of
intermediate signing cert_ _NEW Unassigned (Needinfo from 3 people)_

 _Kevin Brosnan [:kbrosnan]_

 _We have confirmed this issue. Extra comments about this being broken will
not advance this bug to being fixed._ [2]

Mozilla just left their entire user base unprotected against ads, trackers,
and some hostile code. Then they insult their users.

Undoing the damage is hard. First, they have to update their signing
certificate. Then they have to re-sign all the add-ons. Then _users have to
reload all the addons._ Then, something users won't do - remove all the
tracking cookies, etc. that slipped in while Firefox was broken.

[1]
[https://github.com/mozilla/addons/issues/978](https://github.com/mozilla/addons/issues/978)

[2]
[https://bugzilla.mozilla.org/show_bug.cgi?id=1548973](https://bugzilla.mozilla.org/show_bug.cgi?id=1548973)

~~~
gpm
> Then users have to reload all the addons

I'm pretty sure Mozilla will implement a fix in a way that users only have to
update their browser, not do anything to all their addons.

~~~
hu3
I'm curious about how an update will be able to differentiate tracking cookies
from legit ones?

------
throw2016
This is a shocking display of not just incompetence and bad practices but of
brazen undisclosed covert control. Why should your local browser depend in
this fragile way on some muckup in Mozilla HQ? And people line up to defend
this?

Where does it explicitly say Mozilla can disable my addons remotely? When did
I give them this power? And this from a so called 'open source privacy
focused' browser. This is a mockery of privacy and open source and they
shouldn't trade on this goodwill to gain users.

There can be no bigger security hole yet security fear mongers preach exactly
this abusive model. This kind of centralized remote power is a far greater
security threat that anything they keep on harping about, 'good intentions'
and 'good faith' are not remotely something anyone should have to depend on.
Why should Mozilla babysit my installation? Shouldn't they be using their
resources to do something productive?

There is something rotten in SV culture and we urgently need to think of
alternatives that are not infused in this 'know it all' abusive surveillance
culture as even after such an egregious abuse of peoples trust and faith all
you will get is hand waving, normalization, apologism and snarky entitled
comments that trivialize people's concerns and choices made on the goodwill of
open source.

------
jaequery
Sure, anyone one can make mistake. We have seen big companies make stupid
mistakes too. But this is Mozilla we are talking about. How this slipped by is
beyond me.

------
swiley
DAMN! That's quite something!

For a piece of open source software you really have very little control with
firefox. It really sucks that the alternatives are worse.

This, likely for almost all of their users, creates more of a security problem
than signature checking actually solves. For me noscript no longer works which
is (IMO) a critically important extension (between mozilla taking away the
disable javascript button and spector.)

~~~
megous
OTOH, it's opensource, and you can re-compile it in like 20-30 minutes with
whatever changes you desire. So you have the control. You can probably even
add some hacked up support for multiple signing certificates (and add yours
there), if you tried.

It's just that it's more work than having what you want implmeneted and
maintained by others.

------
mikeash
Certificates have been in common use online for maybe two decades now, if not
more. This is a common failure mode and it keeps happening. Is there some fix
so we don’t have to keep dealing with spontaneous failures due to expirations?
Or are we doomed to suffer with this until the end of time?

~~~
gator-io
Use a certificate monitor [https://letsmonitor.org](https://letsmonitor.org)
(free)

~~~
yread
Is that site legit? Why don't they even have a HTTPS redirect?

------
M0
They don't use cryptographic timestamps with their signatures ? The
certificate might now be invalid, but the signatures were done at a time when
it was valid...

~~~
fluidcruft
The problem is that "time" is fungible and can be forged. The date on a
signature doesn't really mean anything.

~~~
altfredd
This is a very bizarre justification for an obvious bug. Code-signing does not
work that way anywhere else — neither in Android, nor on iOS, Windows or any
other common platform.

There is a possibility that Mozilla implemented their backwards code-signing
model on purpose — for example, it allows them to oust unwanted extensions
without explicitly recalling their certificates. But personally I think that
they just didn't give the matter enough thought.

------
s9w
What a spectacular failure... it's hours later, and still not working. The
only thing firefox has going for itself stopped working. Already one family
member and one friend switched to chrome.

~~~
59nadir
Just switch for a day. You're making this out to be a much bigger issue than
it has the capacity to be.

~~~
gspetr
NoScript and uBlock Origin have stopped working. This opens up a LOT of attack
surface for malicious hackers.

If this isn't a _critical_ security issue, what is?

~~~
envolt
The Container extension is no longer working. I'm logged out of mostly
everything. There are few sites I don't want to open without container.

~~~
envolt
Update - Container data is lost post addons recovery (I installed Nightly
build). This is ridiculous.

My Firefox usage will be so unproductive for few days. I had around 6-8
containers for different purpose, and somehow I'm habitual to using shortcut
to launch a container and open whatever I'm supposed to (e.g I've access to 3
different AWS account, and I tend to press shortcut key to launch the relevant
container tab)

------
nimbius
another month, another browser vendor that does something inconceivably bone-
headed in "the service of users."

first it was deprecating ALSA for pulseaudio, then it was pocket, then tiles
and their suggestions, then that weird video/voice chat thing, and then
running "studies" as if my use of the browser was some tacit acceptance of my
position as a guinea pig of the internet. Today every extension I use to make
the internet even remotely usable is just...deactivated?

without my consent or knowledge?

enough. im switching to waterfox. Icecat is even worthwhile at this point.
Anything that respects my freedom and intelligence as a user.

~~~
jopsen
This appears to be a mistake.

It's clearly bad that things can break this way. But this happened because a
certificate expired.

Mozilla didn't have to require signatures and certificates for extensions.
They did so because they want to protect users.

Protecting users with signature schemes, increase complexity and, thus, the
risk of debacles like this.

~~~
thekingofh
People take their privacy seriously. Our addons were disabled. I was browsing
for a few minutes until I saw ads and didn't realize what was going on.
Unacceptable that the default to an expiring signature is to disable them
completely.

~~~
jopsen
You would have said the opposite if an expired signature had caused your
browser to be compromised!!!

The default behavior seems desired to me.

The problem was that the certificate was allowed to expire.

A have ton of respect for the fact that it was enforced!

------
Causality1
So that's why I've been struggling for the last hour. It's crazy that it won't
even let me I stall extensions from xpi file, even with extension signature
checking disabled.

------
mirimir
Anyone know when they expect to fix this?

I basically can't (OK, won't) browse anything except HN until they do.

------
kisamoto
So:

    
    
      * Started day with browser, no problems with extensions
      * At some point in the afternoon, all extensions disappeared
      * A couple of hours (cannot be more specific I am afraid), all extensions came back
    

However I'm noticing the saved preferences of some of the extensions has gone.
e.g. Password manager has kept the username however all "Multi-account
containers" are now reset to factory defaults.

Is anybody seeing similar behaviour ?

~~~
tux3
Yep, there's a bug open for that:
[https://bugzilla.mozilla.org/show_bug.cgi?id=1549013](https://bugzilla.mozilla.org/show_bug.cgi?id=1549013)

------
scottcruss
Temporary fix without enabling feedback to Mozilla.

Open the address about:config in the Tor Browser address bar At the top of the
page, search for xpinstall.signatures.required Set the
xpinstall.signatures.requiredentry to false by double clicking it

Note: This workaround should only be used temporarily, as it disables a
security feature. Please remember to set the
xpinstall.signatures.requiredentry back to true again once the Tor Browser
security update is applied.

------
Adamantcheese
I'm surprised nobody's mentioned addon debugging if you really need an
extension working. All of the extension packages are in <your
profile>/extensions folder and you can load them for the duration of your
session by turning addon debugging on in about:debugging and loading them in.
This should be fixed fast enough that this fix will be good enough, just don't
close your browser.

------
bArray
I'm running the workaround suggested by Reddit use @MeaslyTwerp (the one after
the HN solution):
[https://www.reddit.com/r/firefox/comments/bkcjoa/all_of_my_a...](https://www.reddit.com/r/firefox/comments/bkcjoa/all_of_my_addons_got_disabled_and_they_are_all/)

------
BuckRogers
I've been using Firefox since 2002, always loyal because it works as I expect
a browser to work. For users like me who see the benefits of Firefox being the
only browser worth losing the benefits of native browsers (Safari/Edge), the
only real place we'd move to due to sufficient issues is to those native
browsers.

The new Chromium-based Edge[0] for someone like me, who would never use
Chrome, is the only cross-platform alternative that could pull me away from
Firefox. Native power usage advantages on Windows, with portability to other
platforms and Chromium's speed advantages makes it a very attractive choice. I
hate to see this happen to Firefox, but they don't have a lot of room for
error. I've been using "ChrEdge" at work, and it's already a great browser as
it is.

[0][https://www.microsoftedgeinsider.com/en-
us/download/](https://www.microsoftedgeinsider.com/en-us/download/)

------
rb666
Wow this is really bad, breaking millions of people's workflow in one go.

Fixed for now by switching to Firefox Nightly and disabling signing.

------
gxx
My password manager and every other plugin disappeared this morning first on
one of my Macs and a few hours later on the other. I could not log into
anything so I switched immediately to Chrome because I don't have time to fuss
with workarounds. I'll return to Firefox when I hear the problem is definitely
fixed.

------
overgard
Well, a little bit of empathy for mozilla here: I've seen a lot of IT
departments that don't have any sort of great system in place for managing
certs. A lot of places I worked, I always had a suspicion they were a ticking
time bomb. It's not enough work that it's really anyone's full time job to
manage them. Also, at larger companies, you might have divisions that do it in
different ways without cohesion. And then a lot of times certificate
expiration is so far in the future that the people that initially setup a
certificate might have left the company and forgot to document it, etc. So
that kind of thing can easily fall through the cracks.

Maybe a constructive thing I'm curious about: What is considered best practice
for managing certs? How do people do this in a secure way that makes sure they
get renewed in a timely way?

------
donpdonp
"One or more installed addons cannot be verified and have been disabled" just
showed up at the top of all my firefox tabs. #sadtrombone

[https://i.imgur.com/hUe7Nyn.png](https://i.imgur.com/hUe7Nyn.png)

------
Ghost_of_USSR
A positive side effect: my 2 years old FF installation stopped freezing
constantly.

------
phyzome
Still working for me in my main profile in Firefox 60.6.1esr. I could even
install an update to HTTPS Everywhere!

When I opened my alt profile, extensions still worked, but I can't install
updates—including for HTTPS Everywhere.

What's up with that?

------
ggm
I'm Interested in a proper root cause analysis and mitigation report.

------
anfilt
Still beats using chrome.

~~~
mav3rick
Have fun defending this to your friends who switched overnight to Chrome
again.

------
JoshuaRLi
I've had normandy disabled in my user.js for a while, but this was only after
thoroughly perusing documentation and some firefox "hardening" projects. Point
being, no end user should have to do what I did. Sane defaults, and
transparency about things that should be opt-in, are sorely needed.
Regardless, I still stand by Firefox, and am thankful there are chromium
alternatives. The web is what it is, though I dream of a simpler one.

------
nkkollaw
I wonder how it's possible that these kinds of--I assume--easily preventable
problems happen at huge corporations and projects.

I routinely renew certs for clients and it never happened that a website was
down because of an expired cert.

I would think that Apple, Microsoft, Mozilla would be more efficient than me
in avoiding these kinds of fuckups?

I also bet these companies make huge investments to their infrastructure so
that their services have a near-100% uptime, and then they let certs expire.

------
akerro
All my setup of 24 containers has been reset... Not only it forgot which
domains were assigned to which containers, it also removed my manually created
containers.

------
drdon75220
What a nightmare! ---- ALL of my extensions have disappeared and have been
disabled this weekend (with ZERO warning) ---- Trying to use my Roboform
password program is a nightmare---- If the problem is not 100% solved by
Monday (May 6), I plan to exterminate every trace of Firefox of every computer
I own ---- Shame on Firefox for placing uses in such a nightmare situation
with ZERO warning and ZERO solutions ----

------
a3n
Advertisements. My eyes, my eyes!

------
nydel
i’ve a strong emotional reaction to this. it feels like a failure on all of
us. will be fighting toward a more optimistic interpretation of such a snafu.

------
hnaccy
I never liked change that extensions had to be signed by mozilla.

Why do my personal extensions need to be hooked into some third party service
that can go out at anytime?

~~~
lvh
The developer edition allows that just fine.

~~~
superkuh
Developer edition is effectively aurora/alpha. It is buggy compared to
release. And yes, I do mean that. If it were not it'd be 'release'.

Asking people to either give up control of their software (ie, walled garden
release versions) or use buggy and insecure software Dev/Nightly/etc is not
acceptable.

It's why I switched to a freedom respecting Firefox fork as soon as they
announced walled garden extension signing in Firefox 37.

~~~
lvh
Sure: debranded versions ("freedom respecting") will also do that.

You say walled garden, I see what random WebExtensions people install on their
work laptops and think "yeah maybe someone policing this thing isn't the worst
thing". But most importantly: it sounds like it's not actually a problem for
you?

------
pcvarmint
This was probably an unintended consequence of trying to ban Dissenter. [0]
[1]

0\. [https://dissenter.com/download](https://dissenter.com/download)

1\.
[https://youtube.com/watch?v=f0Cc8RpqH1g](https://youtube.com/watch?v=f0Cc8RpqH1g)

------
bluejay2387
Not happy that Firefox didn't ask before it did this. I am getting really
tired of my technology trying to tell me how to do things.

------
333c
Can someone explain exactly what went wrong? I don't think I quite understand,
but 7/9 of my extensions have been disabled.

~~~
lvh
Firefox requires extensions installed via their "store" be signed with a
certificate to make sure they're actually from there. That certificate has an
expiry date. It expired, so now all of its signatures are invalid -- and
Firefox no longer trusts the associated extensions.

~~~
sfink
A minor added detail is that it wasn't the leaf certificate that expired. I've
heard that that would have been handled properly. It was an intermediate cert,
and I guess that possibly wasn't fully taken into account? (This is 3rd hand
knowledge and speculation, note.)

~~~
lvh
That is accurate :)

------
spicytunacone
Amusingly, my surviving extensions are for WAVE, Print Preview and other
accessibility tools. Thank you Mozilla, you really do care!

------
new_guy
This is absolutely going to tank Firefoxs market-share and reputation. I have
a site full of tech-illiterate users and they're all uninstalling Firefox and
searching for alternatives. None of them are interested in the workarounds
presented, most of them even struggle to turn their computers on! Mozilla
really dropped the ball here.

------
m3kw9
That’s a bad sign on what they do with security

------
measlytwerp
Here is a temporary workaround:
[https://www.reddit.com/r/firefox/comments/bkcjoa/all_of_my_a...](https://www.reddit.com/r/firefox/comments/bkcjoa/all_of_my_addons_got_disabled_and_they_are_all/emggvbx/)

------
Metropolix
I was able to temporarily work around this problem by going to
"about:debugging" and using the "Load Temporary Add-On" button to load up the
disabled add-ons inside of ~/.mozilla/firefox/<my_profile>/extensions/

------
rachelbythebay
5 years since Heartbleed, yeah? Was this cert one that was pulled into sync
due to being reissued?

------
masterfooo
After learning about Normady thing, I am literally now blocking every Mozilla,
Firefox and their affiliate domains and their Ip adresses. The trust is lost.
I rely on Debian repo and like other's mentioned, Normandy seems to bypass
Debian oversight.

------
campbellmorgan
For anybody using Firefox Android and ublock origin who now sees ads, worth
checking out the duckduckgo Android privacy browser. So far, it seems to be
working beautifully so something good for me at least has come out of this
mess!

------
terrycody
adblocker seemed stop working, no idea why, anyone know how to fix them or
just wait?!

~~~
tapanjk
This helped me discover Firefox's Content Blocking setting, which is set to
Standard by default, but now I set it to Strict. Works better than an ad
block!

Preferences > Privacy and Security > Strict

------
Causality1
It happened hours ago. Why in the blue blazes of hell isn't this fixed yet?

------
happypuppy
Waterfox works in Windows, no issues with extensions and supposed to have a
greater compatibility with legacy extensions.

Fennec works in Android, no issues with extensions and has the third party
trackers removed.

------
F-B-Aye
This problem has screwed up Firefox 53 as well so unfortunately reverting to a
superior version won't fix the problem either. At the moment Waterfox is
looking like the best fix.

------
Grue3
What kind of idiot thought that the add-ons I have _personally_ installed on
my browser need to have a capability to be remotely disabled despite literally
nothing being changed.

This is absolutely inexcusable. I want to see everyone being responsible for
this "verified add-ons" fiasco fired from the team (after they roll it back of
course).

~~~
swalladge
Exactly. If me/firefox has verified the signature (or approved the download)
when downloading or updating the addon, that should be all that's necessary.
Why does firefox have to check signatures constantly?

~~~
egwor
in case it was revoked? Seems fairly reasonable approach

~~~
TeMPOraL
Disabling with no option of user override is not reasonable. It only creates
unnecessary dependency on third parties.

When product recalls happen, the manufacturer isn't taking your product away
by force.

------
kgwxd
As I'm reading these comments, the Unverified addons warning popped up :)
Well, I refuse to use the internet without uBlock Origin. Looks like this is
going to be a productive day.

------
verisimilitudes
This relates to my opinions about encrypted HTTP, which is that it shouldn't
be mandatory.

If you have a well-designed system that only works with encryption, then sure,
but this idea of using the same mistaken systems as the WWW clearly doesn't
work well.

I've never seen a Tor Hidden Service fail because of something expiring.

Much of this nonsense about encrypting everything, without reason and excuse,
is to protect advertisements from being modified.

That this hit Tor Browser and disabled NoScript is damning, but I already
disable JavaScript in about:config and I'm not even using a version of Firefox
this new, anyway.

I can't tell if my opinion of Mozilla is lower or if it can't get lower.

~~~
sfink
Requiring https is a different situation and much more defensible in my mind.
It's way too easy to rewrite the web pages of everyone using library or
coffeeshop wifi and thereby hack/phish a lot of people's browsers.

------
homero
Does this happen when you open firefox? Forgetting what i need to reenable and
what i had disabled is going to make me mad. If i just don't open ff will i be
fine?

------
chaz6
Maybe everybody should set app.normandy.user_id to "anonymous". I am
disappointed that Mozilla still has unique tracking ids embedded in Firefox.

------
joeyh
Firefox extensions installed using apt are not affected by this problem
AFAICS, so to get a working ad blocker, apt install xul-ext-ublock-origin

------
sushikokk
Shit, I just switched to lastpass and deleted all my saved passwords in
firefox. Good timing. On windows, so the config hack is not working.

------
jopsen
Wow, that's a bad slip up... I hope users stick with Firefox.

I feel like this sort of thing is a risk when being vigilant about security.

------
2T1Qka0rEiPr
> Steps to reproduce: Wait until it's past midnight on 2019-05-04 UTC.

This has got to be one of my favourite bug reports ever.

------
v8engine
So let's say I'm the IT department in my company. I've already got my root
cert on every employee's PC(including Firefox because they can't browse
otherwise). Can I act like the Normandy endpoint and let's say remotely
disable the ability to install any extension including those pesky VPN ones
and also do a lot of other such things I would like, you get my drift? Am I
right? Am I right?

Please tell me I'm wrong.

~~~
fjsolwmv
You are wondering if an IT admin can admin machines in its network? Yes, an IT
admin can admin machines in it's network.

------
GRBurst
And now, all my plug-in / extension settings / preferences are gone after
updating...

------
jtl999
Running Firefox 66 from Ubuntu repositories on Ubuntu 18.04 and all my
extensions are enabled.

Does it only occur after a restart or?

~~~
phyzome
I believe it happens the next time Firefox goes to check for addon updates.
You may want to proactively set xpinstall.signatures.required=false which... I
think might work for 66 on Linux? It worked for 60.

~~~
jtl999
Ah, checking `app.update.lastUpdateTime.xpi-signature-verification` shows that
it hasn't checked since "yesterday"

:/

    
    
      jtl@laptop-linux:~$ TZ=UTC date --date="@1556919381"
      Fri May  3 21:36:21 UTC 2019

------
hexo
This almost turned me into raging monster. How dare you mozilla disabling
adblock? now i want an UNSEE function :D

------
gminialpha
I déinstalled most of my addons and/or extensions How can I return to my
previous profile to get them back?

------
en
The saddest thing today is to realize that I can't use Firefox without
extensions on privacy/security.

------
hex12648430
I'm curious to see what kind of impact on ad revenue we'll see in the next
couple of days.

------
user17843
Is it possible that this wasn't an expired certificate but someone
accidentally changed the signing process?

The disabling happened right after the announcement by Mozilla to implement a
new policy towards extensions.

Maybe someone didn't realize their mistake, so now everyone thinks it was an
old certificate.

------
dvdgsng
Is Tor Browser also affected?

~~~
Chirael
Looks like it. I had the same yellow "One or more installed add-ons cannot be
verified and have been disabled" banner in the Tor browser as in Firefox, the
NoScript extension icon is missing, and I went to two different "do I have
Javascript enabled" sites and they both said JS is enabled.

------
XorNot
So this just happened in my browser...when exactly will this be fixed?

------
dazmax
Ah, so I need not to restart Firefox while they're still enabled.

------
rhizome
I tell you what: Grammarly is making a mint off this right now.

------
StreamBright
I guess there was no monitoring on cert expiry.

------
tempodox
Mozilla continues the war against its users.

------
geniium
This is huge. This had a huge impact on my work in the last hours. Still not
fixed for me. What's going on???

------
abdulmuhaimin
its been so long since i browsed the web without adblock. Now I remember why

------
dwdz
Because of this my Firefox crashed and I lost all my open tabs. Just fking
great...

------
former_mozzer
Will Mozilla IT finally be held accountable? Whether or not the direct cause
of the expired cert is IT failure, the fact is that a healthy IT would have
prevented this. Take a look at Mozilla IT leadership. Take a look at all the
people who have left in the last few months. For a year people tried to bring
attention to the IT leadership disaster, and every person who did that was
penalized for it and left Mozilla.

A disaster like this has been brewing for a long time, and Moz leaders didn’t
listen to the canaries in the coalmine.

It’s a very sad day for all the great people at Mozilla who work so hard only
to see Mozilla IT let them down.

------
gordiancaesar
when might the fix be?

------
gordiancaesar
when is the fix?

------
carmate383
Well, it was fun whilst it lasted Firefox. Sorry, but this is a deal-breaker
for me.

Hello Chromium again...

------
brainfeed0
If they can't get this right, what hope is there of browser security?

~~~
lvh
That's silly. Browser security is not one function. It's not even one team.

------
thekingofh
You've fucked up mozilla. I had no option other than to switch over to a
browser that would let me install my ad blocker. The web sucks too bad for
this. My addons are mine, not yours to disable and enable at will.

------
driverdan
First they force code signing on everyone without a way to disable it then
they break it. This is an extreme level of incompetence I didn't expect from
Mozilla.

They'd better have the best post mortum ever, possibly with someone being
fired.

~~~
cheeze
Oh relax. A cert expired. An intermediate cert at that...

This has probably happened to every major cloud provider and countless
companies at least once. Certs are _hard_.

Should Mozilla have had monitoring on their cert expiration? Yes. Will they
after this? Probably. Is any one person ever at fault for something like this?
No.

Firefox is an open source project. You're welcome to contribute and make
things better.

~~~
yjftsjthsd-h
> Oh relax. A cert expired. An intermediate cert at that...

Everyone's extensions broke. Including security ones. Including the ones
bundled into the TOR browser. And end-users can't fix it. Because Mozilla
decided that it was too dangerous to let users choose what extensions to run
for themselves. This is an excellent moment to be upset.

~~~
__david__
Being upset is ok! I'm not particularly happy that I can't just override the
certificate check on stable. But demanding someone get fired is just
pointlessly punitive.

------
mirimir
So why is this taking so long to fix? From
[https://github.com/mozilla/addons/issues/978](https://github.com/mozilla/addons/issues/978)

> diox commented 2 hours ago

> I'm locking this like I did in #851 because no new information is being
> added. We're aware and we're working on it.

I mean, two hours? WTF.

~~~
hu3
It was Friday night on most parts of the Western hemisphere. I'm guessing it
took some time to get the right people back to work and assess the situation.

~~~
mirimir
Yes. I was upset. Sorry. I ought to know to stop when I'm upset.

~~~
hu3
Nah don't worry. We're all upset and a bit disappointed.

------
cnst
Is it perhaps a good time to remind folks that the same thing could happen to
all your "secure" HTTPS websites that are completely unavailable via HTTP,
where the only thing served over HTTP are the 301 Moved redirects, even for
sites that don't collect any user information at all, and only serve static
and public content, which really hardly benefit from the mandatory encryption?

Or is HTTPS / LetsEncrypt too big to fail? HTTPS still always a good choice? I
see…

~~~
swiley
But it's easy to override broken https certificates. Worst case you have trust
on first contact style security.

This is just plain bad.

