
Firefox/Normandy/PreferenceRollout - mlthoughts2018
https://wiki.mozilla.org/Firefox/Normandy/PreferenceRollout
======
whym
Relevant context:

[https://bugzilla.mozilla.org/show_bug.cgi?id=1548973#c57](https://bugzilla.mozilla.org/show_bug.cgi?id=1548973#c57)

> Update: We have rolled out a partial fix for this issue. We generated a new
> intermediate certificate with the same name/key but an updated validity
> window and pushed it out to users via Normandy (this should be most users).
> Users who have Normandy on should see their add-ons start working over the
> next few hours. We are continuing to work on packaging up the new
> certificate for users who have Normandy disabled.

Further context:
[https://news.ycombinator.com/item?id=19823701](https://news.ycombinator.com/item?id=19823701)

~~~
user17843
This partial fix via Normandy means a relevant part of the user base may still
be in the dark: tech-savvy power users with many extensions who have chosen to
disable normandy. (This is probably also the group which is most affected by
reset extension settings)

~~~
tux3
Indeed, I had opted out of Normandy, SHIELD studies & co after the Looking
Glass/Mr. Robot promotional ad fiasco.

Tentatively re-enabled, it's now 2 hours since the announcement but I haven't
received the preference flip study yet. A couple more hours before my fox's
lastUpdateTime rolls around and my extensions get disabled, patiently waiting
to see if I'll have to take manual action!

Edit: Funny, my app.normandy.run_interval_seconds was set to 24h for some
reason, the new default seems to be 6h. I wonder how many people are also in
that situation.

~~~
bwat49
I found that setting app.normandy.first_run to true and then restarting
firefox triggers the hotfix study to be installed (and the pref then
automatically sets itself back to false)

------
RpFLCL
I understand that this isn't a backdoor, it's a frontdoor insofar as there is
a wiki page about it. What concerns me is that Normandy isn't communicated in
the UI of the browser settings, and that Firefox is allowing Mozilla to target
me based on any of these:

    
    
      Targeting can be based on many criteria, including:
      Firefox version
      channel (release, beta, nightly)
      a percentage of users
      country
      Firefox locale
      installed add-ons
      profile age
      any preference value
      many keys in Telemetry
    

_Any_ preference value? Is this targeting being done locally or remotely? Once
I've been 'targeted' what information about me is then sent to Mozilla?

I'm not sure I want any of those shared, especially the last several. Yet this
was enabled by default and the only way to disable Normandy is through
about:config? Does disabling that also stop whatever allowed the targeting in
the first place?

It just seems like there was a real lack of informed consent regarding this
feature and it only came to light when the team used this as a shortcut for
fixing the add-on disaster.

~~~
nyuszika7h
Unticking "Allow Firefox to run and install studies" under Settings > Privacy
& Security should disable Normandy.

~~~
HNthrow22
I have all of the privacy options unchecked including studies but Normandy
(which I've just found out about via this issue) is still enabled.

------
hexo
I don't get it. Is someone going (and I don't really care if is from mozilla
or microsoft) to RECONFIGURE my browser without my consent? This is
unacceptably bad idea.

~~~
Althorion
Yes, sort of. The change is made to _default_ configs, so if you changed
something, it won’t touch it. And while I get that changes to that can be
annoying, I also find them necessary to keep the application easy to use and
productive and cannot think of any piece of software that never have changed
their defaults.

The reason for Normandy to exist is to allow the developers to check if some
change to that defaults is production ready yet. For example, you can start
enabling by default hardware video acceleration for some people and compare
the number of browser crashes they experience compared to the general public
and use that knowledge to know when this feature is stable enough to be
enabled for all.

~~~
lifthrasiir
It should be mentioned that Mozilla had once wrecked their reputation by
allowing a corporate-supported study [1] (yeah, Mr. Robot one). It was a
really bad PR disaster for Mozilla, and I think Mozilla has at least learned
their lesson by not abusing the studies system in such way (as far as I know).

[1] [https://drewdevault.com/2017/12/16/Firefox-is-on-a-
slippery-...](https://drewdevault.com/2017/12/16/Firefox-is-on-a-slippery-
slope.html)

~~~
philipwhiuk
So they'll put it directly in Normandy instead of wrecking Studies?

~~~
lifthrasiir
Not sure, but Mozilla had taken another PR risk by sponsored contents to the
Packet integration [1]. So we need to keep our eyes to Mozilla.

On wrecking Normandy, you can actually see all enabled recipes [2] and nothing
seems smoky. It even seems that the hotfix (id=721) was used to unbreak Office
365, supporting the positive uses of this system. But I strongly agree that
there should be more approachable list of them.

[1] [https://blog.mozilla.org/futurereleases/2018/01/24/update-
on...](https://blog.mozilla.org/futurereleases/2018/01/24/update-on-pocket-
and-firefox-integration/) (HN discussion:
[https://news.ycombinator.com/item?id=16229927](https://news.ycombinator.com/item?id=16229927))

[2]
[https://normandy.cdn.mozilla.net/api/v1/recipe/](https://normandy.cdn.mozilla.net/api/v1/recipe/)

------
mcny
I absolutely hate this because it is so tone deaf.

What is the point of people like me using Firefox Nightly? Do your tests on
me. Don't do stupid shit with people who choose Firefox Stable. Who came up
with this idea anyway?

~~~
fabrice_d
Unfortunately the population of Nightly users is too small to get relevant
data in many cases.

~~~
mlthoughts2018
That would seem to be a market signal indicating people generally don’t want
to be experimented on by Mozilla.

How is a lack of data for Mozilla my problem? Why does it mean that can inject
default preference changes?

~~~
dralley
>How is a lack of data for Mozilla my problem? Why does it mean that can
inject default preference changes?

It would mean they can't roll out things like hardware acceleration or Stylo
or Webrender as quickly despite their numerous manifest benefits.

~~~
mlthoughts2018
I don’t understand. If users valued those things more than having the browser
be a stand-alone piece of software after it is installed, then users would opt
in to testing.

By not opting in, users would indicated that experiment-avoidance is a feature
that gives them more value than the fast rollout of those other features.

~~~
jamescostian
Would you prefer stable only getting crucial security updates and never
release updates to speed things up? Eventually Nightly would be completely
different from stable, especially with the switch to Rust. So then Mozilla
would have to maintain 2 completely different versions of FF - that's a lot
more work!

A better middle-ground is to let things get tested by those who opt-in to it
(using Nightly and Beta versions), and slowly trickle changes down. That way
none of the published versions are so different that Mozilla needs more staff
to handle the different versions. And of course, stable users have far fewer
issues than nightly/beta users. Certificate expirations throw a wrench in the
whole system, but even if you made FF stable never update, you'd still have a
problem because the cert expired.

~~~
mcny
GP said

» It would mean they can't roll out things like hardware acceleration or Stylo
or Webrender as quickly despite their numerous manifest benefits.

You said:

» Eventually Nightly would be completely different from stable, especially
with the switch to Rust. So then Mozilla would have to maintain 2 completely
different versions of FF - that's a lot more work!

As unofficially mandated by the new owners of the Internet - the Google Chrome
team - the time between two "major" versions of Firefox is six (to eight)
weeks. Yes, we can wait six to eight weeks for new features or twelve to
eighteen weeks from trunk to stable (x2 of six to eight).

What we should do is enourage a wider swath of the population to adopt Firefox
developer edition and Firefox nightly.

» A better middle-ground is to let things get tested by those who opt-in to it
(using Nightly and Beta versions), and slowly trickle changes down. That way
none of the published versions are so different that Mozilla needs more staff
to handle the different versions.

Thank you. This is exactly what I want. I am not saying things should never
change. I'm just saying don't experiment in stable. We are already
hemorrhaging market share as is and this nonsense doesn't help.

------
Tharkun
I had never heard of this Normandy nonsense. I have certainly never willingly
enabled it. Yet according to my preferences, it's enabled. And apparently it's
a feature that lets Mozilla remotely mess with my preferences. What the actual
fuck? When did Firefox go from being a privacy-conscious browser to being this
pile of nonsense? I'm not amused.

~~~
jimrandomh
Most large software projects have something like this. The typical use case is
rolling out a feature gradually, to limit the number of users impacted if a
new feature has problems: first you include the feature in a release, but
disabled by default; then you turn it on for a small percentage of users. If
instrumentation from those users reports crashes, you abort; otherwise, you
enable it for everyone else.

------
gilfoyle4ceo
Dear Mozilla, Please, please, please stop the automagical updates. Is it
really so hard to prompt your users if they'd like to allow a temporary
fix/feature to be installed? You use webcompat to push quick fixes for
specific sites (hidden) you use Normandy to push future features/fixes
(hidden), when your users check their version number via the help/about option
you automatically d/l and update the next version(!!??). Why is it so hard to
ask if this is something that your users want to do???

~~~
anonymousab
>Is it really so hard to prompt your users if they'd like to allow a temporary
fix/feature to be installed?

People might say no, and that would likely very much anger a PM somewhere.

------
gouh
It's okay to change certain settings if they don't touch privacy, for eg
testing WebRender on some subset of users. But it shouldn't touch Privacy
related settings. They should separate the settings into Privacy Sensitive and
non Privacy sensitive and be only able to remotely change the latter ones

~~~
jillesvangurp
Technically, this is just a lightweight way to package up minor settings
changes as an alternative to pushing a normal update to do the same. Both are
perfectly normal and I think today's situation totally justifies using this to
fix this. They do offer a way to turn this off just like you can opt out of
security updates if you insist. For the vast majority of users, automated
updates are a good thing.

It's kind of cool that this worked without a browser restart. My extensions
just reappeared while I was watching some netflix.

------
Santosh83
So I have Normandy turned on and 'install and run studies' turned on and STILL
my extensions have been disabled just now, hours after Mozilla pushed the
temporary fix through Normandy which I presumably haven't got, despite having
it turned on. Latest Firefox on Windows 10.

~~~
neogodless
I had Normandy enabled, but Studies disabled. Same version and OS. A couple
minutes after I enabled studies, my add-ons came back online. Maybe try a
toggle/pause/toggle?

~~~
Santosh83
Indeed. Toggling app.normandy.firstrun from its usual false forces a check
upon next browser restart and the addons are back.

------
zzo38computer
Changing default settings when upgrading the version of a software can be
useful sometimes (especially if the new default value is a value that was not
possible before), although you should be allowed to force a setting to have a
specific value even if that value is the same as the old default value, and be
able to require a list of changes to be mentioned so that you can individually
enable and disable them before installing the new version of the software.

------
ptx
"Pref Rollout is a feature that allows Mozilla to change the default value of
a preference for a targeted set of users, without deploying an update to
Firefox."

Maybe if we put this in terms of _user stories_...

As a user, I don't want to be "targeted".

As the person deploying updates, I don't want the deployment updating without
an update having been deployed.

~~~
dralley
An example of "targeting" in this context is "windows user with an Nvidia
graphics card". For example, enabling WebRender (the new rendering engine) for
those users once it is determined to be sufficiently stable.

~~~
ptx
That sounds like a new version of the software and should be deployed as a new
version, with release notes noting that WebRender is now enabled for Nvidia
users, so that users are prepared for the change.

~~~
dralley
The point is gather information on whether it's ready for a full scale
rollout. The Firefox developers might know that it _might_ be ready, but the
sheer number of software and hardware combinations out there might reveal
unknown issues. So instead of rolling it out to 100% of targeted users, you
roll it out to 1% of targeted users, and measure regressions to your metrics
amongst that group.

------
rectang
I'm one of those users who had data collection (including studies) disabled.
Changing `app.normandy.run_interval_seconds` to 60 (via about:config) didn't
work until I restarted Firefox AND enabled data collection including studies
(via Preferences).

Then all my plugins came back, and I disabled data collection once again.

------
Phenomenit
Does the fix reach tor browser as well? Is Normandy available in tor browser?

~~~
Phenomenit
Seems like tor browser is unaffected.

------
32032141
I didn't realise Firefox came with that sort of backdoor.

~~~
rcthompson
Browser updates can also change default preferences, and change a lot of other
things too. How is this a backdoor any more than auto-updating of the browser
is a backdoor? The one issue I can think of is that if you turn off browser
auto-updates, this should probably be turned off too.

~~~
user17843
it's one step closer and more direct control, which is why this is now being
used to deliver the quick fix.

The downside is that the process of updating the software becomes a bit
fragmented, which is probably confusing users now.

------
stevenwliao
I don't understand the criticisms in this thread. Why would anyone trust
Mozilla's code but not their preferences configurations?

~~~
pdkl95
Trust isn't a single static Boolean value. It depends on the situation,
reputation, and many other factors. Trust is continuously re-evaluated; just
because someone's coding ability was trusted in the past does not imply that
their _current_ or _future_ actions will also be trusted.

However, the current problem people are criticizing is _not about Mozilla 's
choices for default preferences_. The specific changes they ship with the
browser or update with Normandy are not (currently) particularly interesting.
The problem is that a new way to remotely control the browser was added
unannounced that bypassed existing update methods.

If you want to change _other people 's property_, you get permissio9n first.
If someone doesn't want to give you permission and you change their property
anyway, we usually call that something like "trespassing", "vandalism". It
doesn't matter if you think it's an important change or if you don't
understand (or even know) their reason for not granting permission; their
computer is their property, and they don't have to justify why they want to
use it in any particular way.

------
benatkin
Firefox has done so much lately to hurt its image, that I wonder if Google has
double agents working inside Mozilla to sabotage Firefox.

------
OJFord
about:config > app.normandy.enabled > false

~~~
mlthoughts2018
One thing that’s still unclear to me is whether you must disable this entry of
about:config even if you go through the traditional privacy & settings drop
down menus and disable the Firefox studies and usage stats options.

If the _only_ way to prevent having a remote entity modify your settings
unannounced (even if not for malicious purposes) is to enter about:config and
change app.normandy.enable to false, that seems like a situation where the
absolute best case, most charitable interpretation is to call it an incredibly
deceptive dark pattern from Mozilla.

~~~
scolby33
This. After extensive reading about studies and Normandy here and elsewhere,
I’m unable to find a straight answer about this. Does anyone know?

~~~
OJFord
Studies and Normandy are different things, the former often using the latter.
Only 'studies' has a checkbox in about:preferences, 'Normandy' is hidden in
about:config .

------
dschuetz
Oh, I get it now. "Look how fast we could fix an issue with our _Normandy_
preference rollout feature!" Best keep Normandy activated at all time, eh?

~~~
akvadrako
Isn't this an abuse of the system, breaking the contract about what kind of
changes will be pushed on users without their knowledge?

~~~
dschuetz
I don't know. Ironically, I had Normandy enabled (without my knowledge) and
that "hotfix" they supposedly rolled out didn't reach my own client. So, what
ever that feature is for - it doesn't even work. Apparently you also need to
have "studies" enabled as well. It didn't work either. So, I'm waiting for
official updated release. If they don't fix this soon - I'm done with Firefox
and Mozilla.

~~~
dralley
The client checks periodically, they can't really "push" out that change
immediately to everyone.

~~~
dschuetz
I'm willing/trying to _pull_ the fix manually, but that doesn't work either.

------
hartator
Why not just update their SSL certificate?

~~~
teddyfrozevelt
Because it's not an SSL certificate.

------
dschuetz
I switched to Vivaldi today.

