
Google Chrome Keystone is modifying /var symlink on non SIP Macs causing k-panic - frandroid
https://mrmacintosh.com/google-chrome-keystone-is-modifying-var-symlink-on-non-sip-macs-causing-boot-issues/
======
Scapeghost
I've always hated that "service" (more like malware given this news) like
everything else that installs itself into the autolaunch sequence without
permission, and remove* it whenever I notice/remember it, but it keeps coming
back whenever I touch Google Chrome, which I prefer not to use in favor of
Safari/FireFox because of reasons like this.

Things like these (including secretly signing you into Search when you sign
into YouTube† or refusing to support PiP on iPadOS/macOS) just solidify
Google's image in my mind as a forever scummy, intrusive company that I wish I
could leave behind like I did Microsoft, but sadly Google Search and YouTube
still don't have good enough alternatives yet.

* (startup items usually reside in the LaunchAgents/ and LaunchDaemons/ folders in your user ~/Library/, the root /Library/ and /System/Library/)

† (you can fix this by deleting all Google cookies after signing into YouTube,
on any OS)

~~~
apostacy
The trends that Google has spearheaded have had a real effect on me over the
years.

I feel alienated from my computer. Subtle things will just change. If I really
dig I might be able to find out why, but I don't have the time, so I just
accept it.

Usually very small things that are barely noticeable. My Chromecast extension
disappeared and was integrated into the browser. My brain could not help but
notice this benign change, which caused a hard to place sense of unease.

Or when Google decided to remove rotation from the home screen on Android 2.3
-- it wasn't a huge problem, but I could have sworn that something changed.
Users were conflicted, many convincing themselves that the homescreen never
rotated at all.

It has made me not trust my computer. I second guess myself much more. If some
option no longer exists, I wonder if it was just my imagination or if it was
quietly deprecated while I wasn't looking. Does it even matter?

I think that we are being trained to see devices as ephemeral, and not to get
too attached to them.

~~~
cortesoft
You feel alienated from your computer because there has been a conscious
decision to take away options and user control in modern software. And I get
why that decision has been made, even if I hate it as much as you and every
other computer enthusiast.

For 90% of people, they have always "felt alienated" from their computers.
They didn't understand what was happening or why things changed either, and it
was easy to get yourself into trouble if you didn't know what you were doing
and were trying to figure out how to fix something.

So companies decided to make their software have fewer options, and do more
things automatically, without asking the user to have to make a choice. They
don't give the users an option to customize, so they don't have to worry about
those customizations causing breakage.

For advanced users this is crippling, but there are a lot more of them than
there are of us, so they are going to be catered to.

~~~
yesco
Ironically despite there being less choices in an effort to make it easier for
that "90%", the amount of tech support friends and family request from me has
only increased in the past years.

Personally I don't buy the whole "removing choices to stop users from hurting
themselves" excuse. To me it seems like over zealous designers trimming far
more than necessary to make things look nicer at the cost of usability. But
what do I know?

~~~
grenoire
Regarding the increase, I would argue that is less about increased device
complexity but more about the increase in the amount of people who are using
these devices on a daily basis. More they use, the more problems they
encounter; I think the intersection between complexity and usage is determined
moreso by the latter.

~~~
nvrspyx
I doubt the number of people using smartphones and computers has increased
significantly in developed/first-world countries, especially when it comes to
one’s family and friends.

As anecdotal as it may be, my friends and family have used computers and
smartphones for years, but I’ve experienced the same increase in requests for
tech support as the parent comment.

Further, no one said there’s increased complexity. The argument is that the
oversimplification, the removal of features, and overzealous design
assumptions have made UX go in the wrong direction. It’s also an argument I
agree with.

A lot of UX design today fails to recognize the _spectrum_ of “tech literacy”
and it should, ideally, accommodate all within that spectrum, rather than
pander to the least “tech literate” end. It’s not always possible, but it
should be strived towards. Instead, we have UX trending towards attempting to
be so “intuitive” that it becomes counterproductive.

~~~
apostacy
I had an interesting incident recently, where I was with some relatives and we
were trying to plan the next leg of our trip; what restaurants to go to and
what directions to take, etc.

Anyway, we had to start using a paper notepad and pens to keep track of the
information! Even for people who just want to paste an address from a text
message to look up in maps, and especially if you want to do anything with the
calendar.

I just remember 15 years ago on my Treo 650 never needing to do that, and
having no problem copying text between different apps seamlessly, between
calendar, email, text, maps, and other apps. Same with Blackberry. Using
modern Android is as awkward as driving a car with a mouse.

But I think there was an intentional push to minimize options for users, to
make fewer pathways for things to go wrong. Forcing people to use pen and
paper when they have a smartphone next to them is a UX success for them,
because they don't have to improve handling text.

------
norberg
Hey. Google Keystone tech lead here. We are aware of the issue, and we've
stopped the release. We're building a replacement that fixes the problem. In
the meantime, to fix affected machines:

    
    
      sudo rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle
      sudo ln -F /private/var /var
    

This deletes the affected version of Keystone and reinstates the damaged /var
symlink.

The version of Keystone packaged with Chrome is not affected by this bug, so
allowing it to reinstall Keystone will not recreate the issue.

~~~
Doctor_Fegg
You're missing the word "sorry" from your response.

My wife's a primary school headteacher (or K-12 as you say in the States). Her
MacBook was disabled by this. Yes, she takes weekly backups, but schools don't
have free money to spend on spare laptops for a few days' work, nor on
unnecessary technician time to fix it. Fortunately I spotted this posting
(thanks, HN poster!) on blearily checking HN this morning and instantly
recognised this was what's happening.

Have some decency for the people whose lives you've just affected and
apologise to them.

~~~
dang
I understand the frustration, but please don't attack someone like this when
they come to HN to supply information. It creates a hostile environment and
disincentivizes people who have inside knowledge about a situation from
showing up here. That makes HN a strictly worse place. It also breaks the site
guidelines, which ask us all to _Be kind_ , regardless of how strong and
justified one's feelings are.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

~~~
Doctor_Fegg
Not intended personally to @norberg, but corporately to Google.

~~~
dang
OK, but norberg is the person you blasted, and intent unfortunately doesn't
express itself on the internet.

~~~
ectospheno
A truth stated passionately doesn't become false. A falsehood stated calmly
doesn't become true. This is at the heart of why appeals to emotion are almost
always logical fallacies.

~~~
squeaky-clean
I don't think dang is saying that the commenter was making false claims or
anything. Just that it's very unlikely an upset comment will cause an overhaul
in the google auto-update system. But it is very likely an upset comment will
scare developers away from commenting on future situations like these. It just
affects the health of HN negatively while not affecting Google. There's
probably a reason norberg chose to register and comment on HN and not
somewhere else like Reddit.

~~~
ectospheno
Content is wrong or it isn't. Tone is a logical fallacy.

Your true statement that tone will often matter is an interesting discusson on
society and education. That it is also relevent on a site otherwise dedicated
to intelligent discourse was the nugget I was hoping people would think about.

------
michaelt
Reminds me of the Steam bug back in 2015 [1] where on Linux, if you tried to
move where Steam stored downloaded games, it would wipe your hard drive by
running "rm -rf "$STEAMROOT/"*" with $STEAMROOT being null.

[1] [https://github.com/valvesoftware/steam-for-
linux/issues/3671...](https://github.com/valvesoftware/steam-for-
linux/issues/3671#issuecomment-70021818)

~~~
olafure
I start bash scripts, by setting the holy trinity: set -eou

That prevents end-of-the-world scenario like the one above, if the script
derails.

~~~
saagarjha
Maybe also add "pipefail" to the end of that?

~~~
scarejunba
Haha right, I feel like these should always be popularized as `set -eu` and
`set -o pipefail` rather than making it look like you an `set pipefail`. I
wonder if that chap has been uselessly printing options at the beginning of
scripts for a while now.

------
SloopJon
I've been wary of Gatekeeper and SIP as moving Macs towards an iOS-style
walled garden, but this is a perfect case of SIP protecting the user from bad
software.

~~~
userbinator
On the contrary, I think that sort of protection just hides problems --- like
this one. As a general rule, bugs with the highest impact are also the ones
which are most likely to be fixed quickly. If you tested with SIP on, it'd try
to remove /var but wouldn't succeed, and you'd think everything is OK when the
application's logic is actually faulty.

~~~
yjftsjthsd-h
If SIP blocks program from accessing system files, shouldn't it alert or
something?

~~~
makecheck
There’s plenty of bad code that fails to check for errors so the OS may well
have flagged something here and the program just didn’t know/care.

It seems even more likely that the result of unlink() would be ignored (right
up there with ignoring printf()), not because it’s the right thing to do but
because lazy programmers will assume that failures are incredibly unlikely or
unimportant. For example, if the code is a cleanup phase that just wants to
remove a list of files, what are the odds that the program dutifully checks
that the files actually went away?

~~~
userbinator
_For example, if the code is a cleanup phase that just wants to remove a list
of files, what are the odds that the program dutifully checks that the files
actually went away?_

Or, as the reason for the omission of such checks is more likely to be, what
to do if something that shouldn't fail, fails? And if whatever you decide to
do to handle the error itself _also_ fails? Repeat ad infinitum. To even try
to go down that rabbithole is simply a waste of effort and does nothing but
introduce unnecessary complexity, to put it bluntly.

------
teamspirit
So this caused me to have to reinstall the OS yesterday. Glad to know what the
issue was.

And if anyone wants to know, I have to disable SIP because Apple won't let me
use an eGPU on my Macbook with TB2.

~~~
chmarr
Have you tried going into System Preferences, Security, Advanced (I think it's
"advanced", it's at the bottom right of the screen.) There you'll find a list
of drivers, or something, that you can enable. I can't be more specific than
that since my mac here is under IT control, and the feature is disabled.

~~~
saagarjha
I believe those are the instructions for kernel extensions that are "approved"
but distributed by third-parties.

~~~
stunt
Someone here suggested that you could do `csrutil enable --without kext` to
load untrusted kernel extension without disabling SIP entirely.

------
shortformblog
This is also affecting Hackintosh users. A fix for them is listed here:

[https://www.reddit.com/r/hackintosh/comments/d8tm8z/psa_goog...](https://www.reddit.com/r/hackintosh/comments/d8tm8z/psa_google_chrome_updaterkeystone_rendering/)

I wonder what role Apple’s aversion to working with Nvidia played in all those
Avid users having SIP turned off.

~~~
presidentscroob
Hackintosh or not isn't relevant because it's a macOS + Google update service
issue.

PS: Written on a Hackintosh

------
jbverschoor
Google, please tell me how to update Chrome without keystone.

Found non-functional system update engine. Please reinstall Google Software
Update from
[https://dl.google.com/mac/install/googlesoftwareupdate.dmg](https://dl.google.com/mac/install/googlesoftwareupdate.dmg)
KSUpdateEngine no ticket to update for the specified product ID.

I don't want a "System update engine"... This is baked into Apple's AppStore.
It works very well. Use that. You don't need access to my system.

~~~
masklinn
> Google, please tell me how to update Chrome without keystone.

That's easy, just regularly download a new Chrome. The difficulty is managing
to stop keystone from reinstalling and re-enabling itself.

> This is baked into Apple's AppStore. It works very well. Use that.

I hate keystone with a passion, but TBF getting a modern browser into the
appstore is not possible, even ignoring all the limitations the store puts
upon its software, there's no way you can actually get a _browser_ (as opposed
to a shell UI around the platform webkit) in the appstore by its rules.

------
lostmsu
Can somebody explain in simple terms what Keystone is, what was it trying to
do, which caused /var unlinking, and why does it cause MacOS to panic?

~~~
dreamcompiler
Keystone is Google's auto-updater program. It updates not only Chrome but also
Earth and other Google programs. It's a notorious resource hog and it tries
_very hard_ not to let you ever turn it off. If you manage to uninstall it, it
will try even harder to reinstall itself the next time you run a Google app.

Keystone is malware made by Google. The incident this week was the first time
it contained an actual destructive payload, but it's been malware for years.

~~~
kovrik
Where can I find a manual on how to uninstall Keystone, Chrome etc. on Mac
properly? Has anyone done it?

------
rgovostes
There are a few legitimate reasons to disable SIP, but too often I see people
turning it _entirely_ off, rather than just disabling the parts that are in
the way:

    
    
        csrutil enable --without kext
        csrutil enable --without fs
        csrutil enable --without debug
        csrutil enable --without dtrace
        csrutil enable --without nvram
    

If you want to load some untrusted kernel extension, the first one will let
you do so, but still keep all the other SIP protections on. If you want to use
DTrace, use the corresponding flag. Etc. You can mix and match flags.

~~~
stunt
Thanks for sharing.

Is it documented anywhere? man page didn't help.

~~~
rgovostes
I do not think that they are documented officially. The only "supported"
configurations are all on/all off.

------
parliament32
Why do userland installers need root again?

~~~
londons_explore
In the case of Chrome it's because some of the sandboxing can't be done by a
regular user.

Same with both Linux and windows.

Bit of a design flaw with the OS - in all cases a process should be allowed to
restrict itself to have fewer permissions and access to fewer API's without
being root, but sadly that isn't universally the case.

~~~
mehrdadn
> Same with both Linux and windows.

Could you clarify how this is the case on Windows? I thought Google Chrome
installs and runs just fine without admin privileges. I'm not aware of any
security downsides for doing so.

~~~
judge2020
Maybe it's possible but the current installer has windows pop up a UAC prompt
before it continues.

~~~
mehrdadn
That's just because it wants to install machine-wide if possible. You can just
tell it to continue without admin permissions and it tells you explicitly that
it can be installed without that.

Note that Windows doesn't work like Linux with setuid bits and whatnot. The
permissions a file is installed with don't dictate what permissions the
program that executes it has. That's entirely a function of the program's
security context. Hence, for a machine-wide installation to actually make a
difference security-wise, Google would actually have to install e.g. a high-
privilege service that would run when you try to start Chrome. I don't think
it does such a thing.

So I think Windows is already designed correctly in this regard and hence I
don't think this is an issue on Windows as claimed.

------
dang
Related thread:
[https://news.ycombinator.com/item?id=21057157](https://news.ycombinator.com/item?id=21057157)

------
jontro
Looks like the issue has been reported here:
[https://bugs.chromium.org/p/chromium/issues/detail?id=100735...](https://bugs.chromium.org/p/chromium/issues/detail?id=1007358)

------
forgotmypw3
Chrome hasn't been present on my desktops for a couple of years, and I don't
miss it.

------
olliej
Ok, for those who don't use Chrome (unless absolutely necessary):

* what the heck is keystone?

* why is it running any time Chrome isn't?

* why is a browser installing a root service?

* why is a piece of software changing root level symlinks in the first place? Clearly it doesn't need to because SIP prevents that nonsense

* Finally: is this enough to explain why SIP/rootless is a good feature?

~~~
saagarjha
> what the heck is keystone?

Keystone is Google's updater service for their software.

> why is it running any time Chrome isn't?

It runs updates in the background, so it needs to run when Chrome doesn't.

> why is a browser installing a root service?

¯\\_(ツ)_/¯

> why is a piece of software changing root level symlinks in the first place?
> Clearly it doesn't need to because SIP prevents that nonsense

Probably a bug.

> Finally: is this enough to explain why SIP/rootless is a good feature?

Well, a number of people decided that SIP was hindering them enough to turn it
off, so I'm not sure…

~~~
olliej
> > why is it running any time Chrome isn't? > It runs updates in the
> background, so it needs to run when Chrome doesn't.

Gnah

> > why is a browser installing a root service? > ¯\\_(ツ)_/¯

Ok so it needs to replace the bundle - I feel Apple should add support for
replacing binary A with binary B if A and B has the same signing key, although
obviously there are a bunch of fun issues involved, I think that case
shouldn't necessitate an update service running as root :-/

> > why is a piece of software changing root level symlinks in the first
> place? Clearly it doesn't need to because SIP prevents that nonsense >
> Probably a bug.

Wah wah

> > Finally: is this enough to explain why SIP/rootless is a good feature? >
> Well, a number of people decided that SIP was hindering them enough to turn
> it off, so I'm not sure…

The general problem is that it's still easier for developers to say "disable
SIP by doing ..." without saying "we haven't written our
[drivers/application/whatever] properly", rather than just writing the
software properly. Which you know is possible because even in kernel driver
land you hardly ever see driver's claiming that it's necessary.

e.g. its necessary from an end-user PoV but only because companies don't want
to pay devs to put effort into working with SIP enabled when there's a much
cheaper "tell the user to disable security" option available.

~~~
saagarjha
Actually, the issue is that Apple won’t sign drivers for eGPUs that film
professionals use.

------
J5892
Holy crap, that's what's happening?

It took me hours to figure out that the failed boot was caused by the `/var`
symlink being removed. I was literally a minute away from reinstalling the OS
when I saw a post from 2014 that had a passing reference to the `/var`
symlink.

Then I went through and disabled every conceivable startup program, and even
created a bash script to fix `/var` when it randomly disappeared. I didn't
even consider that it could be Chrome causing it.

This all happened with SIP enabled, btw.

~~~
kovrik
How come SIP didn't catch it? Is Chrome asking for a root access during
installation?

~~~
J5892
I have no idea. After I fixed the issue, I even tried disabling SIP, then
enabling it again. It's still randomly removing the symlink. (well, as of last
night around 10pm. I haven't opened the computer since then)

That computer has been through a lot, though. So it's totally possible that I
did something stupid to permafuck SIP.

~~~
NobodyNada
When you recreated the symlink, did you add the SIP flags to it?
[https://news.ycombinator.com/item?id=21066472](https://news.ycombinator.com/item?id=21066472)

------
mlang23
To sum up most of the comments here: Google is the new Microsoft. Fascinating,
how a company can go from "we will not do evil" to "fuck you all" in just 5
years.

~~~
shantly
I thought they started to get kinda crappy back in '08 or '09, it just took
until recently for the consensus to catch up.

~~~
jm4
Agreed. That's around the time I started using some weird setup where my
google searches would go through Tor and I was blocking their cookies. It was
overboard, probably unnecessary and probably didn't accomplish much, but it
was the result of getting a creepy vibe from google. Once I discovered DDG, I
switched and never looked back. It took a little while to get used to it, but
the results are good enough that I keep using it. I run maybe a few google
searches a year if I can't find what I need on DDG and I usually don't find it
there either. I don't think google is significantly better, although it is
noticeably faster. There's nothing about it that's appealing enough that I
want to accept the bad things that also come with it.

------
saagarjha
I have SIP disabled and Chrome installed…will my /var be broken by sometime
tomorrow when Keystone runs? Can I just disable the launch agent to fix this?

~~~
norberg
We have stopped pushing the affected version of Keystone, so if your computer
has not been broken yet, it won't be.

If it has been broken but not rebooted:

    
    
      sudo rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle
      sudo ln /private/var /var
    

should fix it. That deletes Keystone and fixes the symlink. If it has been
rebooted, these commands at a recovery console should repair the computer.

Chrome will subsequently ask for admin credentials to reinstall the updater
next time you run it. This will not re-break the computer; the version of
Keystone bundled with Chrome is older, and we have stopped serving the version
affected by this issue.

~~~
im3w1l
Please don't run that command as stated. It deletes /var (because HN merged
adjacent lines)

Edit: It has since been fixed.

~~~
saagarjha
I won't, don't worry ;) That code should probably be indented two spaces, like
this:

    
    
      sudo rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle
      sudo ln /private/var /var

~~~
norberg
yikes! thanks, fixed.

~~~
akie
I see a pattern here...

~~~
Thorrez
I doubt Hacker News automatically merging 2 lines into 1 line was the cause of
the original bug.

~~~
J5892
You don't run your code through HN as a post-commit hook?

------
neonate
[https://web.archive.org/web/20190924204425/https://mrmacinto...](https://web.archive.org/web/20190924204425/https://mrmacintosh.com/google-
chrome-keystone-is-modifying-var-symlink-on-non-sip-macs-causing-boot-issues/)

------
Animats
The "fix" looks more like a virus removal job. It includes

    
    
        rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle
    

and doesn't replace it with anything.

------
londons_explore
What's the link with AVID?

~~~
msbarnett
It's coincidental: the first widespread reports of this were from Hollywood
editing shops: [https://variety.com/2019/digital/news/avid-mac-pro-
corrupted...](https://variety.com/2019/digital/news/avid-mac-pro-corrupted-
hollywood-1203347033/). This lead to the initial presumption that it was a
being caused by a bug in AVID.

The reason the AVID community popped it first seems to come down to the fact
that this is their busy season, so a lot of their machines were active last
night as this Keystone update was rolling out, editors for whatever reason
(technical issues or superstition) reboot their workstations fairly often,
and, crucially, a lot of editing workstations are using third-party GPUs that
require them to disable SIP (whether this is particular to AVID or just an
intrinsic property of using the Mac Pros with third-party GPUs, I don't know).

~~~
Mindwipe
> (whether this is particular to AVID or just an intrinsic property of using
> the Mac Pros with third-party GPUs, I don't know)

Definitely more the latter.

------
1GR3
Since this is a thread where men can talk about their feelings, I'd like to
say that I feel angry and frustrated! If I wanted a machine for browsing web
and watching youtube videos while always been safely signed into my google
account, I'd bought a f@©4!n9 Chromebook.

------
will_hoskings
It's funny how Google Chrome even needs this in the first place, to be honest.
This is another great reason to move to Firefox, with the rest of us :p

------
jbverschoor
This is the same shit Adobe is doing.

------
pier25
Does Chromium also install keystone?

------
kzrdude
Switch to Linux or stop using Chrome, unfortunately.

------
jbverschoor
Does chromium suffer the same malware?

------
fortran77
Why does the Mac OS allow this?

~~~
kyralis
Modern MacOS in default configuration does not.

~~~
fortran77
So did all these Hollywood people change the "default configuration" or did
the standard AVID installer disable it?

~~~
gumby
Some people wanted to use video cards with unsigned drivers. The hardware Mfr
said “disable this malware blocking security feature in order to use our
hardware”. As it happens most people with exotic video cards are avid users.

They disabled the malware protection and got killed by the malware.

~~~
fortran77
On other platforms, those cards aren't "Exotic" (I run Premiere Pro on Windows
10 with a pair of 2080Ti cards for rendering. Premiere and After Effcts _fly_!

One problem is that Apple abandoned the Pro market, but some users are very
loyal.

~~~
gumby
They're exotic in that their manufacturers can't be bothered signing their
drivers. But at least there are drivers.

They're also exotic in the sense that only a very small proportion of the
overall user base cares/requires them.

There are supported video cards / TB3 video systems that are natively
supported but at the moment they are possibly even more exotic in the sense of
shipping in low volume (per your second point). I really don't like "apple
abandoned the XXX market, which I am in and wish they had just the right
product for _ME_ " statements but in this case, I think your comment is
unfortunately correct.

