
What Happened: Adobe Creative Cloud Update Bug - slyall
https://www.backblaze.com/blog/adobe-creative-cloud-update-bug/
======
jwr
It is unbelievable how invasive Adobe software is. I try very, very hard to
avoid installing anything by Adobe, because I know that the software will do
whatever it wants on my drive, install itself all over the place, install
various (ugly) menubar items which are non-removable, run things at login,
make a "Creative Cloud Files" on my Desktop (!), make me a member of the
"Community", demand my password to get administrative privileges, without
explaining what for, make lots of network connections to various servers,
sending unknown information about me, run various processes in the background
(AAM Updates Notifier, anyone?) and run various updaters with annoying popups
whenever it feels like it. I think CC will also pester you with upselling
popups, or new features information.

Back when Flash was still a thing, they got so low as to try to sneak in more
of their products when I just wanted to update Flash.

It's crappy behavior and should be called out for being what it is. We are too
quiet about it and we got so used to taking crap from software producers that
many people don't even complain.

~~~
mattmanser
This is a fairly common programming mistake to make, nothing to do with Adobe
being invasive. I remember when eve online's updater deleted C:\boot.ini in
2007 for example:

[http://community.eveonline.com/news/dev-blogs/about-the-
boot...](http://community.eveonline.com/news/dev-blogs/about-the-boot.ini-
issue/)

That said I completely agree Adobe's software is disgustingly invasive, just
not for this particular reason.

~~~
cypret
This is not a common programming mistake to make at all, I'd also say it has
everything to do about being invasive. The EVE reference you mentioned was 9
years ago for a video game in an early release stage.

~~~
matheweis
I don't know the exact nature of the Adobe bug, but Steam had a similar
problem recently too "Moved ~/.local/share/steam. Ran steam. It deleted
everything on system owned by user" [https://github.com/valvesoftware/steam-
for-linux/issues/3671](https://github.com/valvesoftware/steam-for-
linux/issues/3671)

Honestly (and imnsho) the prevalence of that sort of thing makes it even less
excusable for a large company like Adobe.

~~~
revelation
That is incredible.

> Line 468: rm -rf "$STEAMROOT/"*

Without a check for $STEAMROOT validity. I'm not sure you couldn't sue Valve
for the incredible recklessness. That is just beyond the pale.

~~~
cmurf
In general you can't. Most all software EULAs pretty much says you agree there
is no warranty and no liability no matter what happens.

~~~
Sharlin
EULAs are not magic, at least in sane jurisdictions. A lot of stuff they
contain is not legally binding.

------
korginator
Kudos to BB for their swift actions.

Under no situation should any software randomly delete folders and files it
doesn't own, period, specially something so widely used by professionals. This
is not just a little bug on a rarely used OS platform not limited to people
using backblaze on the Mac, it should have been a show-stopper, and should
have never left the test environment (do they even have one?). It reflects on
the abysmal quality of Adobe's software and their QA.

There's no reason Adobe should even touch the root directory "/" on the Mac,
leave alone delete anything from there. Why does Adobe CC need root
permissions to install on the Mac in the first place?

I hope the people responsible for this disaster, their managers and the QA
team were fired. In my business, this sort of mess can sink our company.

~~~
michaelt

      It reflects on the abysmal quality of Adobe's software
      and their QA. [...] I hope the people responsible for
      this disaster, their managers and the QA team were fired.
    

Eh, I can kind of see how it might happen myself. Remember that time Valve had
a bug in Steam for Linux that would wipe the user's hard drive? [1]

It's easy to imagine how this sort of thing would happen - manual test guys
don't have hidden folders in their root directories because their boxes are
locked down by corporate IT and they can't create any folders in the root
directory. Developers don't have them because they understand the unix way and
know they shouldn't be putting files there. CI servers don't have such folders
because, well, why would they? None of the test environments include the
specific condition needed to trigger this bug - a hidden folder in the root
directory - so the bug never appears in testing.

Is it bad and serious? Absolutely. But I don't think the fact testing missed
this is a sign of testing incompetence - you could have cutting edge testing
following every industry best practice and still miss this bug.

[1] [https://github.com/ValveSoftware/steam-for-
linux/issues/3671](https://github.com/ValveSoftware/steam-for-
linux/issues/3671)

~~~
speeder
Honestly? Adobe has been pulling this sort of bad stuff for a while now, it is
not just a dumb mistake.

I am a hardcore fan of Macromedia Fireworks, and Adobe releases caused so much
problems that I decided to use Macromedia Fireworks in the literal sense (ie:
in my current machine, I have Macromedia Fireworks 8 installed).

When Adobe bought Macromedia, I installed their Adobe Fireworks thinking it
would be just an updated Fireworks... it was right, and wrong, right because
it barely had any new features or bug fixes, wrong because it install for some
reason needed "adobe common files" anyway, and installed 3gb of crap on my
disc (when discs were still 20gb big, thus it was a huge waste of space), AND
somehow it conflicted with their own Adobe Reader and made it stop working.

When Adobe Fireworks CS3 was released, I was in university, I was excited to
try it... and not only found again it had almost no new features, old features
don't worked correctly either, most aggravatingly, the selection boxes would
always render in the wrong place, and not just 1 pixel off, they would be
completely off, when I tried to input text for example, the text box appeared
300 pixels away from the actual text.

When a friend installed it on his computer, the computer started misbehaving.

And the problems kept coming, to the point that eventually I gave up, I have
no idea what version Fireworks is now, because I decided when CS6 came out, to
install on my machine Macromedia Fireworks, and not install any Adobe product
on my computer (eventually I found it was hard to avoid Acrobat Reader, thus
now I install it, but only that).

EDIT: Also I always disable Adobe Updater, the first thing I do after
installing any Adobe product on any computer for any reason, is shut down the
updater process, and forbid it from running.

~~~
Gratsby
Old Macromedia Fireworks (and current Adobe Cloud) user here. Fireworks still
has not changed in any positive way since Macromedia developed it.

The bright side is that they haven't completely ruined it. I haven't noticed
the text box problem, but I skipped many versions before I switched over to
OSX and had to get the new one.

------
markbao
Adobe is handling this terribly. From the forum post [0], a number of people
are reporting losing data, with one poster losing "all of my working files for
my clients" and another saying "197 GB of working files Gooooone!". Adobe
simply says "You can go ahead and update the desktop application. The issue
has been rectified."

Surely there are others who didn't even know they lost data; when asked about
how they'll notify those people, Adobe responds "We have advised customers to
take local backup of their files just to be on a safe-side."

Some talk coming from a company that deleted user files with wanton disregard.
I'm aware that legal culpability is a complicated topic, but Adobe is doing
themselves no favors here.

[0]
[https://forums.adobe.com/thread/2089459?start=0&tstart=0](https://forums.adobe.com/thread/2089459?start=0&tstart=0)

~~~
mcmillion
It's deplorable. I follow Adobe on Twitter and pay attention to various sites.
I heard about the problem from BackBlaze.

Absolutely 100% unacceptable. Massive props to the BackBlaze team for handling
Adobe's shitstorm so well.

~~~
atYevP
Yev from Backblaze here -> Thanks! We tried to act pretty quickly since it was
causing deletions of data. As a backup company that's a thing we try to help
people avoid :P

------
fishfacemcgee
Good postmortem from BB. Crappy situation for them to have to be in since it
was all Adobe's fault, but I think they handled it well. They didn't off-load
problem-solving to Adobe, but made it clear what the root cause of the bzvol
error was.

Meanwhile, I'd _love_ to see the postmortem from Adobe. What file(s) were they
expecting to delete and why did they feel it prudent to not do a
strict/stricter check for the files/directories to delete? Presumably some
sort of "adobe" directory, but I would hope to at least find out that they
couldn't reliably expect the folder to be a specific name and the issue was
that their searching was just too loose.

~~~
mirimir
They obviously made no check. The goal was probably to delete all Adobe
configuration files. And somehow the first hidden folder in home became a
target. Because .adobe is obviously going to be first, right?

~~~
ajanuary
Deleting the first directory seems like a harder problem than deleting a
folder called .adobe.

~~~
mikeash
I don't know about that. It's harder to do it right, but they probably didn't
bother with doing it right.

For example:

    
    
        ls / | head -n 1 | xargs rm -rf

~~~
ajanuary
That's easier than

    
    
        rm -rf /.adobe
    

?

My point being that things indicate a more interesting problem they were
trying to solve than just deleting a directory. What that actually was, and
how that lead to such an awful solution, would be interesting to know.

~~~
mikeash
I agree it's pretty far fetched. I'd love to know what the real story is too.

------
mwcampbell
A common response here is to bash Adobe for its lax approach to software
quality. I think it would be better for us to look in the mirror and ask
ourselves what we can do to improve the robustness of our own software. A few
ideas:

\- For production code, categorically reject any language that will happily
treat an unset variable as an empty string. Or, if possible, configure the
language implementation to run in a stricter mode that fails fast instead.
Question: Is that possible with shell or PHP?

\- Favor manipulation of data structures over concatenation of strings. See
Glyph Lefkowitz's blog post "Data In, Garbage Out" [1]. When manipulation of
strings is practically unavoidable, as with filesystem paths, use a well-
tested library such as Python's os.path module or .NET's System.IO.Path class.

\- Embrace the principle of least authority at every level. For example, when
developing for the Mac, work within the Mac app sandbox as much as possible.

Any other ideas on how to prevent this kind of catastrophic bug?

[1]: [https://glyph.twistedmatrix.com/2008/06/data-in-garbage-
out....](https://glyph.twistedmatrix.com/2008/06/data-in-garbage-out.html)

~~~
emilburzo
> Is that possible with shell [...]

#!/bin/bash

set -eu

<actual work>

Explained:

    
    
       -e  Exit immediately if a simple command exits with a non-zero status, unless
           the command that fails is part of an until or  while loop, part of an
           if statement, part of a && or || list, or if the command's return status
           is being inverted using !.  -o errexit
    
    
       -u  Treat unset variables as an error when performing 
           parameter expansion. An error message will be written 
           to the standard error, and a non-interactive shell will exit. -o nounset
    

Worth a read: [http://ss64.com/bash/set.html](http://ss64.com/bash/set.html)

~~~
mioelnir
There is also the variable-expansion abort, even with optional custom error
message:

    
    
        "${VAR:?Error message goes here}"
    

Will exit2 if VAR is unset or the empty string.

Or, which I often use, build the path you want to delete, call realpath on it
- then check if your base directory you never ever have any business leaving
is a prefix string of the realpath resolved path. This still allows to symlink
stuff into subdirectories but will catch if something resolves suddenly all
over the place.

------
nl
File delete bugs are (obviously) dangerous and sometimes have surprising
causes.

A long time ago, we had a Java web application that we'd deploy, and after a
couple of days would stop working. We'd investigate and find it hadn't been
deployed properly, yell at the person who installed it and go on our way.

Until the day data disappeared. That was weird.

Then we saw files physically disappearing while we had Windows Explorer open
in front of us.

So it turned out that a library we were using was using exception handling for
flow control, and in a limited set of circumstances this meant a file wasn't
closed. That was fine. Usually.

Except that Java 1.4.1_01 had a bug on Windows where if you opened more than
2036 files open THE NEXT FILE THAT WAS OPENED WOULD BE DELETED[1].

I'll never forget the pain.

[1]
[http://bugs.java.com/bugdatabase/view_bug.do?bug_id=4779905](http://bugs.java.com/bugdatabase/view_bug.do?bug_id=4779905)

------
0xcde4c3db
Several sources (including Adobe's own blog post on the issue [1]) suggest
that the problem is in the update code, but the Backblaze guys say (and show,
in videos) pretty clearly that the deletion recurs every time the user signs
into the Adobe Creative Cloud account. Is this actually some DRM bullshit?

[1] [https://blogs.adobe.com/adobecare/2016/02/12/creative-
cloud-...](https://blogs.adobe.com/adobecare/2016/02/12/creative-cloud-
desktop-on-mac-update-issue/)

~~~
atYevP
Yev from Backblaze -> no idea really, we initially thought that it was "on
update", but then when we dug further it appeared that it would fire any time
a sign-in occurred on the broken version. Not sure what the exact code was
though, or what the intent there was.

------
atYevP
Hello gang! Yev from Backblaze here to answer any questions. Sorry, just woke
up. Who posted this? Australians? :D

------
Pxtl
Once again, Adobe demonstrates that they have the worst software quality if
any company at their tier of the market.

Such a shame that they're market-leader in so many ways, because once you get
passed the pretty uis and impressive features, their software just a train
wreck.

~~~
Jgrubb
If anything, a testament to the importance of a good UI if you want your
software to be used by people.

------
admiun
This just shows we really do need a step towards an 'appstore-like'
permissions model for desktop operating systems. There is just no reason for
all these software packages to have full write access to everything in your
user account.

~~~
derefr
You know, Thinstall (later VMware ThinApp) has existed since 2001. I'm
surprised its particular breed of containerization (effectively an LD_PRELOAD
shim that defines virtualized versions of libc calls) never took off. It was
perfect for providing this kind of "isolation": the kind where you want the
app to "not make a mess of things"—but aren't aiming for container-like
isolation-based security, because the apps you're running are known quantities
(if stupid ones.)

It's interesting, also, that Windows' WoW16, and then WoW64, both provide
their own levels of filesystem virtualization for "messy" apps... but those
same constraints aren't pushed on "native" apps.

I still don't really understand why no OS just virtualizes _every_ app's
filesystem, without having to opt into something like sandboxing. It'd
actually be able to provide a much nicer programming model, a lot like Plan9:
just spew all your program's files into the virtualized equivalents of system
directories, because they're directories that are really just for you. No
subdirectories; you just put configuration right in /etc, manuals right in
/usr/share/doc, etc.

That could then be combined really well with a database-filesystem: going in
the file manager to /usr/share/doc would display a "virtual library directory"
with virtual subdirectories for each app-container that had made use of the
directory. (Or you could skip the virtual subdirectories and get a merged
view. Good for e.g. a Fonts directory.)

------
jmspring
Honestly, the best part of this story for me is I am finding out the details
from Backblaze. I remember when I worked at Zetta.net and I chatted with a
couple of members of the founding Backblaze team at the time about their DC
issues (we were in the backup biz as well). Thoughtful, pragmatic, and
straight forward.

I was using Backblaze back then and still do.

A great company and well worth supporting.

Thank you for the analysis and sorry for the troubles you encountered due to
Adobe -- a company that had multiple "we are doing better for the employee" HR
moments which mean "you are screwed" over the last 5-7 years.

~~~
atYevP
Yev the non-founder here -> Hah, that's great to hear! The founding members
are still a silly bunch.

------
dilap
It's a shame Apple is doing such a bad job with the Mac app store -- it would
be really nice to have a system that limited the damage to the app's own
playground and files.

Giving random 3rd party applications access to your whole computer feels like
a relic from another age.

~~~
derefr
The MAS is in a strange place of having to push out _some_ stuff (e.g. Xcode)
that _is_ entitled to mess around with your system, for necessary reasons
(e.g. to install a debugger.)

There only a few ways they could have worked around this, and I don't blame
them for their choice:

• Require apps to sandbox themselves to the degree they're able, and then
request "entitlements" for anything they need to do outside of the sandbox.
Use entitlement list on app submission to guide the app review process.

This is what Apple chose; it's not very costly _for them_ , since they were
_already doing_ review on app submission, and the entitlement list can
actually make review _faster_ than before, since it gives each review-
iteration more of a clear picture of the app's architecture.

• Divide apps into "system" vs. "pure-userland" apps. Run all "pure-userland"
apps in containers. Optionally, require that "system" apps are as minimal as
possible—more like "system extensions"—and set it up so that most "system
apps" end up as "system extension" \+ "pure-userland app" pairs that use IPC
to coordinate, where one has some sort of official UX to trigger the app store
to download+install the other. Require, additionally, that the "pure-userland
app" still retains some sort of usefulness even without the system-extension.

This is a cool solution, and the one I'd expect Linux (or Android) to use in
the future, pairing "app" packages with "system" packages. It can be nearly
completely automated with little-to-no review required. But it is very
restrictive: either you're containerized or you're not. The components that
require elevation can't benefit from isolation at all. There's no partial
sandbox in the sense of Chrome's "even if you get write-memory access to the
browser, you're not getting anywhere" sandbox.

• Do what iOS does: have only pure-userland "apps", but where some apps can
prompt you to install _configuration profiles_ , that then apply a layer of
(reversible-by-uninstalling) changes to the OS, which the app can rely on.
This is how e.g. the MaaS360 app "takes over" and manages corporate iOS
devices.

I could see OSX being written this way. OSX _has_ configuration profiles(!),
though they're not used for nearly anything _other_ than enterprise management
(and, I think, the recent-ish OSX beta program.) Personally, I think they're a
cool system, a bit like if the Windows Registry was composed in terms of
immutable, introspectible patch-layers rather than being a single mutable
tree. (By analogy: applying a configuration profile is like applying a filter
in Flash or Fireworks, while installing a .reg file is like applying a filter
in Photoshop.)

The flaw with configuration profiles is that you have to anticipate every way
an app might want to modify the system, and provide an API to request that the
system modify itself in that manner instead. You might be able to anticipate
e.g. MacFUSE's desire to install filesystem drivers, or f.lux's desire to play
with screen gamma, or even Synergy's desire to take over your input device
handling. But you'll inevitably miss things, like e.g. Dropbox's desire to
inject a shared object into the Finder that overrides some of its exported
functions with new ones.

------
bobinator606
Amazing that the only contact from Adobe mentioned in this blog post is from
their PR folks

>At 12PM I received emails from Adobe’s PR who wanted to make sure we were on
the same page with how we were addressing the issue

------
mikeash
Has anyone figured out the reason for this bug? What was this "delete the
first directory" code actually supposed to do, and how did it fail? I can't
figure out any plausible scenario.

------
joosters
I'd guess that the cause is very similar to the Steam bug that also deleted
random files. Probably a shell script with a glob like '$SOMETHING/*' where
the $SOMETHING variable was accidentally unset.

------
userbinator
Some previous discussion on this most unusual bug here:
[https://news.ycombinator.com/item?id=11090630](https://news.ycombinator.com/item?id=11090630)

------
glasz
all because adobe's shit is running with privileges it doesn't need.

~~~
Animats
Exactly. People get "Adobe Creative Cloud" because it's the only way to get
Photoshop, Illustrator, and a few other related tools. None of those tools
need root privileges. But Adobe, in their arrogance, think they have the right
to take over the user's machine, run tasks in the background, communicate with
servers, and do other things they have no business doing.

------
makecheck
This is exactly the kind of mistake that a sandbox was designed for.

In a sandbox, _by default_ your software can ONLY write to a specified place
like the "/some-random-globally-unique-string/yourapp/yourcrap" folder. Then,
guess what: the _most_ that "rm -Rf /" can do is obliterate your own stupid
app's files. And, if you further compartmentalize features, it may only damage
a segment of your app.

Adobe's still stuck in the decades-ago software development model of "well,
guess I need full root permission" with no thought invested beyond that. No
wonder Apple further locked down root folders in El Capitán so that you now
need to restart from a special partition and essentially invoke "sudo --act-
from-god --yes-I-really-know-what-the-hell-I-am-doing --no-really-yes-I-do
--yes-really" to remove the protections from most root folders.

------
joch
Is there a viable alternative to Lightroom? It's the only reason I'm stuck
with Creative Cloud.

~~~
perardi
I've heard good things about Capture One from PhaseOne, especially in regards
to the RAW rendering. I'm still on the Lightroom bus myself.

~~~
joch
Thanks for the tip! Looks like they have a perpetual license. :)

------
Tempest1981
Nice write up. I wish they wouldn't use light grey text, however. Form over
function?

Preventative steps: [https://backblaze.zendesk.com/entries/98786348--bzvol-is-
mis...](https://backblaze.zendesk.com/entries/98786348--bzvol-is-missing)

~~~
atYevP
We've gotten one or two mentions of this before, but our designer really likes
it. Creative types...what are you gonna do?

~~~
Tempest1981
I'm guessing your unsympathetic "creative types" have nice Retina displays
where the thin "Open Sans" font is more readable, even at #4D4D4D. If I switch
to Lucida Grande, #4D4D4D is OK.
[http://contrastrebellion.com/](http://contrastrebellion.com/)

~~~
atYevP
You'd be right!

------
tedmiston
Is there really no official press release from Adobe?

~~~
atYevP
Well, they did release this official blog post ->
[http://blogs.adobe.com/adobecare/2016/02/12/creative-
cloud-d...](http://blogs.adobe.com/adobecare/2016/02/12/creative-cloud-
desktop-on-mac-update-issue/)

~~~
tedmiston
Hey, it's something. Even if this is a blatant lie:

> In a small number of cases, the updater may incorrectly remove some files
> from the system root directory with user writeable permissions.

------
itsthisjustin
I got super lucky and happened to see the original tweet about this.
Immediately ran the terminal commands.

------
tedmiston
__Uncheck__ "Always keep Creative Cloud desktop up to date"

Ahh, much better.

------
Grue3
Yet another reason to use GIMP: it doesn't randomly delete your files.

~~~
curt15
GIMP will become a more viable option once they implement non-destructive
editing.

~~~
Grue3
Considering Photoshop just erased an entire folder of files, it can hardly be
called non-destructive. You're basically at the mercy of Adobe to not mess up
and not delete everything when "the cloud" updates. The workflow being
slightly more convenient doesn't matter at this point.

~~~
curt15
So to supposedly avoid one-off bugs (which can be entirely mitgated with
proper backups), graphics professionals would rather suffer every day?

