
Apple’s fix for corrupt binaries - shawndumas
http://www.marco.org/2012/07/06/apple-repushes-corrupt-binaries
======
justinschuh
Good on Apple for fixing this quickly. Even the best team can let a bug like
this slip through, and the best solution is a fast response. However, I'm
confused by the blog post implying that the solution to unbricking the apps
was somehow novel or praiseworthy. This is pretty much textbook for an auto-
update system. You just bump the revision to force an application update;
there shouldn't be any need to reinstall or muck with the data files. I'd
honestly have been shocked if they hadn't been able to handle it just the way
they did (which would have been noteworthy, but not in a good way).

~~~
thought_alarm
_However, I'm confused by the blog post implying that the solution to
unbricking the apps was somehow novel or praiseworthy._

In the 4 years of the App Store we've never seen a distribution problem of
this magnitude (it would be an absolute nightmare for any devs affected) and
we really have no idea how Apple would respond to such a problem.

We've also never seen Apple unilaterally update a specific group of apps like
this. Many App Store devs likely didn't think Apple would or could take such
action, and those of us familiar with what it takes for a device to accept and
install new bits are now wondering just how this is being done.

Are they manually bumping each affected app's version string? Is there there
some hidden field that forces a device to reinstall the same version of an
app? It's curious.

~~~
pooriaazimi
My theory: there's a version number that devs set on an app, and there's
another version (say an integer starting from zero) that Apple puts on apps.
They bumped their version, leaving app's version intact.

~~~
quesera
I'd say that's a good theory, and if they baked it in from the beginning, it's
a great solution to this sort of problem.

The fact that Apple is so strict about monotonically increasing dotted numbers
for version strings led many of us to believe that they were being parsed for
important things inside the App Store publishing platform.

But maybe they are just part of the UI. Apple is known to have strong opinions
and strict adherence requirements about that, too.

------
phuff
Does anybody have any insight on how a process like this is debugged at Apple?
I've heard, for example, that at Amazon there is a process of blame-finding
after a major issue/outage like this, whereas at Google I hear things are more
post-mortem let's fix the process that led the human error involved in the
outage, rather than blame the person who made the faulty commit.

Anybody know how things work inside Apple's culture?

~~~
oemera
This is Apple culture at its best: No one ever talks about it. All employees
are so loyal to this company that you can't even imagine one speaking about
such insights.

~~~
recoiledsnake
>All employees are so loyal to this company that you can't even imagine one
speaking about such insights.

I think it is more of fear about losing their job rather than loyalty. It is
not uncommon to find that any leak will be tracked down and the leaker
summarily fired.

~~~
snowwrestler
Fear of what, getting fired? Is it hard for Internet-scale application
engineers to get a job in the Valley these days?

Is it really so hard to imagine that employees at Apple feel loyalty to the
company? Employee loyalty is not an uncommon thing.

------
crazygringo
Does anyone know what the actual problem ever was?

~~~
dljsjr
There was an error with the FairPlay DRM signing process that lead to a lot of
app binaries becoming corrupt; people would go to download updates from the
app store, and after doing so would be completely unable to launch the app.
Not even a splash screen or a display of the freeze-dried screenshot from
multitasking. Half a second of black, then kicked back out to Springboard.

Marco did a really good job of cataloging it because Instapaper was one of the
affected applications: <http://www.marco.org/2012/07/04/app-store-corrupt-
binaries>

------
psychotik
How did they fix/change the version string, or did they? I don't see how
simply re-released a previously released version would cause affected users to
update their binaries.

What am I missing?

------
spaghetti
This snafu, the Galaxy Nexus thing and the fact that the simple update to my
highly rated and heavily used iOS app has been "waiting for review" for ten
days made me start Android Dev. Sorry Apple. You took too long. Hire more
reviewers already.

------
thechut
Can somebody please provide some context for this?

~~~
btucker
<http://www.marco.org/2012/07/04/app-store-corrupt-binaries>

~~~
thechut
Thanks! That sounds bad, glad I'm not an iOS developer

~~~
pooriaazimi
Come on... Shit happens. And it was the first glitch in the App Store after 5
years of operations! They've sold 30 billion apps (and updated probably well
over 200 billion).

:D

~~~
mikeash
The first glitch? You're kidding, right?

~~~
pooriaazimi
Well, many, many things suck about the App Store and the submitting process,
but they are not _that_ important. This one really ruined a lot of people's
holiday, tons and tons of angry and 1-star reviews and if I'm not mistaken,
people lost local data. If it was a little more widespread, it could've been a
real fiasco.

