
Apple Developer Center still down - vinhnx
http://www.marco.org/2013/07/21/adc-downtime
======
nwh
My money is personally on massive, unrecoverable data loss.

If you've ever poked around with the way that Apple's website works, you can
see that the entire place is a huge mess. There's old servers running ancient
(pre-2004) perl scripts alongside the brand new iCloud gear. I can't imagine
how the authentication for AppleID is working as login details still work on
the ancient pages (think pinstripes and glassy buttons). Depending what URL
you hit, the webserver is using php3, php4, perl, python or maybe WebObjects
(java).

At one point I wrote a scraper that was targeting one of their product pages,
and kept getting random, unexplainable results. It turned out that one of
their product areas was behind a round-robin load balancer, with three
completely different apache versions on each server. The page was dying on one
but not the other two. In the end I just had to repetitively scrape until I
hit a good response.

Even the domain for the "maintenance" page for the developer section is
telling. It's just a broken template system regurgitating a bit of the
homepage.

[http://devimages.apple.com/maintenance/](http://devimages.apple.com/maintenance/)

[http://devimages.apple.com/](http://devimages.apple.com/)

Truly a hacked together system. Some engineers at Apple must be having a truly
awful weekend, no matter the cause and solution.

~~~
saurik
Interesting; the [http://devimages.apple.com/](http://devimages.apple.com/)
URL seems to return the underlying contents of the file, bypassing the SSI
(I'm so happy they use SSI... seriously: I love SSI ;P). You can then see the
raw <!--#include virtual=""\--> directives, and pull the individual parts. (It
isn't quite then fair to say that it is a "broken bit of their template
system"; it is more that it is a poor way to setup a static large-file caching
endpoint, and may itself lead to a security vulnerability. To be clear: that's
probably worse ;P.)

~~~
legutierr
Might I ask what vulnerabilities exist for SSI, and why you describe SSI as a
"poor way to setup a static large-file caching endpoint"?

I haven't used SSI before, but after researching various technologies, it
looks like a solution for my own caching use-case, and I am planning on
implementing it. I realize that it's a rather old technology, but as long as
there's nothing _wrong_ with it, I think it could work for me.

Is there something _wrong_ with it? The only thing that I have seen is that
the "exec" command is dangerous, but NGINX doesn't seem to implement it.

~~~
saurik
I did not describe "SSI" as a "poor way to setup a static large-file caching
endpoint"; that description was for the way they setup "devimages." to overlap
with the content for "www.".

Let's imagine that the code in question was not SSI, but instead PHP. You
might have this running behind a copy of Apache and mod_php. Your website
includes both code and images.

This website runs like this, but maybe the way you configured Apache is too
slow to handle downloading all the images: these large-file static assets need
to be handled by a different system.

So, you install a copy of nginx and point it at the same folder. You give this
copy of nginx a new hostname, such as "devimages.example.com" as opposed to
the Apache "www.example.com".

Problem solved, right? No. Now if you go to
[http://devimages.example.com/index.php](http://devimages.example.com/index.php)
you get access to the source code to the website (which is generally
considered a big security failure).

That seems to be what's happening here: the "devimages." hostname is
configured to bypass the SSI parser, and you are getting to see how the page
is composited. This could be a problem.

However, it also might not be a problem: it largely depends, which is why I
was just idly speculating; it certainly _feels wrong_ , though. As for SSI? I
was serious: I _love_ SSI, all my stuff uses it.

------
NelsonMinar
Could you imagine if the Linux Developer Center went down? It'd be nearly
impossible to write Linux software, or download Linux development tools, or
generate a Linux binary that a device was allowed to run. It'd be awful.

~~~
benihana
Could you imagine if people could make the kind of money developing things for
Linux that they do developing things for iOS? It would make comments like
yours kind of funny and relevant.

~~~
ninguem2
Android (Linux) has surpassed iOS in number of users worldwide.

~~~
millerm
More people drive Toyota than BMW. More people eat at McDonalds than Five
Guys. More people stay at a Holiday Inn than The Ritz Carlton. The statement
is worthless as it conveys no meaning, other than a factoid, as do my
statements.

~~~
pavanky
How is that smugness working out for you ? Windows users are fewer than iOS
users. Would you now compare Windows phones to Ferraris by using the same
analogy ?

------
zackmorris
Apple's websites are among the worst corporate sites I've ever used. I don't
say that just to be critical or cynical (ahem), I'm merely pointing out that
they have tremendous room for improvement. As opposed to say php.net or
stackoverflow.com which still give me that fresh air "why can't all sites be
like this?" feeling every time I visit them.

Most of the time when I visit Apple's sites now, I just assume that what I
need to find will either be buried in a convoluted maze, or simply won't
exist, and I'll find myself on a "this has been deprecated" 404-style page
which takes me someplace only loosely related to what I was looking for. So I
end up back at google to try to find a copy of the information either cached
somewhere or offsite.

Simply being an Apple developer is a chore. Keeping up with yearly certificate
expirations is taxing when you are contracting for several clients. And they
never really worked out an easy way to allow several developers to share
certs. I just assume now that the other developers will invalidate whatever
shared cert I made.

The situation is bad enough, and exacerbated by Apple stubbornly refusing to
see the flaws, that I wish a startup would encapsulate the friction and just
take care of all the minutia for me. I should never have to personally deal
with provisioning. Anything short of a one click submission to iTunes Connect
is reminiscent of all the TCP/IP details that we used to have to put in our
modems in the dialup days, when all that should have been required was a phone
number and passcode. I can't gently forgive them for it. So I think this
downtime could be a wakeup call for them that the inefficiencies in their
system are even costing them now.

~~~
briandear
Have you spent any time on the Verizon or Sirius XM sites? How about the
average multinational insurance company? "Among the worst corporate sites" is
a huge leap, considering the profound amount of crap. Spending 5 minutes using
the average electronic medical records (EMR) web application would be enough
to make your hair fall out. Forget about interoperability of EMR formats. It's
a mess. I'm not defending Apple at all, I've spent many nights throwing
iPhones over the provisioning and certificate process, but to me, the outcome
of surviving that process is that I'm making money from the apps, so it's not
ideal, but at least my iOS users are actually paying me money, as opposed to
Android users that, as a group tend to expect something for nothing.

We ought to also look at the inefficiencies of the Android system while we're
here. 35% of users are still on Gingerbread! You can't really expect to earn
money as a developer if you ignore 35% of your target market, with iOS we have
over 90% running iOS 6 and only about 6% running Android 4.2, which means that
developers can't take advantage of a new Android feature without leaving
behind the majority of their market, or creating multiple versions and then on
top of that, having to test on a myriad of different hardware configurations.
The fact also remains that Android users are generally cheap -- they don't
like to pay for apps. So you have a highly fragmented OS environment, coupled
with a user base that, in generally spends much less that iOS users and, on
top of that, you have a royal mess in the copy-protection scheme used in
Android -- which had to be disabled in Jelly Bean due to it breaking apps.

Of course, the security in Android is top notch! Great job on that Google.

If we want to talk inefficiencies with Apple, we certainly can, but to ignore
the Google mess is disingenuous.

As far as the effect of Apple's "inefficiencies," is it really affecting them?
Are people still buying apps and computers?

My last point is that the assertion of Apple "stubbornly refusing to see the
flaws.." That's interesting, because unless one works for Apple at a level
high enough to be involved in the conversation, that suggestion is merely
conjecture. That's right up there with the pundits being disappointed that
iWatch has been delayed, despite having no proof or any acknowledgment from
Apple that even such a product exists.

There's an easy solution to not having to deal with Apple's "inefficiencies"
\-- don't deal with Apple and go make your money with all of the people paying
money for Android apps. Or, respond to one of the hundreds of job listings at
Apple and do something about it.

~~~
ludoo
Given that iOS users worldwide are less than 1/3 of Android users, ignoring
Gingerbread would still leave you with roughly double the userbase...

I completely agree with all your other points though. :)

------
qnk
What's even worse is that we, the developers, their customers, will never get
to know what happened. Because that's the Apple way.

I'd really love that Apple proves me wrong on this one and comes clean on the
problem, the cause and prevention measures being put in place so this won't
happen again, whatever it is.

~~~
general_failure
Nope they won't say a thing and they will get away with it. Oh the things
apple devs have to pout upon with.

In another world HN would be up in arms about how to manage downtime, how this
is u acceptable and how they are going to switch to somebody else immediately.
But not for apple because they have no choice...

------
kailuowang
I have received 6 "How to reset your Apple ID password" Email from Apple
during the last couple of days, none of which was triggered by me. Could this
be related?

~~~
tsenkov
This makes our best guess to be "security breach".

~~~
rimantas
I receive similar emails from Gmail now and then. Does not me think that Gmail
had been compromised.

~~~
MAGZine
right, but we don't have multiple people all confirming GMail password resets
while GMail just so conveniently happens to be down.

------
esalman
Just received following email- confirmed breach:

Last Thursday, an intruder attempted to secure personal information of our
registered developers from our developer website. Sensitive personal
information was encrypted and cannot be accessed, however, we have not been
able to rule out the possibility that some developers’ names, mailing
addresses, and/or email addresses may have been accessed. In the spirit of
transparency, we want to inform you of the issue. We took the site down
immediately on Thursday and have been working around the clock since then.

In order to prevent a security threat like this from happening again, we’re
completely overhauling our developer systems, updating our server software,
and rebuilding our entire database. We apologize for the significant
inconvenience that our downtime has caused you and we expect to have the
developer website up again soon.

------
undoware
Let us compare:

World's premier closed-source shop: (presumably) gets hacked; goes down; stays
down.

Github, Rubygems, Linux Kernel, etc. get hacked; restore from SHA256SUM'ed
backups; keep moving.

Turns out what used to be called "hobby" projects matter, because code made
without love has a smell, and no one does a "hobby" for anything but.
(Remember the 'ama' in 'amateur') (ok ok so kernel.org was down for a while.
But remember all the heavy lifting done by git to keep those commits clean. It
was _just_ the server, not the data.)

~~~
angersock
You probably have better developers pushing live code over at those other
places, I'd bet.

------
Jgrubb
This reminds me, yesterday i got an obviously spoofed phishing email from
"apple" telling me to reset my passwords and reenter my CC info. Anybody else
get that?

[http://imgur.com/hyta4bC](http://imgur.com/hyta4bC)

~~~
rossjudson
It never ceases to amaze me that such emails are carefully constructed in the
graphical sense, but miserable failures in the grammatical.

~~~
ams6110
The British spelling of "apologise" is also a giveaway.

------
atgm
I just went to check on the dev center and got this e-mail:

Last Thursday, an intruder attempted to secure personal information of our
registered developers from our developer website. Sensitive personal
information was encrypted and cannot be accessed, however, we have not been
able to rule out the possibility that some developers’ names, mailing
addresses, and/or email addresses may have been accessed. In the spirit of
transparency, we want to inform you of the issue. We took the site down
immediately on Thursday and have been working around the clock since then.

In order to prevent a security threat like this from happening again, we’re
completely overhauling our developer systems, updating our server software,
and rebuilding our entire database. We apologize for the significant
inconvenience that our downtime has caused you and we expect to have the
developer website up again soon.

------
Tloewald
Amazon bas been down for as much as a half day in recent memory — and not dev
stuff but their customer-facing money-making site, and mid-week too.

~~~
mullingitover
Apple's customer-facing store was down last night for several hours.

~~~
Terretta
Doesn't matter. People aren't going to decide, gee, I actually wanted a
Toshiba.

~~~
MAGZine
people also aren't going to decide 'gee, i'm going to go to x store and pay
more for something' anymore then they'll go to BB to buy their apple product
of choice.

------
slowdown
Why is a mere unfounded speculation not backed up by any facts written to
garner pageviews featured on the frontpage?

~~~
freehunter
Because there are no facts, Apple hasn't said a word. The alternative is that
we ignore this. At least with this article (which no one has to read), there's
a place for HN Apple devs to talk amongst each other.

~~~
hrktb
It goes very deep in meta territory, but hey, it's not as if there were any
fact intersting to discuss.

I remember a lot of discussions a year or two ago when marco was very vocal
about the reasons he didn't have comments on his blog, and would prefer to
have all of these on HN for e.g.

 _slowdown_ 's comment goes the other way round, on why HN should care about
commenting on marco's blog. There is another thread [1] with 138 comments and
more quality opinions and speculations than anyone should need, Marco's
speculations are also represented in the top comments. Agree or not with he's
stance, it's a valid point IMO.

I think it's fun to see this kind of reaction on why some contents should be
commented or not, and the motivations to push a site or another.

[1]
[https://news.ycombinator.com/item?id=6071233](https://news.ycombinator.com/item?id=6071233)

~~~
freehunter
I've been checking on that post since it first came up looking for anything
new, but old HN posts get to be hard to read for new stuff pretty quick,
especially with no way to collapse a thread. If there was anything new, I'd be
hard pressed to actually find it.

That thread is quickly sliding down the list and is already off the front
page. I don't think a new one is out of the question.

~~~
shabble
[https://userscripts.org/scripts/show/138037](https://userscripts.org/scripts/show/138037)

A way to navigate to new _unread_ entries would be nice.

Sometimes I really wish NNTP wasn't quite so dead.

------
allwein
Apple is set to report their earnings on Tuesday, July 23rd. If this isn't
back up by Tuesday evening, it's going to be a very interesting earnings
conference call.

------
peterkelly
I would be _very_ interested to see what the implications would be if this had
happened to iCloud.

I'm sure Apple are as unhappy about whatever has happened as the rest of us
(likely much more so), but I think at least some communication from them about
it would be in order.

------
tsenkov
Instead of fighting about which platform is the best, I hope all of you will
agree that Apple should not leave us in the dark for 15min, let alone 3 days,
or not even "post mortem" to answer the question "what happened?".

------
navs
Apple just released an update (developers should be getting it in their inbox)
but still no ETA on when it'll be up. Very unfortunate for those of us waiting
to release apps before August.

------
ttflee
> or device provisioning and certificates (potentially very profitable)

Well, an `App'(, the name of which I would not like to identify, ) which was
installed using Safari exploits and whose intended use is to help users search
and install apps free of charge from AppStore, is still up and running.

I guess there exist various exploits up and down the App Store chains, till
now.

------
xixora1
So… Did everyone get the email saying they had an intrusion and they're
rebuilding the system?

~~~
vinhnx
yep, I got it a few hours ago.

------
tater
Forstall has the backups.

------
trackztar
If it weren't Sunday, I wouldn't speculate, but it is. My 2 cents: iOS7
updates galore. Just imagine everything that could be overhauled. Even the old
gray textured background we see is not very much like iOS 7.

iOS is in a big transition here. You can't even update apps at all unless you
now include 'widescreen' support, for example. If you don't, iTunes reports an
Invalid Binary (no default 586 image, etc).

So I would guess a huge overhaul, and typical Apple, is taking care of that
vs. arguing or commenting on theories.

~~~
grey-area
_So I would guess a huge overhaul, and typical Apple, is taking care of that
vs. arguing or commenting on theories._

A design overhaul wouldn't require the dev centre to go down at all - they
could just prepare all the assets etc on testing servers and switch them over
when they are ready.

Given the normal warning given on any maintenance, and the obvious negative
consequences for Apple's business of any extended outage (extended in this
case being over a day or so), this is unintended, and most likely caused by a
security breach. I think we can safely rule out a design overhaul unless Apple
are incredibly incompetent.

NB itunesconnect is still up (the bit which deals with app upload, itunes
store metadata etc), only the dev center - dealing with certs/device
registration/distribution etc - is down.

~~~
jordanthoms
Keep in mind, this is the company which takes their whole store down when they
need to add new products to it.

~~~
deletes
Did you mean when they have their WWDCs? In that case it might be a marketing
reason.

~~~
ben1040
They take it down during WWDC keynotes and other major announcements, yes.

But there are other instances where the store will go down outside of those
periods. Sometimes it's for maintenance, but more often than not a new or
updated product appears (e.g. a product line gets a speed bump across the
board).

------
coldtea
Gee, thanks Marco, we would have never known ourselves...

~~~
nicholassmith
That sounds like "I just read the title so I could post a snarky remark". He
does put some suggestions forward of what he thinks could be the root cause.

~~~
coldtea
I read the whole article. The suggestions cover all the cases one would that
thought himself.

He is being Captain Obvious.

Including the gem to "set some time aside" as developers, because we might
have some work to do after the site comes back up.

Everything in the comment is what men wiser than us called "idle speculation".

~~~
TwistedWeasel
If you don't like reading idle speculation then you should probably avoid 95%
of tech blogs.

~~~
FrankBlack
I think that comment is Quickmeme-worthy. [http://www.quickmeme.com/Captain-
Hindsight/](http://www.quickmeme.com/Captain-Hindsight/)

