
Facebook Down, Like Buttons Vanish, Internet Implodes - aresant
http://techcrunch.com/2010/09/23/facebook-down/
======
antidaily
They closed shop after DHH's scathing criticism.

~~~
alphadog
DHH's fantasies realized.

~~~
scorpion032
More like zuckerberg making a statement to detractors, how important they are.

------
bl4k
Well somebody has screwed it up on their end. Only the www record is not
resolving, a lot of the others are, for eg:

<http://developers.facebook.com/>

    
    
      $ nslookup facebook.com ns1.facebook.com
      Server:	ns1.facebook.com
      Address:	204.74.66.132#53
      
      Name:	facebook.com
      Address: 69.63.189.11
      Name:	facebook.com
      Address: 69.63.181.12
      Name:	facebook.com
      Address: 69.63.189.16
      
      $ nslookup www.facebook.com ns1.facebook.com
      Server:	ns1.facebook.com
      Address:	204.74.66.132#53
      
      Non-authoritative answer:
      *** Can't find www.facebook.com: No answer
    

(YMMV based on local cache etc.)

~~~
klochner
I imagine that would be a relatively easy fix, more likely they took the
record down to debug internally without getting flooded with requests.

~~~
bl4k
It looks like you were right - they dropped that record to stop the requests
coming in

------
aresant
It's hard to overstate the damage this is causing for merchants that
prominently display FB like on their pages - lots of big ugly blank spaces /
errors out there.

Realize this is temporary but shows why if you install any external vendor
tool as a part of your site make contingency plans that switch on errors.

~~~
byoung2
Or people who rely on Facebook for shared login...that must be down as well.

~~~
joey_bananas
Well frankly, If you _rely_ on some other business for such critical
functionality, you deserve to lose.

~~~
kgermino
Unfortunately its difficult to avoid relying on other businesses for crucial
systems.

Example: Payment Processing, try building a business, especially an online
one, without depending on Paypal, or at least Mastercard/Visa.

Sure you can reduce it by say not having FB Connect as the only log-on option
but the simple fact is that you will almost always have to depend on other
businesses, the best you can do is diversify and hope for the best.

~~~
byoung2
Exactly...people made this same argument a few years back when Amazon S3 went
down for a few hours. Just have a few different options, and the chance that
they will all be down at the same time is pretty slim.

~~~
joey_bananas
If you have a few options, then you don't _rely_ on a single company, do you?

~~~
kgermino
Yes and No. If you have a few options than your entire business won't come
grinding to a halt but if half of your customers choose to use FB Connect as
their login option than they likely don't care that FB's down, or that people
with email logins can still access your site. To 50% of your user-base your
site has failed and so in that sense I would say that you still rely on FB for
your business.

------
spolsky
cauliflower farms everywhere are withering for lack of water.

~~~
danielnicollet
Please don't downvote me. I am not a native speaker. But I still don't get it.
Someone care to explain ;-)

~~~
patrickmclaren
FarmVille.

~~~
wazoox
As a non facebook user, I didn't get it either... :)

~~~
athom
I use Facebook (sometimes), played FarmVille and Farm Town up until this
January, and I STILL didn't get it right off. Guess pulling out of those games
was a good idea after all. :)

~~~
wazoox
My co-worker just told me his mother is level 137 at Farmville :) Apparently
she spends 6 hours a day playing it...

------
mjgoins
This affects me in no direct way. I officially have a good life.

~~~
ElbertF
Same here, I already have most of Facebook's domains in my hosts file. I
haven't seen a Like button in ages.

------
rwhitman
And that, my friends, is why its risky to build your business on top of FB
Graph API.

In terms of reliability, FB is not twitter, but it certainly isn't google
either...

~~~
ryanwaggoner
I came here to post this same thing, but I just started thinking about it.
When was the last time FB went down like this and took a bunch of core
functionality down with it? Has it ever happened? And how long are they
down...30 mins? Is that 30 mins of downtime every few years really more
damaging than that the (potential) value that integration with the graph API
or FB Connect can bring? Yes, you look like an idiot to a few users, but maybe
you have 100x as many users as you'd have otherwise. I think the real reason
not to build your business on the FB API (or Twitter or whoever) is strategic,
not because of potential downtime.

~~~
rwhitman
You're right, I should have phrased it "And that, my friends, is _one of the
many reasons_ its risky to build your _entire_ business on top of FB Graph
API."

Just went through a nightmare building a product entirely on top of fb graph.
FB fails, a lot. And not just in terms of downtime, in terms of inconsistently
implemented technology, poor documentation, lack of transparency, inconsistent
policies etc.

Not saying don't use FB ever, just that it's risky.

~~~
ryanwaggoner
Oh, don't get me wrong. I won't do FB apps for clients anymore, despite the
high demand and great pay. It's just way too stressful to have the clients
always angry at me because of the platform's shortcomings.

~~~
rwhitman
Amen! I tell ya, I have had so much stress over the last few months as the
direct result of this. Well put.

------
djcapelis
Their SSL site still continues to simply not work:

    
    
      www.facebook.com uses an invalid security certificate.
    
      The certificate is only valid for the following names:
      a248.e.akamai.net , *.akamaihd.net  
    

This is the biggest problem with the SSL PKI. It is partially responsible for
the lackluster deployment of SSL. We need to switch to a TOFU model
supplemented by a more nimble version of the current PKI for first connect
acceptance.

You would not believe how often I run into problems just trying to use SSL on
_major_ _sites_. If you can't trust a large organizations like facebook and
akamai to mess up SSL support during problems, you simply will never get it
deployed well on all the rest of the sites.

SSL is simply a poor user experience. Twitter's SSL site often goes down more
often than the main site too. SSL users get a second class experience and that
is the most damming thing I can think of for a technology where deployment is
so critical.

~~~
lanstein
Excellent points, but please be sure to simply never split infinitives. ;)

~~~
lukev
I shall continue to boldly split all the infinitives I please. We speak
_English_ , despite the best efforts of medieval Latin grammarians.

------
chrisgoodrich
Everyone should calm down and realize that Facebook being down for a few hours
is not going to kill anyone's business. It may cause trouble and disruption,
but not anything more intense than a daily commute in Los Angeles.

That being said, this makes me question Facebook's ability to deliver on the
"Social Graph" promise. If they truly want to build a completely social
internet, this kind of downtime can't happen.

~~~
Terretta
> _"If they truly want to build a completely social internet, this kind of
> downtime can't happen."_

Why not?

Even commerce sites such as Amazon or Ebay have downtime. Studies demonstrate
that unless you're selling a perfectly commoditized generic product,
obtainable instantly from multiple sources, downtime doesn't materially impact
revenue.

Users seem to manage to "queue up" their intended actions and do them when a
site is back. And it's not like Facebook's users can gravitate en masse to
anywhere else.

The founding principle of the internet is not "5 nines" uptime for all
components, but resiliency. Switch off your mail server for a few hours;
you'll still get all your mail when you turn it back on.

"Kids these days" build apps and APIs (and sites that rely on APIs) that are
much more brittle, because today's developers are spoiled into thinking
communication conduits are reliable. They're not.

The fabric fails, so your site, your apps, your protocols, should fail
gracefully -- and recover gracefully later.

------
smackfu
I like how their API response time chart intentionally doesn't have a Y scale.
But clearly it is busted:

[http://www.facebook.com/developers/chart.php?type=at_total_t...](http://www.facebook.com/developers/chart.php?type=at_total_time)

~~~
someone_here
Oh dear, DNS error on that link.

~~~
alphabeat
As somebody pointed out, the www record was removed probably to stop the
barrage of requests coming in.

------
dansingerman
As an aside I notice that 'Facebook' is not a trending topic on Twitter. Seems
the term must be blacklisted in some way along with all the swears and other
undesirable terms.

Which seems a bit underhand by Twitter.

Edit: I notice that 'Facebook isnt working' has appeared now. I've blogged
about this issue here: [http://blog.dansingerman.com/post/1174729229/twitter-
mostly-...](http://blog.dansingerman.com/post/1174729229/twitter-mostly-a-
force-for-good) (if anyone cares)

~~~
bl4k
during one of the twitter hacks the blacklist application was accessed and
facebook was on the list (as was google and a bunch of other brand names)

~~~
Astro9k
Wow. So twitter censors disparaging comments about companies? Wow.

~~~
seldo
More likely that terms that are continuously mentioned in huge numbers (and
therefore don't count as "trending") are omitted.

~~~
_delirium
There are more principled ways to do that than a blacklist, though, like
looking at current frequency compared to typical frequency. That way even
something commonly mentioned could still genuinely be trending if it gets
mentioned much more than usual on a particular day.

(It's possible there are scale-related reasons that make this infeasible,
though.)

~~~
smackfu
Just not tracking the frequency of the most common words that will never be in
the trending topics is probably a big performance win. I'm sure there is an
additional blacklist that includes stuff like "and" and "the".

~~~
bruceboughton
You know you've made it when your brand name is a stopword.

------
bbuffone
In researching web performance DNS configurations issues are common problem.
It is usually related to www.foo.com and foo.com being configured differently.
This seems to be at a high level the issue with facebook.com. if you go to
"facebook.com" the DNS will resolve correctly, but it will redirect you to
www.facebook.com. I guess they could stop the redirection and fix the problem
immediately, or wait till the sorry.ak.facebook.com.edgesuite.net is fixed.

=====www.facebook.com

new-host-3:dpu2 robertbuffone$ dig -trace www.facebook.com ;; Warning,
ignoring invalid type race

; <<>> DiG 9.6.0-APPLE-P2 <<>> -trace www.facebook.com ;; global options: +cmd
;; Got answer: ;; ->>HEADER<<\- opcode: QUERY, status: NOERROR, id: 19626 ;;
flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION: ;www.facebook.com. IN A

;; ANSWER SECTION: www.facebook.com. 0 IN CNAME
sorry.ak.facebook.com.edgesuite.net. sorry.ak.facebook.com.edgesuite.net. 0 IN
CNAME a1030.g.akamai.net. a1030.g.akamai.net. 4 IN A 63.84.59.59
a1030.g.akamai.net. 4 IN A 63.84.59.10

;; Query time: 10 msec ;; SERVER: 198.6.1.142#53(198.6.1.142) ;; WHEN: Thu Sep
23 16:57:29 2010 ;; MSG SIZE rcvd: 144

========facebook.com

new-host-3:dpu2 robertbuffone$ dig -trace facebook.com ;; Warning, ignoring
invalid type race

; <<>> DiG 9.6.0-APPLE-P2 <<>> -trace facebook.com ;; global options: +cmd ;;
Got answer: ;; ->>HEADER<<\- opcode: QUERY, status: NOERROR, id: 19984 ;;
flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION: ;facebook.com. IN A

;; ANSWER SECTION: facebook.com. 352 IN A 69.63.181.12 facebook.com. 352 IN A
69.63.189.16 facebook.com. 352 IN A 69.63.189.11

;; Query time: 13 msec ;; SERVER: 198.6.1.142#53(198.6.1.142) ;; WHEN: Thu Sep
23 16:57:44 2010 ;; MSG SIZE rcvd: 78

------
joebasirico
When I heard FB was down I went over to twitter to see how many tweets this
was generating, I was surprised to see the first refresh of "XXXX more tweets
since you started searching."

I decided to start keeping track and I made a graph of the difference between
refreshes. You can see my graph here. It seems like in the course of just a
few minutes (I was probably conducting my little experiment for 10 min?) you
can see the frenzy gaining momentum then it seems to die down until only
hundreds of people are talking about it per 15 second refresh.

A little bit about the graph, each tick mark on the bottom represents a
refresh (15 seconds or so I think). The Y axis is the difference between
refreshes.

Graph: <http://drp.ly/e3oH>

------
tfh
It's a funny coincidence that this comes right after the recent facebook
valuation news.

<http://en.wikipedia.org/wiki/Evil_eye>

------
yarapavan
Current Status: API Latency Issues We are currently experiencing latency
issues with the API, and we are actively investigating. We will provide an
update when either the issue is resolved or we have an ETA for resolution.

------
stevefink
DNS issues are the cause for so many outages by a myriad of prestigious web
shops. It's incredible how often overlooked something so pivotal to your
infrastructure goes without being properly monitored. I have seen entire
businesses suffer hours of downtime because their monitoring systems would
query an authoritative name server from RFC1918 IP space - but not the same
from an externally visible address. An NS not responding to local queries can
be just as detrimental internally as they are externally because of how web
services are architected, obviously. In their defense, Facebook rarely incurs
outages of this capacity, and I can bet it won't happen again and someone is
getting seriously reamed. They can and will happen to the best of them, no
matter how many MIT PhDs you have designing your systems.

------
ljf
maybe its the 1 billion like buttons they serve a day that tipped them over
the edge. ;)

~~~
noodle
perhaps the Stuxnet virus was built to target facebook? ;)

~~~
byoung2
You don't get to 500 million friends without making a few enemies

~~~
judofyr
Wait, doesn't Facebook already have 500 million enemies?

------
jacquesm
Don't place 3rd party javascript on your pages if you don't really need it.

~~~
natrius
Facebook is our biggest traffic source after Google. I'm inclined to forgive
them for occasional outages.

~~~
jacquesm
I did say:

> if you don't really need it.

In your case, you obviously do.

~~~
natrius
I don't think we're uncommon. Most sites that I see plastered with Facebook
embeds are publications, and I'm sure they all see results similar to ours.
Facebook is RSS for the masses.

------
daten
The significance of the "like" button missing from other sites is lost on me.

I've been using the Application Boundaries Enforcer feature of no-script in
Firefox to block facebook content when I'm not visiting a facebook site
directly.

    
    
      Site .facebook.com .fbcdn.net
      Accept from .facebook.com .fbcdn.net
      Deny
    

This is motivated by privacy concerns of Facebook's ability to track which
partner sites I visit.

------
parbo
First I just had an empty feed, then I got the feed back but posting links and
comments timed out. Then it worked for a minute, and now I get the DNS error.
Seems like this affects several of Facebook's applications.

------
zppx
I experienced a facebook outage yesterday afternoon (-03:00 UTC) although the
DNS resolved the facebook I could not ping their IP addresses. As you can read
in facebook's twitter timeline.

------
Tichy
At least Hacker News still works. All is well.

------
grandalf
Is this a coordinated blackhat attack with DHH's PR attack intended to benefit
some investor somewhere? (joking)

------
code_duck
It's odd, I've been hearing about this all day but I have had no problems at
all contacting facebook.

------
doron
IT Dept. around the world will now get washed with calls, on how the internets
dont work.

~~~
nkassis
the BOFH will be making a list of people wasting time on facebook for later
blackmail.

~~~
doron
joking aside, this actually had this effect, as so many pages actually resolve
some components to facebook.com, the slower response was felt on a wide
variety of sites.

------
whereareyou
Aaaaaand, they're back. Down for about 15 mins (from Tucson, AZ).

------
chewbranca
heh... ironically facebook's error rate is at an all time low!!

<http://developers.facebook.com/live_status>

------
KingOfB
The one day I choose to update my FBConnect code!

------
mtrn
Who's next to go down? downrightnow.com?

~~~
rryyan
downforeveryoneorjustme was throwing errors when I tried it:
<http://downforeveryoneorjustme.com/facebook.com>

------
aquarin
But I can still harvest my cows ...

------
akadien
Thank goodness for Ping ...

------
dirktheman
Nice to see the big boys experiencing mundane issues as well now and then...

------
chopsueyar
I demand a refund.

------
ax0n
I [Like] this.

------
9ec4c12949a4f3
In A.D. 2010

War was Beginning.

Internet: What happen?

MarkZ: Somebody set up us the bomb.

FB: We get signal

MarkZ: What?!

FB: Main screen turn on.

MarkZ: It's you!!

DDH: How are you gentlemen !!

DDH: All your Information are belong to us.

DDH: You are on the way to destruction.

------
bond
Rarely use it so I didn't notice...

------
orblivion
I'm sure somebody out there told you so

------
StavrosK
I thought I had a crappy router, I didn't realise it was _that_ crappy!

