
Twitter Completely Down - Braasch
http://www.isup.me/twitter.com
======
steve8918
I think this is another good example of how we as an industry are still unable
to adequately assess risk properly.

I'm fairly certain that the higher-ups in Twitter weren't told "We have pretty
good failover protection, but there is a small risk of catastrophic failure
where everything will go completely down." Whoever was in charge of disaster
recovery obviously didn't really understand the risk.

Just like the recent outages of Heroku and EC2, and just like the financial
crisis of 2008 which was laughably called a "16-sigma event", it seems pretty
clear that the actual assessment of risk is pretty poor. The way that Heroku
failed, where invalid data in a stream caused failure, and the way that EC2
failed, where a single misconfigured device caused widespread failure, just
shows that the entire area of risk management is still in its infancy. My
employer went down globally for an entire day because of an electrical grid
problem, and the diesel generators didn't failover properly, because of a
misconfiguration.

You would think after decades that there would be a better analysis and
higher-quality "best practices", but it still appears to be rather immature at
this stage. Is this because the assessment of risk at a company is left to
people that don't understand risk, and that there is an opportunity for
"consultants" who understand this, kind of like security consultants?

~~~
frossie
_Whoever was in charge of disaster recovery obviously didn't really understand
the risk._

That's not necessarily true. People don't die when twitter is down, and
whatever twitter's business model actually is, I am not even sure there is a
monetary penalty to them being down (unlike, say, Amazon being down which
results in lost orders). They may have made the calculation that it was not
cost effective engineering-wise to chase that extra 0.001% of reliability.

[Edit: Pedantry shield: Ok, ok, should have said people don't die _because_
twitter is down. Obviously people are dying all the time, and some will indeed
expire while twitter is down].

~~~
InfinityX0
They definitely lose money - their ad model is based on engagement, so if
tweets aren't seen/can't be acted on, money is lost.

~~~
walru
Adversely. Twitter being down causes a stir, which gets people talking even
more. They quickly make up for the lost revenue as a result.

There is no such thing as bad publicity.

~~~
utunga
This is going to lead to a loss of confidence amongst many - investors,
advertisers, partners - who thought that Twitter was finally past its initial
scaling/teething issues. It's not like there is anyone that hasn't heard of
Twitter at this point in time. So yes, yes there can be bad publicity once
something reaches the scale of twitter.

~~~
re_todd
You mean once you switch away from Rails, you don't automatically scale?

------
zupreme
I think that a lot of you guys are confusing "Disaster Recovery" with
"Business Continuity".

Disaster Recovery is a reactive approach. It's what you do to get things back
up AFTER a system or site has failed.

Business Continuity is a proactive approach. It's what you do to ensure that
your critical services will remain viable whenever disaster occurs.

In the cases of Heroku, Amazon, Twitter, and many more, their Disaster
Recovery strategies have been successful. The fact that they came back online
without major data loss is proof of that. Their business continuity
strategies, however, have been found wanting.

~~~
antirez
We don't exactly know if the services troubles in this down times you cite
were caused by a _disaster_ , so maybe even the disaster recovery thing may be
lacking.

------
johnyzee
I hope they write up a post-mortem on the fallout (hopefully it won't be a
post-mortem of Twitter). Those things are always extremely interesting with
big infrastructure like this.

~~~
loceng
I think it's almost a requirement of these large platforms now. If they don't
then developers would lose trust in them; I would anyway.

~~~
hinathan
Hasn't Twitter demonstrated a general disinterest in their developers of late?

------
mootothemax
Always fun when you're developing against an API, and then have to perform a
frantic investigation to work out if your latest code change broke
_everything_... or it's just the API endpoint itself.

~~~
DanielN
Unit test ( as already mentioned) and build up static json/xml/whatever files
to run your code against so that you can integration test your stuff without
integration testing the 3rd party stuff.

~~~
Retric
And hope your static files still match the API behavior when your done...

~~~
timdorr
That's actually a completely valid test to run: what happens when the API's
results change on you? You should test that.

------
CD1212
One great thing about this for me: it exposed a bug in my app which relied on
twitter being up.

~~~
wtetzner
If Twitter never went down, would it be a bug?

~~~
MiguelHudnandez
I had a similar issue in one of my sites. It's an unnecessary dependency and
should be corrected. Maybe not a bug, but a problem.

------
jgrahamc
x is down: <http://blog.jgc.org/2010/10/x-is-down.html>

~~~
kristofferR
Well, it is newsworthy at least.

Sure, being upset/getting angry just because of a little bit of Twitter
downtime is stupid, but that doesn't take away from the fact that one of the
biggest and most important discussion and communication channels the web has
is completely down.

~~~
jgrahamc
I don't believe it is newsworthy. In fact, it's sad that it is being written
about because it shows the utter shallowness of what passes for 'Silicon
Valley News' at the moment. And it's sad to see this as a 'top story' on
Hacker News taking up space.

~~~
willtheperson
I think it's only newsworthy in this space - where, as some of us are the ones
responsible for these systems - we are trying to learn why this happened in
order to prevent it.

You know, learn from others mistakes and all.

~~~
jgrahamc
I agree that learning from why Twitter was down will be interesting and when
that story comes out I hope it will be high on Hacker News. But this sort of
news story about something that's happening right now is symptomatic of the
useless '24 hour news' cycle of noise.

~~~
cryptoz
> But this sort of news story about something that's happening right now is
> symptomatic of the useless '24 hour news' cycle of noise.

The single most important part of the internet is the immediate availability
of news (to me, anyway). I've never heard anyone _complain_ about that before;
why do you think it's not worth knowing and talking about events as they
happen? 'Twitter is down' isn't noise. Years and years ago there were stories
about fire departments (SF I think?) that started using Twitter to send out
fire notices. Here in Montreal, the police tweet very quickly and accurately
about our daily student protests.

Twitter is extremely important to a huge number of people, and when it goes
down, a site like HN _definitely_ should be talking about it. It's big news
and it's almost exclusively relevant while it's happening.

I love the internet.

~~~
jgrahamc
So, here in the UK Twitter is back for me. It looks like I was without it for
about 45 minutes. Call it an hour for a nice round figure.

Think about these two scenarios:

1\. During that one hour you spend your time focussed on talking about this
event as it's happening, speculating, having an emotional response (because
you can't access something you want to and find a group of people experiencing
the same thing and all get together to experience the frustration).

2\. Tomorrow you read a story that says "Twitter was down for one hour
yesterday" with some detail about what happened.

I believe that the latter is preferable. It's more efficient, less emotional
and more useful. The former is the same as watching some 'Breaking News' event
while is happening.

Now imagine that the one hour of downtime happened when you were asleep.
You've missed nothing.

There are two scenarios where this news is important: if your business depends
on Twitter, and if you are trying to assess the reliability of Twitter. The
latter can be achieved by #2 above, only the former needs real-time updates
and that doesn't mean general news reporting just your own monitoring.

~~~
Terretta
As a person whose love of news started at age five scrapbooking newspaper
clippings about Nixon, Ford, and Carter, your point makes me very happy.

The 24 hr news cycle ruined TV news, and SEO has made Internet news worse
(first links win).

------
MattRogish
I just ate lunch. It was sooo good! [Checked in to Shake Shack, NYC]

~~~
hendler
LOL @MattRogish RT I just ate lunch. It was sooo good! [Checked in to Shake
Shack, NYC]

~~~
rhizome
yep @hendler LOL @MattRogish RT I just ate lunch. It was sooo good! [Checked
in to Shake Shack, NYC]

------
leot
Twitter is a protocol masquerading as an app.

~~~
k3n
Conversely, this has always been my complaint against the general hoopla and
bandwagonning around Twitter -- they didn't do anything technically
innovative. When they first started generating buzz, I instantly thought, "Oh,
did they create a new messaging protocol that is universally accessible,
redundant and easy to use? That's killer!" But, that was not the case.

Yes, they had a good idea and executed very well, but as I see it, Twitter is
nothing more than your run-of-the-mill 4chan board. I still don't understand
the draw, but then again, there's a lot of facets of modern society that I
simply have no explanation for (reality tv?!?) and have been better off not
worrying about it further.

~~~
kristofferR
99,9% of people doesn't care at all about how technically innovative something
is, they only care about how useful it is. And Twitter is incredibly useful.

~~~
bithive123
"Incredibly useful" would imply that if you told me about Twitter, I wouldn't
believe you. As much as I wish it were otherwise, it's not that hard to
believe that someone invented IRC over HTTP and gave it a silly name.

~~~
glhaynes
People don't expect "incredible" to mean "not credible" anymore.

~~~
bithive123
That's literally the most awesome thing I've read today, for all intensive
purposes!

~~~
gfosco
..... for all intents and purposes.

~~~
bithive123
Christ, enough with the pedantry.

------
ghurlman
<https://mobile.twitter.com/> is working just fine.

~~~
pors
Maybe it's built only on top of the streaming API?
<https://dev.twitter.com/status>

------
timf
"Today's turbulence explained" was just posted:
[http://blog.twitter.com/2012/06/todays-turbulence-
explained....](http://blog.twitter.com/2012/06/todays-turbulence-
explained.html)

Unfortunately there are no details, it just says "there was a cascading bug in
one of our infrastructure components".

------
aeurielesn
Twitter Status - <http://status.twitter.com/>

No news at the status site either, that beats the purpose of having a
dedicated status site.

~~~
xabi
Twitter Site Issue 17 seconds ago

Users may be experiencing issues accessing Twitter. Our engineers are
currently working to resolve the issue.

~~~
Argorak
My favorite part is the Tweet-Button with zero tweets.

~~~
tg3
It has 56 tweets now...perhaps the outage is somewhat intermittent?

------
gms7777
Twitter is down? Now where am I supposed to complain about twitter being down?

~~~
mkr-hn
<https://plus.google.com/s/%23TwitterDown>

------
mkr-hn
I'm glad this made it to the front page. Is the topic itself newsworthy? Not
on its own. Is all the discussion that's flooding into this thread worth
having?

Yep. Even the subthread from the person complaining that this isn't
newsworthy.

------
revorad
<http://mobile.twitter.com> is working.

~~~
mikecane
Looks like that just went down too, while I was using it!

Edit: Nope. Just slow. My tweet appeared.

~~~
m_pagliazzi
for me it works

~~~
mikecane
What's up with the downvotes?

------
tikitaka
this still applies: <http://blog.pinboard.in/2011/12/don_t_be_a_free_user/>

------
Mojah
If you're debugging webservices that suddenly slow down (timeouts of 10s),
this may be your cause if they depend on s.twitter.com, search.twitter.com or
api.twitter.com.

As a workaround for those systems, add s.twitter.com, search.twitter.com and
api.twitter.com in your /etc/hosts file that map back to 127.0.0.1.

This obviously breaks Twitter integration, but it also makes sure page loads
don't explode when waiting for remote resources.

------
Braasch
Not only is the website down, but it seems the entire API is as well.

~~~
dreadsword
Yep - stalling out big time. Good chance to test your error handling in a real
world scenario!

~~~
nulluk
Have just had to add timeouts to a few requests to the twitter API. Was
completely hanging a whole site. Learn't something though, to never trust an
API to respond in a reasonable amount of time!

~~~
dreadsword
Words to live by!

------
zainny
Now how am I supposed to tell people that Twitter is down!?

~~~
kristofferR
It's sad, but my immediate instinct when I couldn't post a tweet was to try
searching for other people having problems with Twitter - on Twitter.

~~~
simondlr
Same. It gives credence to Jack Dorsey's claim of Twitter "holding the public
conversation".

------
sakopov
Millions of people couldn't post a status update for hours. It's a terrible
day on the internets.

------
zbowling
the mobile site is still up. <http://m.twitter.com/> You can tweet and
everything. The streaming API is also still partly up.

------
ntkachov
I like how on status.twitter.com they tell you you might have issues accessing
the site. And then they give you a "tweet this" button.

------
qhoxie
<http://downscout.com/twitter.com> if anyone wants to track it.

------
dreadsword
Its been ages since Twitter has had one of its famous fail whale outages.

~~~
pinko
Notably this outage lacks even a whale.

~~~
darkstalker
The birds finally dropped the whale

------
moolcool
People are going to have to resort to [desperate
measures](<http://nedroid.com/2009/05/people-have-to-know/>).

------
teekarja
Twitter is down. The very fabric of modern life is endangered

------
debacle
Sorry guys. I added twitter to my phone yesterday and tweeted twice. I didn't
think it would break anything.

------
larrys
It's not down in when tried with an anon proxy service:

<https://proxify.eu>

------
rellik
Wheels down @ Chicago O'Hare. Ping me if you wanna meet up while I'm in town

------
yati
If they decide not to hide it, some interesting news is on its way ;)

~~~
voltagex_
Elaborate?

------
arunagarwal
Where should I tweet... when twitter is down!!

------
davidkellis
Why is this on the HN front page? This is an entirely worthless post. It adds
no value. Nobody is going to reread this at any point in the future. Utterly
worthless.

------
larrys
<http://www.klonix.org/jun2012/twitter-20120621-125438.jpg>

------
modarts
How does a service who's sole purpose is to allow users to create and retrieve
140 character messages go down on such a frequent basis?

------
mikecane
I'm not able to sign in. It happened when I tried to send a tweet from my blog
while I was signed in. So, yeah, even the API is down.

------
circa
Not even a fail whale. bummer.

------
cdooh
It's down here in Kenya too

------
jsilence
yes, "walled garden" is "single point of failure" spelled backwards.

------
th0ma5
Mobile is back for me now

------
Matt_Mickiewicz
You forgot the second part of the headline - "Twitter Completely Down,
Productivity Surges"

------
draftable
But if Twitter is down, how can I tweet my complaints about Twitter being
down?!?!?

------
ThomPete
Hmm it's up for me.

------
jason_slack
Wow, Twitter is down, I guess I can get some work done today :-)

------
mck-
It's back up

------
jwu59
twitter is up for me

------
thatusertwo
I can't tweet!

------
SuperChihuahua
Wheres the whale?

------
goronbjorn
[http://www.quora.com/What-are-the-most-commercially-
successf...](http://www.quora.com/What-are-the-most-commercially-successful-
web-applications-that-are-also-poorly-engineered/answer/Sean-Rose)

