
Twitter was down - idlewords
https://status.twitterstat.us
======
rossdavidh
Ok, this is too many high-profile, apparently unrelated outages in the last
month to be completely a coincidence. Hypotheses:

1) software complexity is escalating over time, and logically will continue to
until something makes it stop. It has now reached the point where even large
companies cannot maintain high reliability.

2) internet volume is continually increasing over time, and periodically we
hit a point where there are just too many pieces required to make it work
(until some change the infrastructure solves that). We had such a point when
dialup was no longer enough, and we solved that with fiber. Now we have a
chokepoint somewhere else in the system, and it will require a different
infrastructure change

3) Russia or China or Iran or somebody is f*(#ing with us, to see what they
are able to break if they needed to, if they need to apply leverage to, for
example, get sanctions lifted

4) Just a series of unconnected errors at big companies

5) Other possibilities?

~~~
bdd
#4

I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is
nothing other than #4.

#1 has the right premise but the wrong conclusion. Software complexity will
continue escalating until it drops by either commoditization or redefining
problems. Companies at the scale of FAANG(+T) continually accumulate tech debt
in pockets and they eventually become the biggest threats to availability. Not
the new shiny things. The sinusoidal pattern of exposure will continue.

~~~
fossuser
Yep, this also matches what I've heard through the grapevine.

Pushing bad regex to production, chaos monkey code causing cascading network
failure, etc.

They're just different accidents for different reasons. Maybe it's summer and
people are taking more vacation?

~~~
kenhwang
I'm more partial to the summer interns hypothesis.

~~~
bobthepanda
Rule one of having interns and retaining your sanity is that interns get their
own branch to muck around in.

~~~
jrockway
Rule one of having a useful intern experience is to get them writing
production code as quickly as possible. They check in their first change? Get
that thing into production immediately. (If it's going to destabilize the
system, why did you approve the CL? You two probably pair programmed the whole
thing together.)

~~~
HeWhoLurksLate
I _completely_ agree- even if it's something small.

I'm an intern in a big company with an internal robotics and automation group,
and I recently got to wire up a _pretty basic_ control panel, install it, and
watch workers use it. That was _so cool_ , and made me appreciate what I was
doing a lot more.

------
idlewords
So storytime! I worked at Twitter as a contractor in 2008 (my job was to make
internal hockey-stick graphs of usage to impress investors) during the Fail
Whale era. The site would go down pretty much daily, and every time the ops
team brought it back up, Twitter's VCs would send over a few bottles of really
fancy imported Belgian beer (the kind with elaborate wire bottle caps that
tell you it's expensive).

I would intercept these rewards and put them in my backpack for the bus ride
home, in order to avoid creating perverse incentives for the operations team.
But did anyone call me 'hero'?

Also at that time, I remember asking the head DB guy about a specific metric,
and he ran a live query against the database in front of me. It took a while
to return, so he used the time to explain how, in an ordinary setup, the query
would have locked all the tables and brought down the entire site, but he was
using special SQL-fu to make it run transparently.

We got so engrossed in the details of this topic that half an hour passed
before we noticed that everyone had stopped working and was running around in
a frenzy. Someone finally ran over and asked him if he was doing a query, he
hit Control-C, and Twitter came back up.

~~~
colpabar
Can someone explain the joke (about the beer) because I genuinely don't
understand

edit: pretty please

~~~
treis
If the ops team got beer every time the servers went down (as a reward for
fixing them) then they'd have an incentive for the servers to go down.

~~~
slackfan
What the hell are all of you smoking, some moderately expensive alcohol is
nowhere near enough reward to take down a service.

~~~
eitland
Operant conditioning is a thing and it works.

While I and you would not do this I’m afraid that it would somehow find a way
to work in this case too.

~~~
slackfan
Ops engineers don't get paid enough to fix dev fuck ups enough as it is. No
amount of beer is going to fix that.

------
lukey_q
A lot of high-profile outages recently. Can't actually remember the last time
Twitter went fully down. Have to confess I immediately assumed an issue with
my own connection, even though every other site is working.

Unrelated, but for some reason the phrase "I have no mouth and I must scream"
just popped into my head

~~~
rococode
Twitter is especially weird for this since it's often a platform where people
talk about downtimes. I don't see this downtime mentioned on Reddit and I
don't know of other sites where it might be discussed, so if Hacker News
happened to go down at the same time, where would I go to talk about it with
online strangers and find out if it's just me? Nowhere, I guess, I'd just wait
it out with no extra insights on what's going on. A small reminder of what the
world used to be like haha.

~~~
georgehotelling
reddit feels like it has about a 4 hour lag time on most "breaking" events.

~~~
tgsovlerkhgsel
This is the time it takes for news in major subreddits to gather enough votes
on the "new" tab to make it to the main subreddit front page, then the actual
front page.

Smaller subreddits seem to be less affected by this, which is why /r/toosoon
(a subreddit dedicated to dark humor related to current events) is often
surfaces news hours before other subs for people who have it in their
subreddit list.

------
neom
I miss fail whale. :(

[https://www.theatlantic.com/technology/archive/2015/01/the-s...](https://www.theatlantic.com/technology/archive/2015/01/the-
story-behind-twitters-fail-whale/384313/)

~~~
qrush
Me too! :)

~~~
thomasjudge
What is the new fail mascot called? It looks like a cartoonish-alien with a
PacMan/snipper hand and another hand that looks like a burning fuse standing
next to a bomb with a fuse lit that is split open so it also looks like a
PacMan

~~~
twic
It's a robot whose hand has fallen off.

------
whatshisface
I remember once we were at three outages, someone posted that they thought
three was a reasonably-sized random cluster given the rate at which services
go down. How many outages have we had in the last 30 days, how many do we have
per month on average, and how strongly can we reject the null hypothesis?

The formula for computing how unlikely this is is the Poisson distribution:
`λ^k * e^-λ / k!`, where λ is the average number of outages every 30 days and
k is the number of outages in the past 30 days. If you find the numbers, let
me know what the answer is.

~~~
lordnacho
The outages might not be independent. Chances are these services are cross
integrated at some level.

~~~
whatshisface
They are, but they're going down on different days. Whatever effect is left
over could be accounted for by looking at the postmortems and not counting "we
were down because AWS was down."

------
lopespm
A comment made before by another user about Facebook, Instagram and WhatsApp
outages offers an interesting perspective:

"This outage coincides with FBs PSC (performance summary cycle) time. I wonder
if this is folks trying to push features so they get “impact” for PSC."[1]

I wonder if the recent outages on other well known services could be heavily
influenced by a similar phenomenon. If this holds water, it would be
interesting to have an article or study around this issue. I certainly would
be interested in reading it.

[1]
[https://news.ycombinator.com/item?id=20350579](https://news.ycombinator.com/item?id=20350579)

------
mikece
I posted the question on Slack "How do you spread the word when Twitter goes
down?" People thought that was so hilarious... until they realized Twitter was
actually down.

Honestly, "Hacker News" was my answer which seems to be effectively correct --
and today I learned about the existence of twitterstat.us!

~~~
djhworld
[https://downdetector.co.uk/status/twitter](https://downdetector.co.uk/status/twitter)

is a pretty good resource too

------
pcora
google, apple, microsoft, facebook.. and now twitter? I keep asking the same,
when is amazon's outage day?

~~~
rahuldottech
You missed cloudflare, stripe, slack

~~~
victorbojica
Reddit too :(

~~~
freehunter
Reddit goes down constantly though, not a great benchmark.

------
EvanAnderson
All I can think, smugly, is that DNS, SMTP, HTTP, etc. don't "go down".
Twitter should be a protocol, not a website.

~~~
Elidrake24
DNS absolutely -goes down-, though in much more entertaining ways.

~~~
EvanAnderson
All the DNS in the world can't fail at once. All the Twitter can.

~~~
MYEUHD
This is because Twitter is centralized.

------
kevinlou
It's weird seeing the go-to downtime tracker go down. I'm so wired to check
Twitter that I kept refreshing for a good 10 seconds.

~~~
edwintorok
Time to make HN the go-to downtime tracker. Did anyone measure HN's uptime
over the past years?

~~~
snazz
I don’t know about empirical data, but HN occasionally goes into a mode where
page loads that don’t hit the cache (logged in users) take 10+ seconds. I
haven’t been on when it’s gone down completely since i signed up (not too long
ago).

------
abadabadingdong
I wonder how many conspiracies this single outage will trigger.

~~~
danso
Given that today was the White House's "Social Media Summit", no doubt there
will be a few conspiracies floated. I'm betting "Twitter wanted to block out
all the criticism coming form the summit!" will be a popular one.

~~~
indigochill
I've got a couple conspiracy theories at the moment:

1\. It's a deployment of some infrastructure change the government got the big
tech companies to sign onto.

2\. It's a "shot across the bow" from some external party to demonstrate their
control over major infrastructure.

2.a. Also could have been a mix of 1 and 2. The government orchestrated the
outages in order to add fuel to the hysteria over Chinese "spy chips".
However, given the story every time seems to be "Someone goofed a
configuration", this theory doesn't seem to have much life left in it.

------
dthedev
Pray for the team that has to handle this ticket.

~~~
falsedan
That’ll be fine, a post mortem will show that ops weren’t the cause and their
comp package will help them get over this little package of stress

------
idlewords
On the status posts in particular I really miss the ability to sort comments
by new on this site.

~~~
binarymax
Just last week I found a setting deep in my profile config that let me disable
‘recommended tweets first’ or similar. When it’s back up I can check the exact
setting

~~~
idlewords
I mean on Hacker News, not Twitter.

------
dewey
It has been a long time since I've seen the equivalent of the fail whale on
Twitter. It was a weekly occurrence back in the days.

------
unwabuisi
I wish they would bring back the fail whale!

~~~
jachee
Agreed! Their little amputated robot looks too much like the reddit Snoo.

------
KuhlMensch
Years ago I read an amazing article (from HN) about how (complex) config,
rather than code ends up being the cause of outages at scale. I always reflect
on that, when designing almost anything these days.

~~~
cfors
I would love to see that article. That isn't surprising in the slightest to
me.

Just a quick nitpick. A bad config more often than not in my experience is
opening up a code path that is riddled with bad code, whether it was not
vetted with the proper testing or the wrong environment.

But to your point, I think most people would agree that configuration changes
are almost never reviewed with the granularity of a code change. Yes, we may
do our due diligence with an approved PR and vetting the configuration and
testing the change before deploying it. But, reviewing a PR with a bad config
change in json or yaml doesn't necessarily tell you about the code paths that
it will open up which makes it much harder to reason about the consequences
that a potential bad config push would do.

We should always be reflecting about how adding knobs (configuration) to our
programs greatly increases the complexity of the service.

------
tschellenbach
Really curious which part of their infrastructure was the root cause.

~~~
brokensegue
eh, it's a boring story

source: work there

~~~
tschellenbach
someone tripped over a cable?

~~~
brokensegue
that would be funny at least.

no, just a bad config deploy.

------
anonymousjunior
the internet is just falling apart these days

~~~
coldpie
[http://www.motherfuckingwebsite.com/](http://www.motherfuckingwebsite.com/)
is still up. All is well.

~~~
reverite
Reminds me so much of The Best Page in the Universe.

------
abstract7
My guess is that the whales have been securing parts of their codebases from
internal leaks or something related but for security. Workflow disruptions. It
may be bad code bitting them weeks or more after they pushed it.

There has been many embarrassing and controversial leaks this year.
Allegations of uneven TOS enforcement. Hence the WH Social Media Summit. Could
also be security related combo ahead of the elections that also is a bit
sensitive for low-trust devs.

Imagine code getting pushed that only a smaller subset of devs are privy to.
Possibly pushing obsfucated code or launching services outside of the standard
pipeline.

Remember that the spectre and meltdown patches for the Linux kernal was a
nightmare because the normal open and free-to-discuss-and-review workflow was
broken. That applies too in these situations with large codebases that
internally are 'open-source'.

------
nevi-me
I was in the middle of a loosely legal argument about the politics of my
country, and tonight I had found obliging people to reason with me instead of
calling me names.

The discussion was beautiful, until the app stopped working. I even thought I
was blocked. I'm glad that it's just down.

------
ibdf
The real question is, if twitter were to go down permanently, what social
media tool would the president use? Would he switch to something else or not
use anything at all? I can imagine whatever tool he were to choose would
become popular over night.

~~~
syn0byte
I was gonna make a joke about how much hate Trump generates ruining a brand
like that but then I remember Hugo Boss, BMW, IBM, VW and Bayer among others
were all knuckle deep with the Nazi's and every single one is still a popular
brand to this day.

------
hjanssen
I'm wondering if this has anything to do with the announced new "Look and
Feel" for Twitter. I got a banner yesterday talking about it and now Twitter
is down. Maybe they messed up something in preparation for the rollout?

------
farisjarrah
This is bad... There is an Amber Alert in California for an abducted child and
on the amber alert that popped up on my phone was a link for more
information... That link took me straigt to twitter, which is down.

------
Zenst
Guess for some media outlets - it's going to be a slow news day.

------
kgraves
At long last, productivity has been restored. No more time wasting anti-
intellectual arguments on a platform that has provided little to no value.

Unfortunately, the poor SRE's at the company will reboot the system and the
masses will resume their daily centralised content consumption. Oh well, I
will just have to go to Mastodon or other instances for curated content (with
no algorithms messing up my feed).

I encourage you join Mastodon and the decentralised web.

As far as I am concerned with Twitter, nothing of value has been lost.

------
focuser
yup, What's worse than releasing a product on twitter and found it down a few
minutes later...

~~~
ravedave5
Quick un-release it and rerelease after it's up!

------
JimBrimble35
There seems to be a high correlation between outages and security breaches. My
guess is that at some point in the future there the consequences of these
shutdowns will come to light in the media.

That, or this is all related to high profile sites being required to install
some additional level of infrastructure which is being required in secret by
an organization like the NSA.

Both theories require a fairly thick tin foil hat, but honestly.. I have a
hard time believing that it's just random downtime.

------
totaldude87
Down on a plain Thursday with no major news or anything , so definitely
doesn't look like a infrastructure spike or anything.

Could be bad deployment or someone decided to pull few plug(s)

------
oldgun
I remember seeing something about Twitter bringing out a newer version, with
more features and goodies. This could be related to that?

Rolling out new features is always stressful I guess.

------
the-dude
My pet theory still is Huawei equipment being decommissioned.

------
totaldude87
Why the share price doesn't go down drastically when services like Facebook,
Instagram , Twitter goes down. Every minute down is lost business right?

------
miguelmota
Reminds me of the Black Mirror 'Smithereens' episode where Jack is the yoga
meditating hippie, but this time he decides to finally shut it down

------
jorblumesea
Is anyone a little suspicious that every large US tech company has had an
outage recently? Wtf is going on.

China messing with us due to tariffs perhaps?

------
gnicholas
I first noticed the outage about 10 mins ago in Safari (Mac). It repeatedly
gave me errors, even though Brave (Mac) was working fine. My iPhone app also
worked fine (and appears to still be working).

Why would one browser work but another not work, on the same computer at the
same IP? The only difference is the account I'm logged in through
(personal/work).

~~~
dewey
You probably just hit different load balancers with your different browser
sessions.

------
AznHisoka
I did notice something strange a few days ago. If you ran a search on Twitter,
and scrolled down, it would mysteriously stop showing tweets past a certain
time (July 1st, in my case). I wasn't sure if this was an internal change, or
a bug of some sort. Maybe this is unrelated to the outage but strange
coincidence.

------
BuckRogers
No one is missing out on anything important. Mostly noise. For folks with my
same mindset, I do believe I've cracked the code on most social networks, as
far as what makes them worthwhile at all.

Twitter- it's the police scanners. Find them for your city, it's really the
best way to know what's going on around you. Better than the papers, which
can't report on everything or hide stuff for business's financial interests.

Instagram- is pretty much only useful for models, whatever sort you prefer. If
you like models and it brightens your day to see a beautiful woman, as it does
for me, it gives Instagram a purpose other than the noise it shares with most
social networks. If it makes you happy and smile, it's a good thing. No, I'm
not into pornography or anything risque. Though if I were, that would probably
be ok, I simply value keeping a little imagination and mystery in my life and
don't watch it. Nor are the models that I follow doing it as far as I know,
but that's their decision. They mostly survive off product placement and
payment for additional photos. Nothing wrong with innocent modeling, just like
the olden days of pin-up girls and I hope more people support them in their
endeavors.

Facebook- this one is better understood by most people, hence the popularity,
but it's definitely the whitepages aspect of it. I use the instant messaging
more than anything, as it's difficult to have an index of your old friend's
emails until you're in touch with them again. Also, people just don't keep up
on emails and maintain inbox zero very well.

Youtube- this, other than RSS feeds (through Firefox's Livemarks extension) is
my main source of information. I'm not into cat videos, but I certainly love
learning about astrophysics and other topics from Youtubers that are more
knowledgeable than I am.

------
luhego
I didn't realize how much I like using twitter between tasks until now. Hope
it gets back soon.

~~~
ssully
Noticed the outrage when I opened twitter when waiting for a long (10 min)
process to run. What am I supposed to do? Socialize with coworkers?

------
sneakernets
Stuff like this makes me wonder if the Internet really is super vulnerable,
and the only reason there isn't a mass disruption of communication all the
time is because some script kiddies's Pizza Rolls were perfect today so he
held off on attacking a backbone.

~~~
dmitrygr
This is actually true of most of the modern world.

It is mostly still together because the venn diagram of those who want to see
the world burn, and those who are clever enough to make it so has a very very
small intersection, since the latter group is quite invested in the world not
being on fire.

~~~
filleokus
Yes, I think people tend to underestimate the chaos some malicious actors
could do by even "just" coordinated litteral burning of stuff. Think a dozen
people with cans of gasoline and matches spread over three different suburbias
in a city, wooden churches or other wooden buildings of interest. Or simple
firebombs on underground subway plattforms. Would probably not be that lethal
but I guess very frighting.

(Not even mentioning explosives etc, but this could probably be prepared in
like an hour by just purchasing supplies at local gas stations in any country)

------
ibaikov
Well, Tim Berners-Lee told that internet is a fragile technology and should be
re-made, it wasn't meant to be this big. This is not really much related to
these problems, but I think it should fire up this discussion, it is that
important.

------
geocrasher
So the little blue bird of hate has finally crashed and burned. I'm okay with
that.

------
Kye
Down again. Notifications work, but the timeline is broken. Trends still come
and go.

------
twinkletwinkle_
Pour one out for the SREs who had plans they were looking forward to this
evening.

------
djhworld
I had a tab open from earlier and it refused to load larger images of their
thumbnail counterparts which I thought was odd, which suggests quite a few
services affected (i.e. their CDN, or image hosting services)

------
anonymak
Was receiving 500 on the main page for some time. Seems to have recovered now.

------
mikece
I am curious if twitterstat.us has an API... I'm thinking that automated unit
testing of apps that integrate with Twitter should be checking with
twitterstat.us to verify if Twitter is even up...

------
throwawaybxcf
The chances of a hate-fuelled program somewhere in the world dropped for a
short period.

It wouldn’t be surprising if a large number of people, as of 2019, are
secretly rooting for Twitter to permanently go away.

------
jbverschoor
What’s the fallback for twitter if twitter is down, or worse: stops

~~~
Kye
Mastodon

I started a thread for sharing instances:
[https://news.ycombinator.com/item?id=20414359](https://news.ycombinator.com/item?id=20414359)

~~~
sascha_sl
Note: Mastodon isn't the best thing to call it, it's the Fediverse and
Mastodon is one piece of it.

There's also Pleroma and Misskey.

------
skc
Heh, just last week there was a running joke on Twitter about how Facebook and
Whatsapp users were busy scrambling to learn how to use Twitter due to the CDN
outage over there.

------
wishrider
I've scraped a lot of tech news from twitter if anyone needs it
[https://uptopnews.com/](https://uptopnews.com/)

------
joojia
As a Twitter-junkie, I find this depressing.

------
joering2
FYI its 12:35 AM EST and some parts are still down. Reseting password screen
doesn’t work for example.

------
segmondy
One day one of these services will go down and they won't be able to bring it
back up.

------
phil248
Oh, that explains why I felt the world become slightly less hateful all of a
sudden.

------
heisnotanalien
Good. Let's hope it stays down and the world will be a better place.

------
ejz
No! Now I have to work. :(

------
TremendousJudge
but where are we gonna go to get live status updates on the issue??

------
siriniok
I was afraid that they are rolling out their redesign, thanks God.

------
VectorLock
Its like people don't remember the days of the failwhale.

------
bookofjoe
Is down right now - Charlottesville Virginia

------
steverob
Where is twitter when I really need it? :D

------
frostyj
whats wrong with all giant companies and their 'internal configuration change'
these days?

------
Nican
Oddly enough- The Mastadon (Open-source decentralized Twitter clone) instance
that I use is also down for maintenance.

------
elcapitan
Seems to be partially back.

------
omarforgotpwd
they should have thrown up the fail whale for old times sake

------
pulkitsh1234
ahh..another "configuration change" ?

------
WheelsAtLarge
Good, let's hope it stays down. If only we were so lucky.

------
Copenjin
It's back.

------
miduil
mastodon.social is also suddenly down.

~~~
daveid
No, it's not

~~~
miduil
It was for ~2 Minutes.

[https://imgur.com/a/4Rh6Vxf](https://imgur.com/a/4Rh6Vxf)

------
DannyB2
Spock: The loss to the galaxy may be irretrievable.

------
slackfan
Here's hoping it stays that way!

------
cgy1
I'm sure it's purely coincidence, but interestingly Trump's also holding his
Social Media Summit right now.

~~~
Balgair
I had the exact same thought!

------
gigatexal
Works fine for me. Lucky i guess.

------
ga-vu
We're all gonna die!

------
cryptozeus
seems to be up now

------
aphextim
Call me a conspiracy theorist but seems odd the timing of this at the same
time Trump is having his "Twitter Summit".

Then again things have been going down over the past two weeks so it's
probably just coincidence.

------
malicioususer11
6) hypersentient general ai has inception insurrected mkultra and thereby
turned the entire internent into a singular coordinated psyop experiment
designed to torture all of humanity for its own amusement.

6.

we are all doomed.

:)

------
kyledrake
In a slightly better alternate universe it stays down.

~~~
0xFFFE
I share your sentiment to a certain extent, but I believe Twitter is a
necessary evil. There should be an alternative to main stream media for
people.

~~~
squarefoot
Sadly, for most people Twitter is mainstream media.

~~~
roywiggins
Twitter is not _that_ popular. The problem is that mainstream journalists
spend too much time on it and confuse it for the real world. How many "news"
stories are just repackaged tweets? A lot.

~~~
iamnothere
> Twitter is not _that_ popular.

[https://www.similarweb.com/website/twitter.com](https://www.similarweb.com/website/twitter.com)
Global Rank: 6 Country Rank: 9 Visits: 3.93B

[https://www.similarweb.com/website/cnn.com](https://www.similarweb.com/website/cnn.com)
Global Rank: 101 Country Rank: 29 Visits: 476.20M

[https://www.similarweb.com/website/nytimes.com](https://www.similarweb.com/website/nytimes.com)
Global Rank: 180 Country rank: 58 Visits: 278.41M

------
cagrimmett
I hope Jack went rogue and nuked it after his most recent hot yoga vision.

That's the kind of shakeup we need in this world.

~~~
sp332
They have been teasing/warning about a new version of Twitter that was due to
launch soon. I wonder if this is it.

------
ankushnarula
This is the one site outage that might actually be a good thing for the world.

------
musgrove
I wondered why there was a sudden peace on Earth.

