
Facebook, Instagram and Messenger were down - mtsx
https://twitter.com/hashtag/facebookdown
======
bonniemuffin
About 2 days ago, I noticed that Facebook added multi-level reply threads for
a small percent of posters, but as of this morning it seemed totally broken
for me on the latest ios build. Seems unlikely to be related, but I wonder if
it's possible they broke the site trying to hotfix it.

Regardless of the root cause, I'm feeling for the oncall folks whose
Thanksgivings are getting ruined right about now.

~~~
ceejayoz
I work pretty heavily with the Facebook Graph API and things have been pretty
messed up ever since Cambridge Analytica and the scramble afterwards.

I've got a ticket from August about a basic piece of functionality - deleting
a post via its ID - that's completely broken, and all I get is an every-few-
weeks "should be fixed", "nope, it's clearly not", "ok stand by" exchange with
them.

Basic stuff like "show me the user's Facebook pages" breaks for everyone for
hours at a time. When we attended F8 they talked heavily about their intensive
automated testing, but I don't see that much evidence of it. I suspect a lot
of things just got turned off in the mad "shut down EVERYTHING" scramble and
they're still working out the impacts.

------
g051051
If I hadn't been looking here, I might never have known. Yes, I'm bragging.

------
chadlavi
It's a thanksgiving miracle

------
rossdavidh
Man, what a wasted opportunity. If it had happened on a workday, we could have
seen a massive productivity surge to boost the economy.

~~~
AznHisoka
At least families will be forced to talk to each other instead of being on
FB...

------
samwillis
There is a major outage they are aware of effecting Facebook Login, Instagram,
Messenger and the Marketing API:

[https://developers.facebook.com/status/issues/22872935915151...](https://developers.facebook.com/status/issues/2287293591515186/)

[https://developers.facebook.com/support/bugs/555889181863481...](https://developers.facebook.com/support/bugs/555889181863481/)

Edit: I think there is a bug with the status page, the outage hasn't been
going for the 18 hours it is showing me here:

[https://developers.facebook.com/status/dashboard/](https://developers.facebook.com/status/dashboard/)

~~~
pmlnr
"you must log in to continue"

Hang on... to check the status of login you need to login? What a wonderful
circular dependency that is!

------
rinchik
Also curious if FB outages affect global internet traffic? Are there any
relative stats? (E.g. if google to go down, it would have an economic cost
comparable to a natural disaster)

~~~
pdpi
There's several anecdotes internally at Facebook that get told to new starters
about some instances of when facebook.com was down. One involves masses of
people calling the police to report that facebook is offline (wat, why?),
another involves so many people flocking to twitter to complain that fb is
down that _that_ crowd then knocked twitter offline ("How do you know if
facebook is down? Check whether twitter is online" sort of thing).

I don't feel comfortable sharing the exact figures, but during my time there
the cost of a few minutes of downtime on an ads-adjacent message queue cluster
directly turned into millions in lost revenue, so it's safe to say that losing
the capability to _serve_ ads (as opposed to charging for serving them) would
have a pretty damn high economic cost to those advertisers. As Workplace gains
traction, downtime there will creep into the "cataclysmic" side of the
economic costs scale.

~~~
jfim
Did you count the lost revenue as revenue per minute times the number of
minutes the site was unavailable?

If so, that's actually incorrect when calculating revenue loss. There will be
actually be an uptick in traffic after the outage, either through people
postponing their intended visit or checking if the site is down for them after
it comes back.

I know that at LinkedIn we broke part of the advertisement campaign creation
flow from downtime in our service for a short time, and the impact was
negligible since ads were still being served, and new ad creation would be
just be retried later by advertisers.

Now if you just lost the capacity to serve ads, there might be an impact, but
since a lot of advertisements are display ads (to raise brand awareness over
an immediate call to action), it's not clear whether or not it's a simple
minutes down times impact per minute calculation.

~~~
jacquesm
Facebook being down is only lost revenue for Facebook. Every other company on
the planet just _gained_ in terms of productivity and revenues.

~~~
lovecg
How do you figure? Don’t a lot of businesses basically use Facebook/Instagram
as their main (or only) front page?

------
rinchik
the scale FB operates on is staggering! Human factor, while still present, is
very low. Wonder what could have happened there. They don't usually release
postmortems, right? (since the general service they provide to the masses is
"free", meaning minimum obligating to reporting. Would be curious to read the
postmortem, analysis of what happened.

------
chanw
[https://status.fb.com/ads/](https://status.fb.com/ads/)

------
thekyle
I wonder if WhatsApp is also affected. I know they've been trying to merge its
infrastructure with that of Messenger.

~~~
ksec
WhatsApp not affected.The power of FreeBSD and Erlang. ( Not sure if they are
still on that stack )

~~~
notacoward
Nope, nothing to do with FreeBSD or Erlang. Can't say what it is, but there
are _hundreds_ of other differences between which pieces of FB infrastructure
WhatsApp uses vs. what other products use. This is in one of those hundreds.
;)

~~~
rescbr
What always amazed me was the scale of this difference not being larger.
Really fine engineering. ;)

------
dvduval
Down here in LA. Happy Thanksgiving.

~~~
thekyle
Down in Orlando, Happy Thanksgiving to you too.

------
jessaustin
I guess GitLab are no longer the most notable outage of the day...

------
hprotagonist
Ahh. happy thanksgiving!

------
throwaway40324
It must be just a coincidence that PHP 7.4 was released today, and surely FB
doesnt rely on PHP version updates to latest, right?

Just speculating..

Edit: that twitter thread is quite entertaining

~~~
cranekam
Facebook doesn't run vanilla PHP. It runs on Hack, a language developed from
PHP that runs on HHVM, a JIT VM it developed:

[https://engineering.fb.com/developer-tools/hack-a-new-
progra...](https://engineering.fb.com/developer-tools/hack-a-new-programming-
language-for-hhvm/)

~~~
throwaway40324
Right, but up until after Hackv3.3, PHP code was supported, which support
ended 9 days ago.

"HHVM v3.30 will be the last release series where HHVM aims to support PHP.
The key dates are...

2019-11-19: expected end of support for v3.30"

[https://hhvm.com/blog/2018/09/12/end-of-php-support-
future-o...](https://hhvm.com/blog/2018/09/12/end-of-php-support-future-of-
hack.html)

Note: I really dont believe my above speculation is related, just thought it
was an interesting coincidence, at best.

------
djanogo
Move fast and break things during major holiday sales. I wonder if some new
holiday sales feature that was supposed to go live caused outage.

~~~
Benjammer
Move fast and break stable infra

------
drcongo
Perfect opportunity for everyone to wean themselves off this cesspool.

~~~
dang
Maybe so, but please don't post unsubstantive comments to Hacker News.

------
celticninja
Good

~~~
dang
Could you please stop posting unsubstantive comments to Hacker News?

------
smaili
Is it just a crazy coincidence this happened right after Trump’s signing of
the new Hong Kong bill?

~~~
ryanmercer
Given most of America, except those unlucky people like me that are at work
for their 14th Thanksgiving in a row, are hanging out with family not starting
at the internet either watching parades or football instead of tracking
political happenings that the vast majority of Americans are clueless about
because we have no realistic way as an individual to do anything about even
stuff at a town/county level, I'm guessing it has nothing to do with that
whatsoever.

