
Facebook, Instagram, and WhatsApp outages - skilled
https://www.independent.co.uk/life-style/gadgets-and-tech/news/facebook-down-site-app-not-working-pictures-photos-images-whatsapp-a8986426.html
======
cronix
One interesting thing I noticed is the ALT text that is displayed in place of
the missing images. It shows how their image recognition is classifying
everything. I guess a lot of my friends drink as a lot of the alt text was
like "Image may contain: drink"

~~~
oceliker
That's an accessibility feature and has been around for at least a couple of
years, I believe.

~~~
chirau
I think OP is referring to the fact that the alt-text is now all ML generated.
But yes, you are right, it's been around for a while

------
cyanbane
Browsing it right now is a good exercise for devs who need to make sure they
think about low vision/blind users who browse via audio, etc.

ie "Image may contain: a dog, sunglasses"

~~~
rzzzt
Automatic alt text is a nifty feature:
[https://www.facebook.com/help/216219865403298?helpref=faq_co...](https://www.facebook.com/help/216219865403298?helpref=faq_content)

------
nilayj
Looks like Facebook image store is having issues. Facebook, Instagram and
WhatsApp use the same image store, so all the services are affected.

~~~
giancarlostoro
My concern there is my understanding has always been that WhatsApp implements
end to end encryption through the Signal protocol, how can images be stored
and be considered end-to-end encrypted? So are only text messages end-to-end?
Ignoring logs being uploaded to the cloud.

I wouldn't send any sensitive pictures over WhatsApp then.

~~~
octorian
Because the actual data for those images, that is stored on Facebook's media
servers, is just a blob of ciphertext. Only the sending and receiving clients
actually have the keys, not the servers.

------
the-dude
I find the outages of the last few days a curious coincidence : Slack, Google,
Cloudflare, Azure, now FB.

edit : added Slack after @kache_ mentioned it, added Azure after
stevehubertron mentioned it.

~~~
samstave
So the options are:

1) a secret cyber-war is going on...

2) Some entity is installing new spying slurps

3) An (critical) IX that we werent aware of is having major issues... and we
dont know who to blame on that one...

~~~
lovecg
4) People love to find patterns where none exist

~~~
anon_z88
This is a fallacy all humans experience, however, what is the probability
specifically that all these problems are unrelated? At what probability is it
mathematically impossible that these problems are unrelated?

AFAF

~~~
NateEag
The answer to your last question is 0.

"Impossible" means "impossible", not "unbelievably improbable".

That's even more true when you ask about "mathematically impossible", as it
reinforces the idea of formal logic being the relevant domain, where precise
meaning of words is fundamental.

If you adjust your question to be "at what probability is it unreasonable to
claim these problems are unrelated?", then the answer is subjective -
different people have different standards for reasonability.

I think we'd need mountains more data than we have about the incidents to
compute a meaningful probability, anyway.

------
heavymark
I first noticed Instagram then realized it was Facebook and their other
properties such as What's App as well. Is very odd all of these major outages
this week. By Cloudflare at least said it was an employee error, so assuming
it's not subtle attacks by other countries unless companies like Cloudflare
are required to legally provide an alternate story for national security.

~~~
ksec
According to Digital Attack [1] it seems the world is constantly on DDoS.

[1] [https://www.digitalattackmap.com](https://www.digitalattackmap.com)

~~~
Radle
"Notable Recent Attacks" from 2016, the site is dead and shows fancy graphic.

------
supergauntlet
Feels like every day there's another one of these outages. Shame that
everything is so centralized that there's a single point of failure that
affects 3 of the biggest communication platforms around.

------
y04nn
All those outages are strange, I wasn't able to access my personal server for
few hours today, I was not even capable to ping ip). When I was able to
reconnect, everything seem to be ok. Maybe there is some routing/BGP
issues/attack (again). Sadly, I didn't try to traceroute the serveur IP.

~~~
cfitz
Second this, with my DigitalOcean VPS's. DigitalOcean posted an incident
report yesterday. They say "global networking issues" were caused by a "major
provider", but unfortunately there isn't much more detail than that [1].

[1]
[https://status.digitalocean.com/incidents/qvdcj7yx4030](https://status.digitalocean.com/incidents/qvdcj7yx4030)

~~~
devin
That was due to google cloud networking issues.

~~~
jammygit
GCP provides infrastructure for digital ocean?

~~~
captn3m0
No, but their Data Centres may be sharing upstream fibre provider?

Google mentioned a Fiber cut upstream.

------
_nickwhite
Not only Facebook, but Instagram (which shares common infrastructure) has also
been mostly down (read-only mode it seems). The most popular Twitter hashtag
right now is #instagramdown

~~~
parthdesai
WhatsApp as well.

------
agoodthrowaway
This outage coincides with FBs PSC (performance summary cycle) time. I wonder
if this is folks trying to push features so they get “impact” for PSC.

~~~
lopespm
Very good point. I wonder if the recent outages on other well known services
could be heavily influenced by a similar phenomenon. If this holds water, it
would be interesting to have an article or study around this issue. I
certainly would be interested in reading it.

------
edwintorok
Do we need a 'Show Outage:' tag? HN seems to become the defacto outage
reporting place...

~~~
eneveu
Because it's interesting to discuss the impact of those outages and the
reasons for them, to learn from those experiences.

------
smaili
One interesting thing I've noticed from this (not sure if this is just my
experience or if others noticed as well) but none of the ads seem to be
impacted by this. Certainly makes me wonder if those are on a more prioritized
and entirely separate infra and SLA that is designed to be more resilient and
highly available.

~~~
jedberg
It's a lot easier to serve ads than your custom photos. The ads are more
generic, easier to cache globally, and show to a lot of people. They can also
fall back to super generic ads if the database is unavailable to customize
them.

But your photos don't have a fallback.

~~~
londons_explore
Fallback is the main reason. Some webservices even fallback to non-moneymaking
ads for charities in the case of a technical fault, because shareholders react
badly to an ads outage, but they don't notice an hour or two of charity ads...

------
kevlawrence
This gives me a lot of confidence in their crypto currency.

~~~
smt88
1) Cryptocurrency ops are so vastly different from running a social website
that I can't even think of any overlap.

2) I hate Facebook as a company, but as a builder/scaler of web apps for many
years, I'm continually blown away by the speed and reliability of their
website. Their operations are mind-blowing.

The only comparable apps (in terms of scale) are Gmail and YouTube, and Gmail
is simpler in certain key areas (e.g. mail delivery isn't millisecond-
sensitive for a user).

~~~
reaperducer
_Cryptocurrency ops are so vastly different from running a social website that
I can 't even think of any overlap_

I know nothing about how cryptocurrency works, but wouldn't social media
outage sources like multiple server failures, hurricane, tornado, sliced fiber
line, etc... affect the kind of cryptocurrency that Facebook is embarking on?

Or is there something in the "distributed" nature of cryptocurrency that makes
it more resilient? Is Facebook using that model, too?

~~~
dboreham
No. Yes.

------
kofisarfo
Imagine if this meant not being able to send or receive currency.

~~~
buboard
Has there ever been a major bitcoin or ethereum, or stellar outage?

~~~
ceejayoz
Facebook's more akin to an exchange for Libra, isn't it? There's 28 companies
partnering up for the system.

~~~
buboard
They have one wallet. But there is no reason for a wallet to depend on
facebook being online, and presumably there will be 100 points of failure for
the ledger on launch

------
kache_
I wonder what kind of economic damage all these recent outages add up to. How
much harm are they causing, in terms of productivity and value. Facebook's
social networks I'd imagine are less impactful than the slack outages, which
could completely cripple a company if they were primarily remote.

~~~
JMTQp8lwXL
Ad content (images and video) seems to be loading fine on the iOS App. So the
damage is likely minimal.

~~~
partiallypro
Our ad team couldn't upload any images to ads this afternoon, so I'd say it
did do -some- damage.

------
awirth
Some popular images seem to load for me, but longer tail stuff doesn't. Maybe
they accidentally flushed their edge cache and then their origin couldn't
handle the load.

~~~
londons_explore
Could they be in your own browser's cache? Or maybe in your ISP's cache?

Considering the type of content Facebook, WhatsApp, and Instagram host, I'd
imagine the cache hit rates to be pretty low for media. They'd be effective
for JavaScript/CSS though, which might also be served from the same system

~~~
awirth
No, I checked if it was my own cache, and it's all HTTPS so it couldn't have
been my ISP's cache either.

I compared a few celebrity's instagrams with accounts of friends with many
fewer followers. Celeb's images loaded but less and less as you scroll back
through time, friends images all 503'd. Still seems to be the case.

~~~
londons_explore
By ISP cache, I meant fbcdn running off a dell server in a rack at your ISP...

~~~
awirth
Ah, sure. Down detector's map suggests it's global:
[https://downdetector.com/status/facebook/map/](https://downdetector.com/status/facebook/map/)

Instagram also doesn't seem to be serving hot and cold content from different
hostnames/IPs for me, so it couldn't be that either.

Also check out the official status update at
[https://developers.facebook.com/status/issues/34337305659485...](https://developers.facebook.com/status/issues/343373056594853/)
and
[https://developers.facebook.com/support/bugs/610822019411772...](https://developers.facebook.com/support/bugs/610822019411772/)

------
legohead
Noticed it this morning (~2 hours ago) on WhatsApp, voice messages weren't
working, and still aren't.

So, code deploy or server configuration change?

~~~
sna1l
Seems like if it is affecting their services so broadly it is a server
configuration change

~~~
pratap103
Yup, seems like it. This is second time in a couple of months that they've
been down.

------
jbverschoor
Duplicate of
[https://news.ycombinator.com/item?id=20344749#20344824](https://news.ycombinator.com/item?id=20344749#20344824)

------
twsted
Do you remember when we first heard about "the cloud"?

I remind about a video when Larry Ellison mentioned the cloud and I thought:
"What is he speaking about? It's the internet".

How I was wrong...

Now we are in a situation where we have too much concentration and we are
experiencing that. It's a problem.

~~~
supergauntlet
I wonder if there's a large enough market for a "decentralized cloud" along
the lines of IPFS but for compute. And, yknow, actually working decently.
Something where you could pay compute/storage operators for resources and also
be a server host and share your excess compute power.

~~~
nexuist
This is effectively the idea behind the Ethereum blockchain model. You can
make "smart contracts" (think APIs with persistent state) using languages like
Solidity[1] and deploy them onto the blockchain. After that, you can invoke
individual functions by paying "gas" (small amounts of ETH) that goes to the
node operator's account. Smart contracts also get their own blockchain
addresses, so they're capable of sending and receiving transactions. Meaning
you can build financial applications with little to no barriers to entry!
(whether this is a good thing or not, I will leave up to the reader).

The really cool thing about it is that nobody owns a smart contract once it's
deployed. You can't edit the code or even delete the contract itself. It's a
truly autonomous entity that will continue to operate the same way forever
(unless there's a 51% attack or something of that nature).

The obvious benefit for this is that you can mathematically ensure trust. For
example, if you hosted a lottery app on a LAMP stack, you could steal the
money, hackers could get into your server, your database could get corrupted,
etc. On the blockchain, nobody can access your lottery funds or business
logic, not even yourself, meaning that as long as you developed the
application correctly (to be fair, that is a big assumption), it is truly
fair.

[1]
[https://solidity.readthedocs.io/en/v0.5.10/](https://solidity.readthedocs.io/en/v0.5.10/)

~~~
giancarlostoro
I always wondered about this, so how do projects like CryptoKitties profit? Or
do they give themselves the seed cryptocollectibles and profit off of those?

~~~
somebodythere
> give themselves the seed cryptocollectibles and profit off of those

In CryptoKitties' case specifically, yes. Some developers bake a fee directly
into the smart contract.

~~~
giancarlostoro
So they coded the fee system to go to their own address and not just the
systems hosting correct?

------
thrusong
Seems to be completely back online from my end in central Canada.

------
tcarn
Surprised this impacts Whatsapp as well which seemed to store everything
encrypted on your device...

~~~
oasisbob
Likewise. If it's a well-thought out encrypted blob store, fine...

But given that Facebook, Instagram, and WhatsApp would seem to have vastly
different caching and security requirements - its hard to believe.

Is any of this architecture publicly documented?

~~~
thrusong
I think Zuck himself said recently they're moving to encrypt everything and
merge infrastructures for single login across properties (which a lot of us
think is them sneakily trying to make their web properties a lot harder to
split up in an anti-trust case).

------
sonofgod
Thanks, was wondering what was up.

------
o_p
A perfect moment to evangelize my contacts about Telegram...

~~~
edwintorok
I've heard good things about Signal and Wire too.

------
functionCall
I am see WhatsApp issues as well. Can not download media.

~~~
robjan
Yep, their CDN is down. All Facebook properties are affected.

------
manjana
I'm having issues with streaming a series from HBO Nordic as well.

------
hn23
So what is next, productivity peak at maximum?

------
bitL
Anonymous working on post-FAANG world?

~~~
gsich
Gmafia is a better term.

------
buboard
The fact that these are called outages is kind of alarming. Do people rely on
any of these as their exclusive communication when in danger?

~~~
wybiral
I don't think the service needs to be critical for it to be an "outage". If
the service is unavailable to people who use it that's an "outage".

You could have a soft-serve ice cream machine outage.

------
slimshady93
It seems intern season has begun.

------
samstave
I am skeptical about all these outages occurring over the last week...

Many outages.

I think a silent cyber-war is occurring between the US and China.

