
Google Cloud Is Down - markoa
<a href="https:&#x2F;&#x2F;status.cloud.google.com" rel="nofollow">https:&#x2F;&#x2F;status.cloud.google.com</a><p><a href="https:&#x2F;&#x2F;status.cloud.google.com&#x2F;incident&#x2F;compute&#x2F;19003" rel="nofollow">https:&#x2F;&#x2F;status.cloud.google.com&#x2F;incident&#x2F;compute&#x2F;19003</a><p>Status page reports all green, however the outage is affecting YouTube, Snapchat, and thousands of other users.
======
boulos
Disclosure: I work on Google Cloud (but disclaimer, I'm on vacation and so not
much use to you!).

We're having what appears to be a serious networking outage. It's disrupting
everything, including unfortunately the tooling we usually use to communicate
across the company about outages.

There are backup plans, of course, but I wanted to at least come here to say:
you're not crazy, nothing is lost (to those concerns downthread), but there is
serious packet loss at the least. You'll have to wait for someone actually
involved in the incident to say more.

~~~
boulos
To clarify something: this outage doesn’t appear to be global, but it is
hitting us particularly hard in parts of the US. So for the folks with working
VMs in Mumbai, you’re not crazy. But for everyone with sadness in us-central1,
the team is on it.

~~~
digaozao
It seems global to me. This is really strange compared to AWS. I don't
remember an outage there (other than s3) impacting instances or networking
globally.

~~~
murat124
You obviously don't recall the early years of AWS. Half of internet would go
down for hours.

~~~
djsumdog
Back when S3 failures would take town Reddit, parts of Twitter .. Netflix
survived because they had additional availability zones. I can remember the
bigger names started moving more stuff to their own data centers.

AWS tries to lock people in to specific services now which makes it really
difficult to migrate. It also takes a while before you get to the tipping
point where hosting your own is more financially viable .. and then if you
trying migrating, you're stuck using so many of their services you can't even
do cost comparisons.

~~~
StreamBright
I am not sure if a single S3 outage pushed any big names into their own
"datacenter". S3 has still the world record of reliability that you cannot
challenge with your inhouse solutions. You can prove it otherwise. I would
love to hear a solution that has the same durability, avabiality and
scalability as S3.

For the downvoters, please just link here the proof if you disagree.

Here are the S3 numbers:
[https://aws.amazon.com/s3/sla/](https://aws.amazon.com/s3/sla/)

~~~
snicker7
It's not so much AWS vs. in-house. But AWS (or GCP/DO/etc.) vs. multi/hybrid
solutions. The latter of which would presumably have lower downtime.

~~~
solidasparagus
Why would you think that self-managed has lower downtime than AWS using
multiple datacenters/regions?

~~~
KirinDave
Actually, I imagine that if you could go multi-regional then your self-managed
solution may be directly competitive in terms of uptime. The idea that in-
house can't be multi-regional is a bit old fashioned in 2019.

------
Yrlec
Now is a good time to point out that the SLA of Google Cloud Storage only
covers HTTP 500 errors:
[https://cloud.google.com/storage/sla](https://cloud.google.com/storage/sla).
So if the servers are not responding at all then it's not covered by the SLA.
I've brought this to their attention and they basically responded that their
network is never down.

~~~
crazygringo
Ironically I can't read that page because, since it's Google-hosted, I'm
getting an HTTP 500 error... but which means at least that service is SLA-
covered...

Cloud services live and die by their reputation, so I'd be shocked if Google
ever tried to get out of following an SLA contract based on a technicality
like that. It would be business suicide, so it doesn't seem like something to
be too worried about?

------
based2
[https://www.reddit.com/r/sysadmin/comments/bw1gye/most_googl...](https://www.reddit.com/r/sysadmin/comments/bw1gye/most_google_gsuite_services_gmail_calendar_etc/)

[https://www.zdnet.com/article/some-internet-outages-
predicte...](https://www.zdnet.com/article/some-internet-outages-predicted-
for-the-coming-month-as-768k-day-approaches/) 768k Day

~~~
_Marak_
This should be voted higher up.

According to [https://twitter.com/bgp4_table](https://twitter.com/bgp4_table),
we have just exceeded 768k Border Gateway Protocol routing entries, which may
be causing some routers to malfunction.

~~~
dreamer_soul
Isn't it weird that it's happening now even though that number was surpassed
nearly a month and half ago?

~~~
dyu
Different locations see different counts because of aggregation/de-
aggregation.

------
tntn
There goes 3 nines for June and for Q2. I guess everyone gets a 10% discount
for the month?
[https://cloud.google.com/compute/sla](https://cloud.google.com/compute/sla)

~~~
londons_explore
The discount seems way too small.

I would pay a premium for a cloud provider happy to give 100 percent discount
for the month for 10 minutes downtime, and 100 percent discount for the year
for an hour's downtime.

~~~
gundmc
Any cloud provider offering those terms would go out of business VERY quickly.
Outages happen, all providers are incentivized to minimize the frequency and
severity of disruptions - not just from the financial hit of breaching SLA
(which for something like this will be significant), but for the reputational
damage which can be even more impactful.

~~~
WrtCdEvrydy
> Any cloud provider offering those terms would go out of business VERY
> quickly

Minimum spends and a 50,000% markup based on adding that term to your
contract.

~~~
hn_throwaway_99
I think you're proving the parent comment's point. The number of businesses
willing to pay a 500x markup is _exceedingly_ small (potentially less than 1),
and at that point the cost is high enough where it's probably cheaper to just
build the redundancy yourself using multiple cloud providers (and, to
emphasize, that option tends to be horribly expensive).

------
w_s_l
You know this reminds me of a bad taste that Google Sales team left when I
asked for some of my billing that I was unaware of running after following a
quickstart guide.

AWS refunded me in the first reply on the same day!

GCP sales rep just copy pasted a link to a self support survey that
essentially told me, after a series of YES or NO questions that they can't
refund me.

So why not just tell your customers like it is? Google Cloud is super strict
when it comes to billing. I have called my bank to do a chargeback and put a
hold on all future billing with GCP.

I'm now back to AWS and still on a Free Tier. Apparently the $300 Trial with
Google Cloud did not include some critical products, AWS Free tier makes it
super clear and even still I sometimes leave something running on and discover
it in my invoice....

I've yet to receive a reply from Google and its been a week now.

I do appreciate other products such as Firebase but honestly for
infrastructure and for future integration with enterprise customers I feel AWS
is more appropriate and mature.

~~~
WC3w6pXxgGd
Anything created in-house at Google (GCP) is typically created by technically-
proficient devs, those devs then leave the project to start something new and
maintenance is left to interns and new hires. Google customer service
basically doesn't care and also has no tools at their disposal to fix any
issues anyway.

The infinite money spout that is Google Ads has created a situation in which
devs are at Google just to have fun - there really is no incentive to maintain
anything because the money will flow regardless of quality.

Source: I interned at Google.

~~~
kerng
I'm not sure why you are downvoted - seems like a reasonable insight and
explanation for the drop in quality and weird decisions Google is making
recently.

~~~
enneff
It’s not insightful at all. Just one intern’s very brief observations of
something way more complicated and nuanced than is deserving of such a
dismissive comment.

~~~
marcinzm
I'll take brief comments that shed partial light on something over no comments
at all and no insight at all.

------
ksajadi
GCP status page is worthless as it's always happy and green when production
systems are down and then they might acknowledge something an hour later

~~~
JimboOmega
Just like AWS, then. "Some users are experiencing increased error rates" =
"Everything has been down for hours"

~~~
bsimpson
"Everything is fine, unless you're Carl. There's a massage outage, but only at
Carl's house. Sorry, Carl."

~~~
notatoad
I'm also experiencing a massage outage. Please send masseuse.

~~~
bsimpson
Goddammit some (most?) days I can't type. "Massive"

~~~
quickthrower2
I got the missive. Thanks.

------
colinbartlett
Google Cloud is the number 4 most monitored status page on StatusGator and
Google Apps is number 12. In addition, at least 20 other services we monitor
seemingly depend on Google Cloud because they all posted issues as soon as
Google went down.

It's always interesting to see these outages at large cloud providers spider
out across the rest of the internet, a lot of the world depends on Google to
stay up.

~~~
FPGAhacker
I guess we know what steam uses (the store at least).

~~~
colinbartlett
I don't know about Steam, but I know Apple must use Google Cloud:
[https://www.apple.com/support/systemstatus/](https://www.apple.com/support/systemstatus/)

~~~
gowld
Less than 1% of users are affected

Is there any reason to presume these statuses are correlated?

Apple's issue is

> Users may be experiencing slower than normal performance with this service.

~~~
colinbartlett
I'm just assuming they are because it's been previously reported that Apple
uses GCP (and also AWS).

[https://techcrunch.com/2018/02/27/apple-now-relies-on-
google...](https://techcrunch.com/2018/02/27/apple-now-relies-on-google-cloud-
platform-and-amazon-s3-for-icloud-data/)

------
macintux
And thus was ruined hundreds or thousands of pleasant Sunday afternoons.

I don’t miss being on pager duty one bit. I see it looming in my headlights,
sadly.

~~~
newsbinator
It's the Queen's birthday, a Monday off here in New Zealand...

... but not for everybody now.

~~~
Scoundreller
So what happens when the crown changes? They change the holiday? Immediately?
For the next year? Sounds like a bit of a nightmare.

~~~
NeedMoreTea
The holiday is on the official birthday. The sovereign's actual birthday has
been separate from the official birthday for centuries, so the holiday does
not need to change.

------
darkof
That feeling when you open
[https://console.cloud.google.com](https://console.cloud.google.com) and see
that you don't have your Kubernetes clusters and CloudSQL databases, but CTA
to create first.

~~~
vpontis
Gosh, this was so scary... I thought someone had hacked in and deleted
everything...

I hope they come back. This is still pretty scary

~~~
flamingcow
[I am the Cloud SQL tech lead]

This is a networking issue, and your data is safe. Cloud SQL stores instance
metadata regionally, so it shares a failure domain with the data it describes.
When the region is down or inaccessible, instances are missing from the list
results, but that doesn't say anything about the instance availability from
within region.

~~~
grumblez
That's good to know. What confuses me is why they're saying "We continue to
experience high levels of network congestion in the eastern USA", when I'm in
us-west2 (Los Angeles) and none of my CloudSQL instances, nor is my k8s
cluster showing up or contactable...

------
_xerxes_
Nest is down too, not surprising given they are part of Google. What I don't
understand is why I can't still control my devices over my local network. Why
does the system even require access to Google servers?

~~~
kibwen
Let's make the following the tech mantra of the next decade: _" the Internet
Of Things can never possibly work until the LAN Of Things does"_.

~~~
nektro
the IoT trend really needs to stop

~~~
xhgdvjky
LANoT?

------
titzer
This is your yearly reminder to resist centralization of the internet.

~~~
jasonvorhe
Because Internet Exchanges and carriers never had network issues.

~~~
eitland
Because then at least you and your colleagues has a chance to get _some_ work
done.

------
londons_explore
It seems the AdWords anti-spam system is down, which means anyone can put a
billion dollar bid on every keyword and get their ads showing on every Google
search for every query.

Systems that fail 'open'...

------
filvdg
Level(3) one of the biggest backbones has issues too, might be related

[https://downdetector.com/status/level3](https://downdetector.com/status/level3)

------
sarim
Funny how as soon as I realized that Gmail and Google Sheet aren’t working
properly I rushed to HN to figure out what’s going on. I love this community!

------
jshprentz
The two Google Cloud networking incidents are:

Incident #19008 began at 2019-06-02 12:48.
[https://status.cloud.google.com/incident/cloud-
networking/19...](https://status.cloud.google.com/incident/cloud-
networking/19008)

Incident #19009 began at 2019-06-02 12:53.
[https://status.cloud.google.com/incident/cloud-
networking/19...](https://status.cloud.google.com/incident/cloud-
networking/19009)

Times are US/Pacific

~~~
cheeze
Looks like they are having trouble updating their statuses. 19008 was supposed
to be updated over an hour ago. Meanwhile, 19009 has the same comment posted
three times. I'm guessing internal tools are barely working at best.

~~~
londons_explore
That is normal for Google status pages.

They don't want to admit fault or place blame because there can be legal and
commercial ramifications, so they can only say canned responses.

~~~
cheeze
Updating a status page, sure. They aren't going to say "JamesBondService is
having issues because a bug was deployed", but they usually don't repeat the
same message 3 times in the same minute and they are usually pretty good about
sticking to an update "within SLA"

------
tlogan
I also noticed that Google search stop indexing news articles.

So I searched for "gmail down" on bing and I got some results [1]. But
searching on Google for "gmail down" does not return any results [2].

[1]
[https://www.bing.com/news/search?q=gmail+down&qs=n&form=QBNT](https://www.bing.com/news/search?q=gmail+down&qs=n&form=QBNT)

[2]
[https://www.google.com/search?q=gmail+down&source=lnms&tbm=n...](https://www.google.com/search?q=gmail+down&source=lnms&tbm=nws)

------
squarefoot
And Gmail too doesn't feel very well today.

    
    
      [21:55:19] POP< +OK send PASS
      [21:55:19] POP> PASS ********
      [21:55:21] POP< +OK Welcome.
      [21:55:21] POP> STAT
      [21:55:21] POP< -ERR [SYS/TEMP] Temporary system problem.
      Please try again later.

~~~
ToniCipriani
Pretty much Gsuite is out:
[https://www.google.com/appsstatus#hl=en&v=status](https://www.google.com/appsstatus#hl=en&v=status)

~~~
megous
At least G+ is working. /s

------
klon
Anyone using both AWS and GCP that can form an opinion on availability of
both? As a GCP customer I am not very happy with theirs.

~~~
pm90
GCP is incredibly bad at communicating when there are problems with their
systems. Just terrible. Its only when our apps start to break that we notice
something is down, then look at the green dashboard which is even more
infuriating.

~~~
timdorr
AWS is often the same way. No one seems to be good at communicating outage
details.

~~~
obeattie
I really don’t get this. There’s a huge number of complaints about poor
communication from companies like Google and AWS during every outage. Yet they
remain seemingly indifferent to how much customer trust they are losing, and
the competitive edge the first one to get this right could gain.

~~~
atmosx
If 20% of Google Cloud's customers leave after this outage because of _poor
communication_ they'll prioritise accordingly and apply all that nice SRE
theories to their infra. But this isn't happening, because <various reasons>,
so... who cares?

~~~
obeattie
I mean, I care. All else being equal I’m not sure why you _wouldn’t_ want good
communication to your customers.

~~~
manigandham
How much cloud spend do you control? That's the reality of how decisions are
made.

~~~
obeattie
Many millions of dollars per year. I care about how my providers behave when
they have issues, and I can't see why you think it's not at all relevant.

~~~
manigandham
> _" why you think it's not at all relevant"_

Nobody said this.

> _" I care about how my providers behave when they have issues"_

We all do.

As the other commenters stated, the communication is poor because the clouds
are still growing rapidly and there's not much reason to be better. We might
also be underestimating just how much more better service would cost and
whether it's worth the revenue loss (if any). Are you really going to shift
all of your spend overnight because of an outage? And where are you going to
go?

The reality of these decisions is far more nuanced than it may seem and the
current state of support is probably already optimized for revenue growth and
customer retention.

------
javabean_
Nice overview : [https://downdetector.com/](https://downdetector.com/)

~~~
XCSme
I see a lot of Nest complaints, isn't there any security issue if Nest goes
down?

~~~
pmlnr
I see people complain about Nest and the only thing I can think of is "what on
earth where you thinking to have a door or thermostat that doesn't function
without internet?!"

~~~
disillusioned
I live in a house with a Nest thermostat and Nest Secure and Nest x Yale door
locks. The AC is on just fine, and the door unlocked just fine. (The door lock
doesn't require an internet connection, unless you've enabled privacy mode for
some reason.)

~~~
pmlnr
I thought there were versions that don't work offline; my bad if this is not
the case.

------
different_sort
I was playing around this afternoon with appengine, and thought I broke one my
projects when I started getting 502 back.

There appears to be some irregularities on consumer services as well that are
of course certainly related, youtube was behaving a bit oddly for me.

The impact seems to be cascading down from just GCE to other services as well
- that status page certainly does not reflect the reality of the situation.
You can't even sign into GCP right now, and things that run on GCE, like
appengine seem impacted.

------
echelon
Nest is down for me right now.

It's amazing how far-reaching outages can be these days.

~~~
andrewprock
Code reuse is a wonderful thing, until it's not.

~~~
hn23
Code reuse? Couple if years ago some started to call servers "cloud" but the
cloud is just that: computers managed by someone else. If the whole service
layer relies on Three big companies, there is a problem..

------
mandatory
Yep, I can no longer see my Cloud SQL database - it's as if I've never created
one at all. Really hoping this is just an issue displaying it and that Google
hasn't punted my infrastructure and backups.

~~~
vpontis
Praying isn't working. Now, I'll try sobbing :(

~~~
Havoc
Systematic problem solving. I like it

------
brown9-2
When talking about GCE being down _please_ also mention what regions you are
talking about

~~~
chupasaurus
In this case it's a luck if any are working correctly, a problem is global
with some exceptions.

~~~
brown9-2
seems to be some comments here of some regions functioning ok, although
perhaps it’s not 100% in all regions

------
codys
Other google services are also affected:

[https://www.google.com/appsstatus#hl=en&v=status](https://www.google.com/appsstatus#hl=en&v=status)

[https://i.imgur.com/pcqwwA4.png](https://i.imgur.com/pcqwwA4.png)

~~~
mixedbit
Good that Google+ is up again

------
remote_phone
Phase 2 of mitigation was completed a few minutes ago and GCE is expecting
improvement shortly.

------
outside1234
It seems crazy to me that Google Cloud can have this level of instability but
I, on the other hand, can never remember google.com going down.

Why are they operating one with a different networking infrastructure from the
other?

~~~
sakisv
They did have a major outage for a few minutes on 2013:
[https://www.theregister.co.uk/2013/08/17/google_outage/](https://www.theregister.co.uk/2013/08/17/google_outage/)

~~~
escapecharacter
So this is now the longest outage ever, right?

------
gist
Right now this is at the 'good at least it's not mine to fix or worry about'
kind of like 'and the reason I choose IBM' [1]. I can just sit back and wait
for gmail to work correctly. Now at the point it starts to last what I would
consider a long time well then I will have things to worry about.

One thing with gmail though. When it's down it's similar to a snow storm if
you only do business in a city. Everyone is impacted and everyone understands
a missed deadline is unavoidable.

[1] For those not old enough to know what I mean read this:
[https://www.ibm.com/ibm/history/ibm100/us/en/icons/personalc...](https://www.ibm.com/ibm/history/ibm100/us/en/icons/personalcomputer/words/)

------
pitaj
Looks like my Gmail is back, but I don't have any emails from while it was
down. Yikes.

Edit: just got one email from the downtime, so perhaps my initial conclusion
was incorrect

~~~
Stryder
Whoa, if confirmed this would be really bad. Downtime happens, but data loss
is much worse.

~~~
BillinghamJ
This is normal handling for SMTP - the sending server just tries again later.

------
lewis1028282
So just got a new phone and trying to resync my 2FA for AWS but I can't
because Gmail is down. Ffs.

------
PerfectElement
Just 2 weeks after I migrated a DB cluster from Azure to Google Cloud thinking
things would get better.

~~~
AaronFriel
Just remember that "better" can be entirely based on when you pick the
starting and ending points of your graph ;)

~~~
XCSme
It's just like the stock market.

------
markoa
We're hosting an open global Zoom call for all engineers affected by the
outage, join us at [https://zoom.us/j/793450725](https://zoom.us/j/793450725)

~~~
avocado4
One click on this link and it instantly starts streaming your webcam footage
to everyone in the chat room.

~~~
revvx
This is weird, I had Zoom installed a long time ago but uninstalled according
to their instructions [1]. I'm a macOS user.

As soon as I clicked that link, the client downloaded a PKG file, installed
itself and launched itself without asking me if I wanted to share my camera or
audio.

I uninstalled according to their instructions again, searched for all " _zoom_
" files in my disk and rebooted.

This leads me to believe that following their uninstall instructions is
insufficient, and there are hidden files left on my computer.

Sorry in advance for the off topic message

[1] [https://support.zoom.us/hc/en-us/articles/201362983-How-
to-u...](https://support.zoom.us/hc/en-us/articles/201362983-How-to-uninstall-
Zoom-on-a-Mac-)

~~~
narak
Don't be sorry, same thing happened to me just now and I'm trying to figure
out how they are installing locally from a URL click with no further input
from me.

edit: Found this thread with details but no resolution it seems:
[https://apple.stackexchange.com/questions/358651/unable-
to-c...](https://apple.stackexchange.com/questions/358651/unable-to-
completely-uninstall-zoom-meeting-app/358664)

~~~
revvx
I think I found out: there's a daemon process that I somehow missed the first
time around.

Deleting the .app file as instructed is not enough.

This StackExchange reply [1] showed me how to solve it, at least on macOS.

[1] [https://apple.stackexchange.com/questions/358651/unable-
to-c...](https://apple.stackexchange.com/questions/358651/unable-to-
completely-uninstall-zoom-meeting-app/358664)

------
biohax2015
Does anyone know if this is a regional or global outage?

I can see my GKE clusters in one region but not in another, so I am guessing
it's the former.

Looks like we'll need a cluster in each region going forward...

~~~
boulos
To help others: which two regions?

~~~
biohax2015
us-central1 is up

us-east4 is down

~~~
hoonto
us-west2 is down

------
fdgwhite
I tried multiple times to setup a Google Wifi router today. Wifi would work
but the app said it was offline. Perhaps I am not insane or incompetent after
all

------
bprasanna
[https://status.cloud.google.com/incident/compute/19003](https://status.cloud.google.com/incident/compute/19003)
Does status update mean only the status. Won't there be words like "sorry",
"apologize" & "inconvenience"! Only PR is responsible for those words?

~~~
jwandborg
Apologies are unnecessary on status pages. You can be sure that they are
sorry.

------
snisarenko
The status page took a while to show issues. My app was down, and Twitter knew
google cloud was down before the official status page.

------
moon2
Its funny how searching for "Google Down" on Google and filtering for results
within 1 hour yields nothing.

------
mtarnovan
Is shopify on google cloud? i noticed they are having issues too

~~~
brown9-2
yes

~~~
captn3m0
I'd reported a networking issue to Shopify yesterday, which they said was
resolved upstream (GCP?). Could it be?

------
m0zg
Prediction: the final postmortem will say "someone pushed a bad config", just
like most of the previous postmortems (and most of the internal postmortems as
well, for Borg-based services). This is the cause of most other outages in
other cloud providers as well. A really hard to solve problem.

~~~
krackers
Multiple regions seem to be affected though. Wouldn't it make more sense to
start out config pushes with a single region before expanding it to avoid
these types of wide outages?

~~~
m0zg
It's a networking outage. Google is a well known user of SDN. Networks, by
virtue of connecting things, necessarily affect more than one region.

------
seibelj
Not sure if related, but I was going to a BBQ yesterday and myself and 3 other
people got lost because Google Maps app glitched out, directing us to the
wrong places. If you search twitter for #googlemaps tons of people have the
same issue. Surprised no one has posted about it.

------
nopriorarrests
[https://status.cloud.google.com/incident/compute/19003](https://status.cloud.google.com/incident/compute/19003)

>We will provide an update at 16:00 US/Pacific.

it's 16:22 and no updates were posted. a bit unprofessional..

------
dang
Related:

[https://news.ycombinator.com/item?id=20077275](https://news.ycombinator.com/item?id=20077275)

[https://news.ycombinator.com/item?id=20077293](https://news.ycombinator.com/item?id=20077293)

------
leowoo91
0 issues at compute, reporting for europe-west3-b,

------
squarefoot
I appears that logging into the webmail solves at least POP mail problems. I
tried my mail client and failed, then attempted a login to the webmail which
worked. Gmail then asked me to confirm my recovery address and cellphone,
which I did, and finally loaded the inbox page. I immediately attempted a
connection through the POP client and this time it worked.

It might be something security related if it triggers a mandatory identity
confirmation.

edit: I tried to send me a mail from another account and it worked but out of
4 or 5 mail checks at least two failed giving the same error.

[23:44:27] POP< -ERR [SYS/TEMP] Temporary system problem. Please try again
later.

The problem seems much more complex.

------
nkassis
The GCE console also affected, couldn't send a support ticket just getting
errors.

------
el_benhameen
That explains why my google home thing thinks it’s sub-zero even though I’m
warm in shorts and a t-shirt.

[https://pasteboard.co/IhBsyrsO.jpg](https://pasteboard.co/IhBsyrsO.jpg)

------
peterwwillis
The last big outage, iirc, was Google didn't test their rollback procedures
for router upgrades. I'll be very interested to hear if it's yet another
change control problem that caused this outage.

------
joaomacp
I was finishing a university assignment with the deadline 90mins away.

I wanted to upload a video of the project to YouTube and add a link to it in
the report. YouTube takes a long time to process the video, and then says it's
unavailable.

I go to Vimeo: it's down.

I upload the video to Dropbox, and copy its link to the report.

But my report was a Google doc. And when I tried to export it as PDF (which I
had not done yet) it couldn't do it. I never hated google more.

Eventually the video went through to YouTube, and I could export the PDF on
the third try, but this really made me conscious of my dependance on Google.

------
pgoodjohn
Everything looking normal on our GKE / CloudSQL stuff (eu-west1)

~~~
wichert
gcloud tells me:

WARNING: The following zones did not respond: us-west2, us-west2-a,
southamerica-east1-c, us-west2-b, southamerica-east1, us-east4-b, us-east4,
us-east4-a, northamerica-northeast1-c, northamerica-northeast1-b, us-west2-c,
southamerica-east1-b, northamerica-northeast1, southamerica-east1-a,
northamerica-northeast1-a, us-east4-c. List results may be incomplete.

Luckily for us eu-west1 seems to be working normally.

------
arach
Confirming issues on our end. I'm able to load up my console but when I go to
Kubernetes Engine, I don't see my clusters. I'm monitoring closely on twitter

------
40acres
Can't wait for the postmortem!

~~~
murat124
My money is on config push.

~~~
bpye
I would question why they are doing a rollout on a weekend?

~~~
dodobirdlord
A config push on a weekend seems pretty unlikely. Given that it's apparently a
network congestion issue and showed up on a weekend, my guess is that it's
probably a bizarre networking hardware failure like the one that took out
CenturyLink[1] last December.

[1]
[https://news.ycombinator.com/item?id=18789071](https://news.ycombinator.com/item?id=18789071)

------
d2xdy2
Was certainly an interesting alert when my Cloud Functions started reporting
downtime. Among the many things that dip in and out on what seems like a
monthly basis, I’ve not seen them just drop out in quite a while. Hopefully
they get things sorted out. I can’t really imagine what it looks like
internally when this level of outage is going on, but I want to think everyone
is fairly collected

------
tcarn
Once again HN proves to be the best status monitor.

------
chpmrc
This might be all in my head but I've been experiencing really bad latency for
like an hour, while browsing, and then I read this.

------
Exuma
[https://downdetector.com/](https://downdetector.com/)

Pretty much every service is down

~~~
lachs
It's frightening how many services rely on gcloud and which impact this
downtime has

------
ecabuk
My builds are failing because it cannot download the chromium.

> Error: Download failed: server returned code 502. URL:
> [https://storage.googleapis.com/chromium-browser-
> snapshots/Li...](https://storage.googleapis.com/chromium-browser-
> snapshots/Linux_x64/662092/chrome-linux.zip)

------
AdmiralAsshat
And Thunderbird suddenly kept throwing prompts for my Gmail passwords, even
after I sign back in. I hope it's related.

~~~
captn3m0
It says "could not reach imap.gmail.com" for me.

------
ccarpenterg
Google Colaboratory is back. At least I can access my github's notebooks and
public notebooks from google drive.

------
unilynx
G suite was broken for me 20 minutes ago (in Europe) but is working now.
Perhaps things are starting to come back?

------
sdan
Github contribution graphs are also gone

------
AngeloAnolin
Does Google also have some sort of listing on which consumer apps are
particularly affected (i.e. Gmail, Hangouts, Docs, Sheets, etc).

The cloud components may be directly affected but for consumers, there's
nothing which will provide info on what consumer facing services are getting
some issues.

------
derekhh
> We will provide more information by Sunday, 2019-06-02 12:45 US/Pacific.

I'm not seeing anything at 12:47.

~~~
derekhh
[https://status.cloud.google.com/incident/cloud-
networking/19...](https://status.cloud.google.com/incident/cloud-
networking/19008)

404 - Impressive

~~~
ryacko
Cloud status dashboards seem to be hosted on the same cloud, which doesn't say
much about redundancy.

~~~
chupasaurus
AWS changed internals of Service Health dashboard after they couldn't update
it when S3 went down in us-east-1
([https://aws.amazon.com/message/41926/](https://aws.amazon.com/message/41926/))

edit: wording

~~~
ryacko
Someone had to design the status page, and failed to anticipate the issue
about how the status page depends on the systems it is reporting on.

Unlikely the rest of AWS, a cached web page does not require much complexity.

------
cedricium
Majority of G Suite services are suffering service outages:
[https://www.google.com/appsstatus#hl=en&v=status](https://www.google.com/appsstatus#hl=en&v=status)

------
rum3
Keep on centralizing the internet with your stupid clouds and this is what
happens.

------
amasad
Anyone experiencing issues with GCS? Seems highly intermittent and dependent
on the location the request is coming from (maybe that's because it's a
networking bug).

The status page says GCS is fine but that's highly unlikely.

------
ramon
It seems that it`s focused on the database stuff like firebase and firestore.

~~~
dodobirdlord
Apparently (from further up the thread) it's a network congestion issue
causing extremely high rates of packet loss. I imagine pretty much anything
that's homed in the effected regions will be degraded or unaccessible.

------
tr33house
Took me a while to track latency issues to GCP. Wasn't expecting it. This also
seems to affect some GAE instances and some of their products like google
photos. At least according to my observations

~~~
different_sort
I see this as well.

------
ejanus
I just can't reach google apps on my HTC m9 since yesterday. I am in West
Africa. My Whatsapp crashed too and I lost all my previous threads. Is my
issue related to Cloud being down?

------
baalimago
Playing "is google on this website or not?" just became so much easier, simply
see if website works or not.

Scary stuff. What happens when Murphy's law decides to crash things even more?

------
horyzen
Yeah I was having trouble accessing my Gsuite apps, had a couple of 502s,
which led me to check HN. While it doesn't give me 502 now, it's abnormally
slow.

------
typenil
Was trying to set up SSL on a GKE cluster today. Guess I'll have to wait for
tomorrow if I want to be able to tell my mistakes apart from Google's.

------
landon32
u.s. west: all our cloud compute is inaccessible rn.... our API is down, can't
ssh into the servers, and also can't see them on the dashboard.

------
StaticRedux
This must be why GitLab is giving me shit. They recently made the switch.
Wonder if there is any second guessing going on over there right now.

~~~
Uninen
They have a incident report page up at:
[https://docs.google.com/document/d/1RM3QnuJ4FPH10J3UrJS0T26d...](https://docs.google.com/document/d/1RM3QnuJ4FPH10J3UrJS0T26d-mn_Dd11-jmZweHQVV8/edit#)

~~~
StaticRedux
Using Google Docs to report problems with Google, including Google Docs. Irony
at it's finest.

~~~
Uninen
They do have an internal ticket as well, but Gitlab.com is also hosted on
Google, so... [https://gitlab.com/gitlab-com/gl-
infra/production/issues/862](https://gitlab.com/gitlab-com/gl-
infra/production/issues/862)

------
benbristow
Wondered why Snapchat was being weird today. Thought it was my pi-hole setup
blocking something from working, but nope, it's Google!

------
ecthiender
In India, I could access Youtube, Gmail (web) and Google cloud console and GKE
and Compute Engine instances in south-east asia region.

------
vinayan3
Looks like only GCE is down according to the status page now. I'm able to
access my console for instances and GKE clusters.

~~~
miller_joe
They initially opened an incident against GCE
([https://status.cloud.google.com/incident/compute/19003](https://status.cloud.google.com/incident/compute/19003))
then opened one against Networking
([https://status.cloud.google.com/incident/cloud-
networking/19...](https://status.cloud.google.com/incident/cloud-
networking/19009)).

The networking incident looks like the one to follow for updates now.

------
wenbin
btw, Google Analytics realtime is down as well.

~~~
bbayer
I wasn't aware that outage and had small heartattack when I saw huge drop of
visitors. I think other metrics are also affected.

------
sdan
Weird for Twitter to still be up and fully functioning. I thought they
migrated everything to GCP this/last year?

~~~
sorenn111
Not the main functionality of the service, just lots of data analysis tooling.
nothing that end users would notice

~~~
sdan
Interesting. Thought I had read some posts of them migrating their data, but
you could definitely be right.

------
bluefox
You look up. A single ray of light has made its way to the earth. Some day,
you hope, the sky will be clear again.

------
fourty2
google services also f´ed up here in middle europe; cannot reach anything
google related from hamburg, germany.

------
codemati
[https://tour.golang.org](https://tour.golang.org) is also down

------
nmstoker
Seems to have coincided almost exactly with my Chromecast stopping displaying
my photos (in ambient mode).

------
skizm
Ah, wonder if this is affecting Google's SSO. It was _super_ slow when I was
trying it just now.

------
ramon
No wonder I was trying Webcodesk right now and it's not working it's all
firebase, yay!

------
ikeboy
So that's why YouTube was being weird. I thought it was an extension problem
or something.

------
f_martinez
We are on region us-east1 and our systems are still up. Specifically, we are
on us-east1-b.

------
RappingBoomer
youtube streaming is also down

------
bbayer
I think nobody mentioned but Adsense also is not updating for couple of hours.

------
jamisonbryant
I've noticed problems on GDrive (GSuite) and YouTube as well. Connected?

~~~
mindcrime
FWIW, Youtube is fine for me, but I'm seeing intermittent errors saving
updates to a document in GSuite. I had thought the latter error was a problem
with the wifi where I am, until I saw this. Now I'm not so sure. HN is loading
fine on the same wifi...

------
xerxes901
Couldn't load the support console to "me too" this one either!

------
noncoml
There go the nine nines for this year? Is it more like four nines or so now?

------
misrab
Anything to do with China?

------
atopuzov
Hug your on-call engineer.

------
CSDude
With Google Cloud incidents, most of the time whole regions fail, and with AWS
generally only a region fails. Of course there would be exceptions, but Google
Cloud does not make me feel safe as an outsider (and a user of multi-region
AWS)

------
chx
These things happen. That's OK. Here's what's not OK:

> We are investigating an issue with Google Compute Engine. We will provide
> more information by Sunday, 2019-06-02 12:45 US/Pacific.

The next update is at 12:59. Just ... no.

------
finphil
Working here but slow (I'm based in Central Canada).

------
sgammon
So far the Ko list:

GCE, GKE, BQ, Pub/Sub, GAE

asia-south1 us-west1 us-central1 us-west2

~~~
captn3m0
Our GKE stuff in asia-south1 (Mumbai) is up, and the GKE console works fine
here. (Bangalore).

------
griffinkelly
Looks like Google Analytics isnt reporting stats either?

------
CallumRobertson
Snapchat is fixed but snap maps is still disabled

------
bilal4hmed
gitlab is slow too

~~~
amenod
Slow is understatement... some pages on gitlab.com take minutes to load, and
jobs take tens of minutes to start.

EDIT: It's been like that since at least 12h ago though. Not sure if it's
connected to Google Cloud?

~~~
dsumenkovic
Yes, it was definitely related to GCP services.

GitLab is no longer seeing errors and Google Cloud has resolved the issue as
of 23:00 UTC yesterday. Any further information can be found on the issue at
[https://gitlab.com/gitlab-com/gl-
infra/production/issues/862](https://gitlab.com/gitlab-com/gl-
infra/production/issues/862)

~~~
amenod
The timeline seems a bit short, but ok. :)

(the problems with runners and the UI started at least at 2019-06-02 7:48 UTC,
though they were hit-and-miss at the time)

Still, happy this is solved and we can use the (fantastic otherwise) service
again!

------
stuck_in_the_ma
Google Play is also experiencing massive issues.

------
digaozao
I cant see any gke cluster in Brazil, or any VM.

~~~
miller_joe
I'm seeing that with northamerica-northeast1. I can't access anything over the
network in that region and most of the GKE clusters and VMs in that region
aren't listed in the console

------
sneak
A demo of “too big to fail [via antitrust]”?

------
CallumRobertson
Snapchat is back up but snap maps is down

------
sadris
So is Youtube.

------
javabean_
could this be the result of another BGP hack ? cyberwarfare ? I am just
speculating here big time.

------
murat124
GCP has been down since 11:50am and they acked it 35 mins later. They're great
at leaving their customers in the dark.

~~~
copperx
Not much different from AWS, from what I've heard.

~~~
numbsafari
Definitely the case. Neither are super great at this. One issue is that issues
that may 100% impact individual clients may only impact a vanishingly small
amount of their overall service load. That mismatch between customer and
provider experience is one of the ugly aspects of public cloud providers.

~~~
kazen44
Also, it's one which gets hugely understated when people "move to the cloud".

especially if you use your bussiness for B2B services. Stuff like this could
make you loose your bussiness, especially if some entity like google doesn't
communicate and as a result, you do not have a answer for your own customers.

Medium sized private cloud providers are a lot better at this, considering the
communication lines are a lot shorter.

~~~
zepolen
On the flip side - a customer is more willing to be understanding if 'Google
is down' instead of 'our server is down'.

------
eisvogel
gmail is down in Australia...

------
rwbhn
gmail also down/super slow atm for me (East Coast, USA)

------
pishpash
Let's see if perfect leetcode skills will save the day. /s

------
javabean_
vimeo.com is down.

------
the-dude
Huawei just flexing its muscles. Nothing to see here, move along.

------
bahmboo
No, not really.

------
maz1b
Ironically, I moved all of my objects off GCS today.

------
netmonk
There is a loop in the spanning tree.....

------
wenbin
forbes.com is down also?

------
stvkoch
My gmail it's down!

------
cobaimelan
seems like only us down

------
tus87
So that's why I can't login to YouTube this morning...

------
walshemj
just had rolld20 in the USA blow out a game I wonder if it is effected

~~~
zoidq
Looks to be working in the UK

~~~
walshemj
Network peering probs in Canada - looks like an unrelated problem

------
cjohansson
On days like these I’m glad I don’t use any of the affected services

------
captn3m0
What I've realized from this: Google doesn't have an official status page for
GCP. There are a few unofficial ones, but nothing official that I could find.

~~~
cheeze
Like this?

[https://status.cloud.google.com/](https://status.cloud.google.com/)

~~~
captn3m0
I meant Twitter Status Page (in my defense it was 3 am). The only ones are:
@gcpstatus and @gcp_incidents, both unofficial

