
Ask HN: Is Google Compute down? - hellcow
I&#x27;m not able to ssh into any of my boxes or access any of the sites, yet my status monitor isn&#x27;t showing downtime. Spotify is also down for me, which is another GCP customer.<p>I&#x27;m in Los Angeles but the servers are hosted on us-central1
======
sethvargo
Hi all - Seth from Google here. Our team is aware and we are working on
mitigation. In short, a third party telco provider is advertising on one of
our IP blocks. Unfortunately that's all the information I can share at this
time.

~~~
konschubert
EDIT: This is a general statement, I am not complaining to google here.

This kind of thing should not be possible. Are there any protocol proposals or
other kind of upgrades to the routing protocols that would prevent these kind
of mistakes/attacks?

~~~
cm2187
I am surprised how fragile is the internet given how our society is
increasingly becoming critically reliant on it.

~~~
zzzcpan
On the other hand it's not that fragile everywhere and for everyone. When ISP
markets are not monopolized and the service doesn't rely on a big cloud - much
fewer users will get rerouted through random countries and the service itself
can failover to properly working datacenters, tolerating all those BGP
misdesigns.

It's if the internet doesn't like all that centralization with all that market
domination. It's naturally resilient only when there is a lot of competition.

------
regnerba
Google IPs seem to be being routed to China for us.

We have servers in San Jose that cannot access Google services. Trace route
shows everything going to China when leaving the San Jose data center. We can
access the same services from Vancouver just fine.

~~~
docker_up
How many times does this have to happen before China's privileges to do things
like this get revoked? At this point, it can't be just a mistake and must be
some state-sponsored hacking. Seems like a great way to find out where a
particular Spotify user's IP address is.

~~~
QML
What are you going to do? Divide the internet in half? I say that in a joking
way but it’s a possibility.

~~~
tinus_hn
You don’t need to cut the internet in half to limit China to routing IP
adresses that are allocated to China.

~~~
QML
IIRC, Google can try to dictate a BGP policy that says not to accept any
routes that goes through China. However, without any verification checks (via
cryptography), an entity can lie about the path that they are advertising.

~~~
tinus_hn
No, the peers just outside of China can choose to reject advertisements of
Google by China. Google can’t do much.

------
jamalex
Despite the subdomain, the IP for ChinaTelecom-gw.transtelecom.net
(217.150.59.249) seems to be based in Russia, as does the carrier:
[https://en.wikipedia.org/wiki/TransTelekom](https://en.wikipedia.org/wiki/TransTelekom)

~~~
sterlind
Seems likely to be TT's gateway to CT. New theory: TransTelecom brought up a
new gateway to ChinaTelecom, which incorrectly gossiped all advertisements
from ChinaTelecom. This caused a leak, since CT has bgp highjacking of Google
IP ranges for the GFW within China, but ordinarily doesn't leak them outside
the country. TransTelecom misconfigured the gateway to broadcast everything
advertised by ChinaTelecom, bringing external traffic into the GFW.

~~~
londons_explore
I doubt the GFW uses BGP to route traffic to it.

It needs to filter traffic to _any_ address, and wouldn't have specific google
ranges configured.

------
xolox
Reading through the comments here I'm recognizing "China Telecom" from an
article on a BGP hijack that was published about a week ago, I still had the
article open in my browser:

[https://arstechnica.com/information-
technology/2018/11/stran...](https://arstechnica.com/information-
technology/2018/11/strange-snafu-misroutes-domestic-us-internet-traffic-
through-china-telecom/)

In another comment in this thread I read:

> Seems like its time to start or accelerate a working group on secure BGP.

Indeed things can't go on like this for much longer...

~~~
faissaloo
I kept getting SSH bruteforce attempts from IPs on China Telecom a while back.
Wonder what they're up to...

~~~
viraptor
These are botnets. You're going to get ssh bruteforce attempts from every
country. I wouldn't read into it too much.

------
davismwfl
I am on the East Coast, in Florida and seeing the same thing with traffic
heading to China, lots of "chinatelecom-gw.transtelecom.net" in traceroutes I
have never seen prior.

~~~
scrollbar
Getting this as well in SF. transtelecom.net WHOIS says they're Moscow-based

~~~
davismwfl
Yea, just saw that same thing.

Definitely something interesting going on, and I am sure no shortage of some
frantic research and effort to resolve this all at Google and such right now.

------
CydeWeys
We urgently need a solution for routing traffic to IP addresses that is better
than BGP.

~~~
dasm
Agreed. This appears to be a repeat of the attack covered here:
[https://news.ycombinator.com/item?id=18385920](https://news.ycombinator.com/item?id=18385920)

I'm not familiar with BGP routing attacks; the article above seems to imply
the attacker needs to compromise certs in order to glean useful data from the
attack.

If that's accurate, is this Google-oriented traffic vulnerable to this type of
attack?

~~~
raesene9
for Google traffic, assuming certificate pinning is in place, I can't see this
being that successful.

However for more general traffic, well look at the trusted root list in your
browser/OS. Realise that every single one of those trusted routes can issue
certificates for a given domain...

~~~
dasm
Thanks. Since it appears all this traffic is Google-related, any guesses as to
what the attacker could have gained here?

~~~
raesene9
This could just be a mistake of course, malicious intent isn't needed :)

Of the top of my head, assuming malicious intent, well not all browser
(especially older ones) do certificate pinning, so perhaps then Chinese users
of Google services using old browsers would find their traffic being
intercepted?

Past that the leakage would seem fairly minor, a list of source IP addresses
and destination hosts.

------
aviv
Funny, a day after I posted this...

[https://news.ycombinator.com/item?id=18429099](https://news.ycombinator.com/item?id=18429099)

Is our first time actually rolling over the entire stack to AWS - and it
worked!

GCP outage currently is massive, can't even use other regions.

Edit: This also affected AWS Oregon region earlier. I do not know how yet, but
they too were unreachable briefly. Seems to be okay now.

------
infogulch
So... what's the current state of a secure BGP? I feel like this in the top 3
security threats to the whole of the internet.

------
TodayIsTheDay
Does anybody else have chinatelecom-gw.transtelecom.net [217.150.59.249] in
the traceroute for www.google.com

~~~
kacy
Also showing up on a traceroute to spotify.com for me.

~~~
leetbulb
Gitlab.com as well, earlier.

17 195.219.156.146 (195.219.156.146) 152.490 ms 152.423 ms *

18 * * mskn17ra-lo1.transtelecom.net (217.150.55.21) 198.658 ms

19 * * Google-gw.transtelecom.net (217.150.44.9) 192.230 ms

20 * * 108.170.250.111 (108.170.250.111) 172.086 ms

------
dickfickling
yeah, GCP is having a serious outage. Our site is down, so's Pivotal Tracker

Edit: We're also in Los Angeles, connecting to us-central1. Seems to be a
pattern?

~~~
hellcow
Bugsnag's app.bugsnag.com is down as well.

------
syogi
I'm in Los Angeles and I can access my GCP Console but I can't access Google
services like google.com or Maps or Gmail.

EDIT: Some services are intermittently responsive. I had ~5 minutes of no
access to anything. Some are slowly coming back.

------
fxdoublecute
we manage services deployed in every GCE region, and our monitoring in London
is reporting every GCE region having intermittent connectivity. no problems
with our services in the other major clouds (we use basically all of them)

------
vamos_davai
I have trouble accessing YouTube. I live in Sherman Oaks (a town of Los
Angeles).

------
gsibble
Can confirm many IPs are being sent to China.

~~~
crunchlibrarian
This is nuts.

------
cobookman
There's a current BGP prefix hijacking issue currently being mitigated.

------
bifrost
I hate to break it to everyone, but the technology to filter this sorta thing
has existed for a very long time, but people often don't use it. Most of the
time this sort of thing is accidental (IE: operator error)so a lot of
operators kinda ignore it. Check out "IRR Power Tools" if you're interested.

------
jamalex
Same thing here in San Diego. Traceroute to spotify.com going through LA, San
Jose, NY, London, Amsterdam, Frankfurk, "mskn17ra-lo1.transtelecom.net", then
ChinaTelecom-gw.transtelecom.net.

------
RayHawk
I'm getting the same thing. Servers are in us-east1 and tracert is ending at
chinatelecom-gw.transtelecom.net [217.150.59.249]

------
xstephen95x
seems like [https://arstechnica.com/information-
technology/2018/11/stran...](https://arstechnica.com/information-
technology/2018/11/strange-snafu-misroutes-domestic-us-internet-traffic-
through-china-telecom/)

------
johnnyballgame
YouTube and Spotify unresponsive here.

------
linuxbuzz
World War III on the Internet front!

------
jetforme
I'm in LA and seeing similar routing through chinatelecom-gw.transtelecom.net

------
kenhwang
Might be related to: [https://status.cloud.google.com/incident/cloud-
networking/18...](https://status.cloud.google.com/incident/cloud-
networking/18018)

Also in LA, had intermittent issues with google.com and Spotify all morning.

edit: linked to wrong issue

~~~
fxdoublecute
unlikely, IMO. this is a routing problem, either a deliberate attack or else a
gigantic screwup

------
dejaime
Well, my instance seems to be working

------
ntq
We are able to connect to some of our services via LTE but not a local wifi
network

~~~
ntq
Seeing the same China routes as mentioned on this thread

