
Internet Vulnerability Takes Down Google - doener
https://blog.thousandeyes.com/internet-vulnerability-takes-down-google/
======
ahmedalsudani
It should be clarified the vulnerability was not on Google's end. It's the
sort of thing we're used to by now: misconfigured/malicious BGP routes.

~~~
john37386
If google advertised several /24 for these ip networks instead of /19, this
would have not happenned.

~~~
sathackr
And if everyone advertised /24s instead of larger aggregate networks, routers
would have to store about 12 million routes. Better yet, why not just
advertise a /32 for every IP address?

Most routers would crash and the internet would stop long before we ever got
to 12 million routes.

~~~
john37386
Some /24 are more critical than other. There is a price to security and for
sure advertising only /24 is probably not viable.

The article mentionned that some networks within the /19, in this incident,
were critical. So yes, I believe that every businesses should advertise /24
for their highly critical infra. They can advertise /19, /20 and so on for the
less important networks. No need to use /24 for everything.

~~~
sathackr
A valid point that I thought you might have meant after re-reading your post a
couple of times.

Sorry for the quick-fire snark.

------
yesplorer
nitpick but this caught my eye:

TransTelecom (AS 20485) in Russia, China Telecom (AS 4809) in China and
MainOne (AS 37282), a small ISP in Nigeria.

I don't know what qualifies as big ISP but I am certain MainOne with a 14,000
kilometre submarine fibre optic cable may not be one to be classified as a
small one.

From wikipedia:

The Main One Cable is a submarine communications cable stretching from
Portugal to South Africa with landings along the route in various west African
countries.

------
TheRealDunkirk
"... countries with a long history of Internet surveillance."

Pot, meet kettle. Seriously, how is this even worth a mention? The US
_invented_ wholesale internet surveillance.

~~~
Niten
This sort of false equivalency is unhelpful. It requires a great lack of
judgment to believe the U.S. is comparable to the regime overseeing Xianjing
on domestic internet surveillance.

~~~
okmokmz
It seems naive to believe that the US isn't comparable to China when it comes
to surveillance considering the actions taken and information that has come
out, particulary since 9/11

[https://www.aclu.org/issues/national-security/privacy-and-
su...](https://www.aclu.org/issues/national-security/privacy-and-
surveillance/nsa-surveillance)

[https://theintercept.com/2018/06/25/att-internet-nsa-spy-
hub...](https://theintercept.com/2018/06/25/att-internet-nsa-spy-hubs/)

[http://thefreethoughtproject.com/national-id-
hr4760-biometri...](http://thefreethoughtproject.com/national-id-
hr4760-biometrics/)

[https://www.theverge.com/2018/1/26/16932350/ice-
immigration-...](https://www.theverge.com/2018/1/26/16932350/ice-immigration-
customs-license-plate-recognition-contract-vigilant-solutions)

[https://theintercept.com/2018/01/19/voice-recognition-
techno...](https://theintercept.com/2018/01/19/voice-recognition-technology-
nsa/)

[https://www.zdnet.com/article/us-cell-carriers-selling-
acces...](https://www.zdnet.com/article/us-cell-carriers-selling-access-to-
real-time-location-data/)

[https://theintercept.com/2018/06/25/att-internet-nsa-spy-
hub...](https://theintercept.com/2018/06/25/att-internet-nsa-spy-hubs/)

[https://freedom.press/news/revealed-justice-depts-secret-
rul...](https://freedom.press/news/revealed-justice-depts-secret-rules-
targeting-journalists-fisa-court-orders/)

[https://www.eff.org/nsa-spying](https://www.eff.org/nsa-spying)

[https://www.eff.org/cases/first-unitarian-church-los-
angeles...](https://www.eff.org/cases/first-unitarian-church-los-angeles-v-
nsa)

[https://www.eff.org/cases/jewel](https://www.eff.org/cases/jewel)

[https://www.eff.org/cases/hepting](https://www.eff.org/cases/hepting)

[https://www.eff.org/cases/smith-v-obama](https://www.eff.org/cases/smith-v-
obama)

[https://apnews.com/d69a8e6db867477795f4152d0511bbf9](https://apnews.com/d69a8e6db867477795f4152d0511bbf9)

[https://www.washingtonpost.com/news/posteverything/wp/2018/0...](https://www.washingtonpost.com/news/posteverything/wp/2018/01/25/how-
to-fight-mass-surveillance-even-though-congress-just-reauthorized-
it/?utm_term=.ced1b673b9fa)

[https://www.reuters.com/article/us-usa-security-records-
fact...](https://www.reuters.com/article/us-usa-security-records-
factbox/factbox-history-of-mass-surveillance-in-the-united-states-
idUSBRE95617O20130607)

edit: Why the downvotes with no response despite my comment being entirely
relevant? If anyone would like to explain/counter I would be curious to hear
their reasoning

~~~
stareatgoats
I assume the "US is comparable to China" assertion rubbed a few people the
wrong way. This link collection shows the gravity of the situation in the US,
and that US is drifting towards totalitarianism, without having reached that
stage by a long shot. The surveillance scope is quantitatively different in
the two countries, given the expressed totalitarian nature of the Chinese
regime.

~~~
scottlocklin
Something I ask myself all the time: how will I be able to tell when we've
reached actual totalitarianism in the West? It won't be Chinese style
totalitarianism, and I'm pretty sure they won't helpfully dress up in
jackboots and announce things have reached peak totalitarianism. Would we
notice at all?

------
tyingq
_" However, this also put valuable Google traffic in the hands of ISPs in
countries with a long history of Internet surveillance."_

Curious what Google traffic might still be unencrypted now.

~~~
miemo
who, America?

~~~
taneq
Hush, we don't talk about that.

------
whitexn--g28h
Why do ISPs peer with China Telekom? BGP is built on trust, if a peer isn’t
trustworthy why not drop them?

~~~
Aic1kuir
These routes can spill over many different paths. There is no central
authority that would be able to drop them from the whole internet. Even if one
Tier 1 ISP dropped them then their routes would just flow through another one.
And good luck getting all of them to agree on that unless there's some
massive, _persistent_ abuse.

~~~
apathy
China and Nigeria are the types of state actors that might provide that sort
of persistent abuse.

Two billion people living in an autocracy ... if Google still believed in
Don’t Be Evil, this would not be acceptable.

~~~
manigandham
That’s just a corporate motto, not an actual belief. It was, is, and always be
a meaningless PR phrase as empty as any other.

It just goes to show the power of branding and how much people attach to it.

~~~
eloff
I think a lot of people at Google do take that seriously. Don't be so cynical.

~~~
manigandham
Most people aren't evil, so that's not really special. Taking the statement
seriously is the power of branding at work designed to drive employees.

------
Geenirvana
From someone who does full understand BGP, I have a question

Somewhere, a BGP route was misconfigured to send data somewhere else. What
would happen if a BGP route was terminating at China, and the bad actor who
made it happen, decided that they are not going to fix it and just leave it.

How would the rest of the BGP network deal with it?

~~~
dsr_
While investigating the alarms, a network engineer at each major network will
decide to stop taking routes from the Chinese network making the
advertisements, and everything will sort itself out... as far as that network
is concerned.

------
d33
Just wanted to say that I find those visualisations pretty pleasant to see.

~~~
vermilingua
Agreed, this was a very good advertisement for their service.

------
hlecuanda
After glancing at the headlines, I actually did a double take. I was like
"whaa? Taboola creeping into my HN"

Given the audience, it wasn't the PR managers finest hour this week. What
happens in BGP land is discussed publically on the NÀNOG mailing list, and
they are the friendliest crowd ever.

I've read them go out of their way to solve issues that just needed Goodwill
to do in a couple of minutes, keeping the back channels open even between
companies whose rivalry would dictate that they'd talk to each other only via
their law firms.

1K@@ will probably be laughed at in the list

------
elorant
This could explain why there's so little information coming out from Google as
to what caused the outage. China Telekom is state-owned and thus pointing a
finger at them could stir the relations with the Chinese government,
regardless whether this was a bug or an intended act.

~~~
phreack
The official line[1] is that it was very likely accidental and not malicious.

[1][https://arstechnica.com/information-
technology/2018/11/major...](https://arstechnica.com/information-
technology/2018/11/major-bgp-mishap-takes-down-google-as-traffic-improperly-
travels-to-china/)

~~~
a012
Except it's repeatedly BGP leak accident from China Telecom.

~~~
TheGrumpyBrit
From the article, it wasn't China Telecom who made the initial
misconfiguration - it was a small ISP in Nigeria. China Telecom just accepted
their BGP update and rebroadcast it.

~~~
raarts
So China telecom would have been able to fix it real fast? Hmm

------
dsfyu404ed
I get that they need a cool sounding title but they could have just said "BGP
hijacking" and saved a click for everyone who knows what that is. We've seen
this before and we'll see this again.

~~~
i_cant_speel
They don't want to save you a click.

------
cschmidt
Does anyone know if there was any relation of this attack and the Facebook
outage that happened yesterday as well? Seems weird that both FB and Google
have trouble on the same day.

~~~
Allvitende
No idea if there is correlation but something is definitely not being
discussed.

------
fewiron9
Ironic that the company reporting this has security problems themselves. I was
trying to download their cloud research pdf via google and spammers have
gotten hold of them with hundreds of thousands of fake links:
[https://www.google.com/search?q=site%3Athousandeyes.com+file...](https://www.google.com/search?q=site%3Athousandeyes.com+filetype%3Apdf+cloud)

------
carapace
This went by on HN a few days ago:

"Strange snafu misroutes domestic US Internet traffic through China Telecom"

> China Telecom, the large international communications carrier with close
> ties to the Chinese government, misdirected big chunks of Internet traffic
> through a roundabout path that threatened the security and integrity of data
> passing between various providers’ backbones for two and a half years, a
> security expert said Monday. It remained unclear if the highly circuitous
> paths were intentional hijackings of the Internet’s Border Gateway Protocol
> or were caused by accidental mishandling.

[https://arstechnica.com/information-
technology/2018/11/stran...](https://arstechnica.com/information-
technology/2018/11/strange-snafu-misroutes-domestic-us-internet-traffic-
through-china-telecom/)

[https://news.ycombinator.com/item?id=18403999](https://news.ycombinator.com/item?id=18403999)

------
ccnafr
Ugh... that headline is abysmal.

It's just a BGP hijack. Get over it.

~~~
vermilingua
No, it’s actually quite accurate. BGP hijacks are a vulnerability in the very
core fabric of the internet. Just because we’ve seen a lot of BGP hijacks
recently, does not in any way, shape, or form downgrade their severity.

~~~
mcv
If it can be done once, I guess it can be done multiple times. I don't know
nearly enough about this to understand how BGP hijacks work or why they are
possible. Can anyone point me to a simple explanation for a layman?

~~~
Twirrim
BGP is a gossip protocol. BGP roughly works like this:

As BGP nodes come online, they establish connections with "nearby" existing
BGP nodes, saying what version and how frequently they'll check in, and what
their AS number is (a unique identifier)

Once communications have been established, then they can start to report any
network routes they know about.

"I'm AS 123456, and I am the originator for 3.0.0.0/0" (i.e. they're
responsible for it), "Also, I can reach 4.0.0.0/8 with a cost of 6" (A fair
number of hops away on the network).

Any neighbouring BGP peers update their routing table:

"AS 123456 is responsible for 3.0.0.0/8, and I can reach it with a cost of 1,
and I can also reach 4.0.0.0/8 with a cost of 7 via AS 123456". If there's a
cheaper route to a network address, no changes will happen.

Routing changes can propagate quite quickly across the internet. The routing
protocol is nice and lightweight, and updates are happening with reasonable
frequency, as network connections come and go.

The idea is that should damage occur to the network fabric, the network will
automatically update and route around it, without need for any intervention.

It's entirely built on trust, though. You have to trust that AS 123456 is
indeed actually responsible for 3.0.0.0/8.

If you get two parties indicating responsibility for a network range, it's
possible to end up with routing loops etc, as things get in to a mess.

What is legitimate behaviour, though, is for, say, AS 123456 to be the
originator for 3.0.0.0/8, and another AS be the originator for 3.0.1.0/24 (i.e
just 254 addresses under that space). That's not an unusual situation, and it
won't cause routing issues, because more specific is taken as a priority over
less specific, rough analogy: "In general mail for General Electric, should be
sent here, but if it's for the electronics product division, send it straight
to them"

There have been different attempts to put filtering in place, provide
authentication "Yes, AS 123456 is allowed to be responsible for 3.0.0.0/0" and
the like, but nothing has really taken off.

~~~
mcv
Sounds like a bad actor could easily attract as much traffic as they wanted by
claiming to have the shortest route to all sorts of places.

With different data snooping/data protection policies in various countries, it
would also be useful if you could order your traffic to avoid certain
countries.

~~~
Twirrim
Absolutely. ISPs, backbone providers etc can all manually filter out updates
from specific ASs, and in theory habitual bad actors will have that happen to
them. Filters will get put in place even as a temporary measure during
incidents sometimes, depending on how responsive / capable the NOC is.

------
rdl
Why isn’t there a “premium” commercial Internet, similar to DOD NIPRnet, for
b2b stuff and for communications to vetted infrastructure? A random business
on the Internet (eyeball, not servers) cares a lot more about routes to AWS
and Google than to a bunch of other eyeballs, particularly in foreign
countries.

You would still want full connectivity, but when things went wrong I could
lose connectivity to a Nigerian ISP without critical business risk, but losing
access to Google sucks. You could largely accomplish this by prefixing the
hell out of everything except the “special” connection, and ensuring DDoS and
other security filtering could drop the entire normal network if needed
without affecting internal or special.

~~~
basilgohar
If the extremely slippery slope of this idea is not immediately obvious to
you, then I don't even know what to say.

As succinctly as I can put it, this is exactly the opposite sentiment of Net
Neutrality.

~~~
kbirkeland
Internet2 basically provides this service for US research organizations.

------
amelius
Why not change the title to something like "internet vulnerability blocked
access to Google"?

------
doener
An WSJ article about that: [https://www.wsj.com/articles/google-internet-
traffic-is-brie...](https://www.wsj.com/articles/google-internet-traffic-is-
briefly-misdirected-through-russia-china-1542068392)

------
betaby
Such thing relatively easy to avoid
[https://www.youtube.com/watch?v=CSLpWBrHy10](https://www.youtube.com/watch?v=CSLpWBrHy10)
Also problems like that are significantly more rare for IPv6.

------
Quanttek
Can anyone explain to me what exactly it means that BGP routes "leaked"?

~~~
RKearney
By default, Cisco and Juniper routers (and likely others too, those are just
the ones I can speak of) will advertise all bgp learned routes to all bgp
peers unless you specify a route map/export policy.

Leaking a route would imply you’re advertising the route to a peer that you’ve
otherwise not intended to, either due to a misconfiguration or by not
configuring things at all.

------
soonbesleeping
Relevant:
[https://youtu.be/VVJldn_MmMY?t=924](https://youtu.be/VVJldn_MmMY?t=924)

------
i_phish_cats
Now that's how you do content marketing! Great original research and I
definitely know Thousand Eye has great BGP tools :-P

------
the_duke
Could this be a not so subtle middle finger from China to the US? (considering
the recent political turmoil)

------
jayd16
Is this related to the GCP issues people were posting about this last weekend?

------
Raphmedia
How much private information were foreign entities able to collect from this?

------
bdefore
Given its prominence in the article 'BGP' should be defined.

------
Wiretrip
Or is this just an advert?

~~~
wongarsu
It's a legitimate incident that Google didn't explain all that well beyond "it
wasn't us": [https://status.cloud.google.com/incident/cloud-
networking/18...](https://status.cloud.google.com/incident/cloud-
networking/18018)

OP showing off their tools contributes to the article, so I don't mind

------
luckylittle
China is punishing Google for not making the censored search engine?

~~~
slivym
Despite the routes terminating at China's firewall I don't think it's a good
guess to say this was necessarily a Chinese attack. It seems like the
interference of the Chinese firewall was simply a side-effect of the new path
the network traffic was going. If you were perpetrating this attack it seems
more advantageous to re-route the traffic and allow it continue working rather
than discoverably interfering with the traffic. Plus if you were going to try
to block the route, there's probably a better way of doing it.

~~~
bcaa7f3a8bbc
China's firewall has a long history of accidental enforcement of censorship
outside its jurisdiction, leaked DNS and BGP poison are the most common
problem.

------
furicane
Sooo... paint me stupid but I find the title completely misleading and
another, very, VERY annoying thing caught my eye - after reading the text, I
caught myself repeating the word "thousandeyes" and it annoyed me to no end.
Then I re-read the text and I found that you can kick that word out of every
2nd paragraph and the text would still have merit.

Is this yet another marketing ploy where you post something with purposely-
misleading title in order to attract traffic? I don't like the fact that word
"thousandeyes" got stuck in my head, nor do I like the fact that I got
clickbaited. This one is going to my list of "just like every other site since
2015, don't click".

~~~
gawkface
The talk about that clickbait made me click and see whats all the fuss about

~~~
vehementi
Haha. I just refuse to click on anything like that. Much like with Flash, if
they want me to look they'll have to use something proper.

------
cauldron
Reminder that China has some big investment in Nigeria, they even built and
operates a railway there.

~~~
SlowRobotAhead
I’ve been around the world to see some amazing/strange investments by China.
Whole office buildings in Malaysia, South Africa, Costa Rica, etc that sit
empty because a state-backed investor is speculating that this area will
someday be more valuable.

No where was this more obvious than Malaysia which is their neighbor... but
still crazy to see giant brand new office parks that look abandoned.

To your point, it doesn’t at all surprise me about Nigeria because I saw it in
ZA.

China seems to have a world wide plan for the next century.

~~~
yesforwhat
> China seems to have a world wide plan for the next century.

If their government doesn't collapse first.

~~~
mcv
Xi's apparent desire to become president for life is certainly a liability,
but other than that, China comes across as being quite stable.

