Hacker News new | comments | ask | show | jobs | submit login
Internet Vulnerability Takes Down Google (thousandeyes.com)
394 points by doener 3 months ago | hide | past | web | favorite | 125 comments

It should be clarified the vulnerability was not on Google's end. It's the sort of thing we're used to by now: misconfigured/malicious BGP routes.

If google advertised several /24 for these ip networks instead of /19, this would have not happenned.

And if everyone advertised /24s instead of larger aggregate networks, routers would have to store about 12 million routes. Better yet, why not just advertise a /32 for every IP address?

Most routers would crash and the internet would stop long before we ever got to 12 million routes.

Some /24 are more critical than other. There is a price to security and for sure advertising only /24 is probably not viable.

The article mentionned that some networks within the /19, in this incident, were critical. So yes, I believe that every businesses should advertise /24 for their highly critical infra. They can advertise /19, /20 and so on for the less important networks. No need to use /24 for everything.

A valid point that I thought you might have meant after re-reading your post a couple of times.

Sorry for the quick-fire snark.

That's a lot of words to say that BGP is outdated.

How would an alternative deal with that problem ?

Is that a best practice though? Wouldn't the BGP tables get pretty big if everyone did that?

They would, and we would probably see a repeat of this: https://bgpmon.net/what-caused-todays-internet-hiccup/

Ugh, old Cisco trash

Didn’t Pakistan advertise a /28 at one point that blackholed YouTube?

Good times. (/s)

I believe there are routers that break at around 1M ipv4 entries, and we're at ~700k now.


Wow - it's really gone up in the last couple of years. I think it was at 550k last time I had access to an edge router.

Advertising every /24 within a /19 - would that take exactly 32 route advertisements?

You are correct.

24-19 = 5 bits. 2^5 = 32

Now that other ISP would advertise Googles /24's, it doesn't help.

It would have happened, and not even doing it as a mitigation would've worked, as the offending ISP would've re-advertised the same routes as google shortly after google changed it.

nitpick but this caught my eye:

TransTelecom (AS 20485) in Russia, China Telecom (AS 4809) in China and MainOne (AS 37282), a small ISP in Nigeria.

I don't know what qualifies as big ISP but I am certain MainOne with a 14,000 kilometre submarine fibre optic cable may not be one to be classified as a small one.

From wikipedia:

The Main One Cable is a submarine communications cable stretching from Portugal to South Africa with landings along the route in various west African countries.

"... countries with a long history of Internet surveillance."

Pot, meet kettle. Seriously, how is this even worth a mention? The US invented wholesale internet surveillance.

This sort of false equivalency is unhelpful. It requires a great lack of judgment to believe the U.S. is comparable to the regime overseeing Xianjing on domestic internet surveillance.

I assume the "US is comparable to China" assertion rubbed a few people the wrong way. This link collection shows the gravity of the situation in the US, and that US is drifting towards totalitarianism, without having reached that stage by a long shot. The surveillance scope is quantitatively different in the two countries, given the expressed totalitarian nature of the Chinese regime.

Something I ask myself all the time: how will I be able to tell when we've reached actual totalitarianism in the West? It won't be Chinese style totalitarianism, and I'm pretty sure they won't helpfully dress up in jackboots and announce things have reached peak totalitarianism. Would we notice at all?

I think the US is very big on internet surveilance and is probably beyond what China does. You don't have to be totalitarian to do surveilance, there's plenty of other reasons to do it (though once you have the data the temptation grows to use it for totalitarian uses)

It looks like you may be confusing surveillance with censorship.

It requires a small amount of context and history - and virtually no judgement.

You may not like the Chinese or foreign governments, but hacker news isn't a place to express that bigotry.

...and the internet.

...and wholesale.

This is true, and Im not in support of wholesale surveillance with out reasonable cause and the associated court documents. That said, I don't think I would want to go to Russia or China and say, google "How to overthrow the government". I would expect someone would want to speak with me pretty quickly after I did that...

>That said, I don't think I would want to go to Russia or China and say, google "How to overthrow the government". I would expect someone would want to speak with me pretty quickly after I did that...

That doesn't mean the US isn't performing the same level of surveillance/tracking, just that they aren't publicly picking up dissenters. In reality, if they were really worried, they'd hit you with a gag order and charge you in the FISA courts so you can't go public


I suspect that in every relatively developed country nowadays executing systematic requests of the kind you mentioned would get you some kind of personal attention.

That's why they invented it, right? To make sure the political status quo does not change. That includes political systems in the US, UK, China and Russia.

"However, this also put valuable Google traffic in the hands of ISPs in countries with a long history of Internet surveillance."

Curious what Google traffic might still be unencrypted now.

Mostly Google News, Google Maps and Youtube according to their Transparency Report https://transparencyreport.google.com/https/overview

If it was an isolated attack, the damage is probably nil, but if combined with a rogue certificate authority you can do some pretty in-depth spying.

who, America?

Hush, we don't talk about that.

Why do ISPs peer with China Telekom? BGP is built on trust, if a peer isn’t trustworthy why not drop them?

These routes can spill over many different paths. There is no central authority that would be able to drop them from the whole internet. Even if one Tier 1 ISP dropped them then their routes would just flow through another one. And good luck getting all of them to agree on that unless there's some massive, persistent abuse.

China and Nigeria are the types of state actors that might provide that sort of persistent abuse.

Two billion people living in an autocracy ... if Google still believed in Don’t Be Evil, this would not be acceptable.

That’s just a corporate motto, not an actual belief. It was, is, and always be a meaningless PR phrase as empty as any other.

It just goes to show the power of branding and how much people attach to it.

It wasn’t always that way. There were people (especially in operations) that used it as a guide for certain decisions. I was disappointed when the slogan was retired.

I think a lot of people at Google do take that seriously. Don't be so cynical.

Most people aren't evil, so that's not really special. Taking the statement seriously is the power of branding at work designed to drive employees.

On one hand, many people at Google do take it seriously. On the other hand, there are the people making decisions.

I don't think Google has control over peering agreements. The way I understand it, ISPs/backbone providers would have to decide to drop peers from their BGP configs. Outside of calling up ISPs, I'm not sure what Google should have done differently, if anything.

edit/minor correction: I guess they'd have control over peering agreements for initial hops for Google Fiber users, but that wouldn't have resolved the issue in this case, either, unless they stopped peering with other huge providers like NTT. Even then, it'd only help their subscribers.

Is it really going that far out on a limb to say that sharing the internet with China does not you “evil”?

What's unacceptable? Peering with Chinese networks?

It wasn't even CT that was the cause this time, they just propagated what they got from another ISP. There could certainly have filters in place but that only really works for stub networks.

China Telecom is the 2nd largest domestic broadband provider, and the 3rd largest mobile operator, in China.

Lots of people live in China.

China has also made it their goal to bifurcate the internet in order to prop up their authoritarian state. I don't see why the rest of the world should play along with that. It's frustrating in the extreme to see big Western companies repeatedly bow to the Communist party's wishes only to get all of their IP stolen by a domestic clone with the backing of the government.

> China has also made it their goal to bifurcate the internet in order to prop up their authoritarian state. I don't see why the rest of the world should play along with that.

China doesn't need anyone to play along to prevent their population from accessing the global internet. Right now, at least the rest of the world can access Chinese websites. A blanket ban on peering with Chinese ISPs would really bifurcate the internet. I'd count that as "playing along".

Then the solution is more connectivity rather than less, no?

Which 'big Western companies' got 'all of their IP stolen'?


To answer your question, most large companies have been subject to cyber espionage by the chinese military for several years. This accounts for the speed at which China built its manufacturing base in strategically important industries and its ability to rapidly copy products and the required tooling for their manufacture.

The Mandiant report I linked to above focused on tracking the activities of a single but prolific APT group responsible for hacks in 150 major companies worldwide for 7 years since 2006, and that they believe is actually a team of cyber espionage professionals from within the Chinese military.

Also just for fun: Anyone here drive a Range Rover Evoke? Heres China's version:


What you wrote after "To answer your question" doesn't answer my question.

Westinghouse Nuclear for one.

Which haven't? Even Google was hacked by the Chinese military.

BGP is a system build based on two things primarily in this order - shortest prefix to reach an IP ( takes precedence over ) - shortest number of "hops" measured by how many AS paths it takes to get somewhere ( if you have multiple ways to get to, the way with the last number of autonomous systems "AS" in it wins )

So the result is that you have to somewhat trust who you are peering with, and who they are peering with, etc. to the other end of the traffic.

A bad actor can make lots of chaos.

BBC reports that the error was with the Nigerian ISP, not China or China Telecom:


From someone who does full understand BGP, I have a question

Somewhere, a BGP route was misconfigured to send data somewhere else. What would happen if a BGP route was terminating at China, and the bad actor who made it happen, decided that they are not going to fix it and just leave it.

How would the rest of the BGP network deal with it?

While investigating the alarms, a network engineer at each major network will decide to stop taking routes from the Chinese network making the advertisements, and everything will sort itself out... as far as that network is concerned.

Just to clarify, it was a small Nigerian ISP that caused this, CT just propagated it to others. It's likely that this Nigerian ISP was setting up peering with Google and misconfigured their route policy. If you do this incorrectly you can advertise prefixes you get from one peer to other peers as if you own it. (essentially you replace the full AS path (prefix metadata in BGP) with your own AS number, it makes it looks like you originated the prefix to others) If neither the Nigerian ISP or CT refused to do anything then everyone that is their peer or customer would need to manually filter this "bad" prefix" to stop it. Customers and ISPs that only use CT would still be affected.

Just wanted to say that I find those visualisations pretty pleasant to see.

Agreed, this was a very good advertisement for their service.

After glancing at the headlines, I actually did a double take. I was like "whaa? Taboola creeping into my HN"

Given the audience, it wasn't the PR managers finest hour this week. What happens in BGP land is discussed publically on the NÀNOG mailing list, and they are the friendliest crowd ever.

I've read them go out of their way to solve issues that just needed Goodwill to do in a couple of minutes, keeping the back channels open even between companies whose rivalry would dictate that they'd talk to each other only via their law firms.

1K@@ will probably be laughed at in the list

Reminder that China has some big investment in Nigeria, they even built and operates a railway there.

China built the headquarters of the African Union (and then filled it floor to ceiling with bugs: https://qz.com/africa/1192493/china-spied-on-african-union-h...)

I’ve been around the world to see some amazing/strange investments by China. Whole office buildings in Malaysia, South Africa, Costa Rica, etc that sit empty because a state-backed investor is speculating that this area will someday be more valuable.

No where was this more obvious than Malaysia which is their neighbor... but still crazy to see giant brand new office parks that look abandoned.

To your point, it doesn’t at all surprise me about Nigeria because I saw it in ZA.

China seems to have a world wide plan for the next century.

It's the belt and road initiative.


If it's not a government project, it could also be capital flight. I think it's one of the main drivers of real estate price expansion around the world. There's concern in China that things could collapse in the next 10 to 20 years. Many people seem to think that losing most of your real estate investment is better than losing ALL of your money.


> China seems to have a world wide plan for the next century.

If their government doesn't collapse first.

Xi's apparent desire to become president for life is certainly a liability, but other than that, China comes across as being quite stable.

China has big investments everywhere, also in the US and Europe.

This could explain why there's so little information coming out from Google as to what caused the outage. China Telekom is state-owned and thus pointing a finger at them could stir the relations with the Chinese government, regardless whether this was a bug or an intended act.

The official line[1] is that it was very likely accidental and not malicious.


Except it's repeatedly BGP leak accident from China Telecom.

From the article, it wasn't China Telecom who made the initial misconfiguration - it was a small ISP in Nigeria. China Telecom just accepted their BGP update and rebroadcast it.

So China telecom would have been able to fix it real fast? Hmm

I get that they need a cool sounding title but they could have just said "BGP hijacking" and saved a click for everyone who knows what that is. We've seen this before and we'll see this again.

They don't want to save you a click.

Does anyone know if there was any relation of this attack and the Facebook outage that happened yesterday as well? Seems weird that both FB and Google have trouble on the same day.

No idea if there is correlation but something is definitely not being discussed.

Ironic that the company reporting this has security problems themselves. I was trying to download their cloud research pdf via google and spammers have gotten hold of them with hundreds of thousands of fake links: https://www.google.com/search?q=site%3Athousandeyes.com+file...

This went by on HN a few days ago:

"Strange snafu misroutes domestic US Internet traffic through China Telecom"

> China Telecom, the large international communications carrier with close ties to the Chinese government, misdirected big chunks of Internet traffic through a roundabout path that threatened the security and integrity of data passing between various providers’ backbones for two and a half years, a security expert said Monday. It remained unclear if the highly circuitous paths were intentional hijackings of the Internet’s Border Gateway Protocol or were caused by accidental mishandling.



Ugh... that headline is abysmal.

It's just a BGP hijack. Get over it.

No, it’s actually quite accurate. BGP hijacks are a vulnerability in the very core fabric of the internet. Just because we’ve seen a lot of BGP hijacks recently, does not in any way, shape, or form downgrade their severity.

What's particularly maddening is BGP hijacks have been happening for well over a decade. It's one of the weakest parts of the Internet infrastructure and I'm convinced something more disastrous is going to happen any time now, either by accident or malice.

If it can be done once, I guess it can be done multiple times. I don't know nearly enough about this to understand how BGP hijacks work or why they are possible. Can anyone point me to a simple explanation for a layman?

BGP is a gossip protocol. BGP roughly works like this:

As BGP nodes come online, they establish connections with "nearby" existing BGP nodes, saying what version and how frequently they'll check in, and what their AS number is (a unique identifier)

Once communications have been established, then they can start to report any network routes they know about.

"I'm AS 123456, and I am the originator for" (i.e. they're responsible for it), "Also, I can reach with a cost of 6" (A fair number of hops away on the network).

Any neighbouring BGP peers update their routing table:

"AS 123456 is responsible for, and I can reach it with a cost of 1, and I can also reach with a cost of 7 via AS 123456". If there's a cheaper route to a network address, no changes will happen.

Routing changes can propagate quite quickly across the internet. The routing protocol is nice and lightweight, and updates are happening with reasonable frequency, as network connections come and go.

The idea is that should damage occur to the network fabric, the network will automatically update and route around it, without need for any intervention.

It's entirely built on trust, though. You have to trust that AS 123456 is indeed actually responsible for

If you get two parties indicating responsibility for a network range, it's possible to end up with routing loops etc, as things get in to a mess.

What is legitimate behaviour, though, is for, say, AS 123456 to be the originator for, and another AS be the originator for (i.e just 254 addresses under that space). That's not an unusual situation, and it won't cause routing issues, because more specific is taken as a priority over less specific, rough analogy: "In general mail for General Electric, should be sent here, but if it's for the electronics product division, send it straight to them"

There have been different attempts to put filtering in place, provide authentication "Yes, AS 123456 is allowed to be responsible for" and the like, but nothing has really taken off.

Sounds like a bad actor could easily attract as much traffic as they wanted by claiming to have the shortest route to all sorts of places.

With different data snooping/data protection policies in various countries, it would also be useful if you could order your traffic to avoid certain countries.

Absolutely. ISPs, backbone providers etc can all manually filter out updates from specific ASs, and in theory habitual bad actors will have that happen to them. Filters will get put in place even as a temporary measure during incidents sometimes, depending on how responsive / capable the NOC is.

Why isn’t there a “premium” commercial Internet, similar to DOD NIPRnet, for b2b stuff and for communications to vetted infrastructure? A random business on the Internet (eyeball, not servers) cares a lot more about routes to AWS and Google than to a bunch of other eyeballs, particularly in foreign countries.

You would still want full connectivity, but when things went wrong I could lose connectivity to a Nigerian ISP without critical business risk, but losing access to Google sucks. You could largely accomplish this by prefixing the hell out of everything except the “special” connection, and ensuring DDoS and other security filtering could drop the entire normal network if needed without affecting internal or special.

If the extremely slippery slope of this idea is not immediately obvious to you, then I don't even know what to say.

As succinctly as I can put it, this is exactly the opposite sentiment of Net Neutrality.

Internet2 basically provides this service for US research organizations.

With two completely separate networks, the concept of net neutrality doesn't even apply.

You can purchase things like AWS Direct Connect.

Only really at colos. What scares me is businesses with key business partners, satellite offices, etc who just rely on the Internet working normally to communicate between them. If someone like L3 sold a premium network it would be worthwhile.

Isn't that MPLS, essentially?

Yeah, you could accomplish this for hq to satellite office over private circuits, MPLS, etc. That wouldn’t be enabled by default. If a carrier made a point of on-net connections being protected by default, it would probably be easier for more businesses to adopt.

Why not change the title to something like "internet vulnerability blocked access to Google"?

Such thing relatively easy to avoid https://www.youtube.com/watch?v=CSLpWBrHy10 Also problems like that are significantly more rare for IPv6.

Can anyone explain to me what exactly it means that BGP routes "leaked"?

By default, Cisco and Juniper routers (and likely others too, those are just the ones I can speak of) will advertise all bgp learned routes to all bgp peers unless you specify a route map/export policy.

Leaking a route would imply you’re advertising the route to a peer that you’ve otherwise not intended to, either due to a misconfiguration or by not configuring things at all.

most likely these routers in Nigera were configured to send traffic destined to Google through China first ( a more specific /24 out of the Google prefix was configured -> China Telecom ASN ), through a route-map mistake this was advertisied out to the public internet instead of staying within the Nigerian ISP ASN.

Now that's how you do content marketing! Great original research and I definitely know Thousand Eye has great BGP tools :-P

Could this be a not so subtle middle finger from China to the US? (considering the recent political turmoil)

Is this related to the GCP issues people were posting about this last weekend?

How much private information were foreign entities able to collect from this?

Given its prominence in the article 'BGP' should be defined.

Or is this just an advert?

It's a legitimate incident that Google didn't explain all that well beyond "it wasn't us": https://status.cloud.google.com/incident/cloud-networking/18...

OP showing off their tools contributes to the article, so I don't mind

China is punishing Google for not making the censored search engine?

Despite the routes terminating at China's firewall I don't think it's a good guess to say this was necessarily a Chinese attack. It seems like the interference of the Chinese firewall was simply a side-effect of the new path the network traffic was going. If you were perpetrating this attack it seems more advantageous to re-route the traffic and allow it continue working rather than discoverably interfering with the traffic. Plus if you were going to try to block the route, there's probably a better way of doing it.

China's firewall has a long history of accidental enforcement of censorship outside its jurisdiction, leaked DNS and BGP poison are the most common problem.

Sooo... paint me stupid but I find the title completely misleading and another, very, VERY annoying thing caught my eye - after reading the text, I caught myself repeating the word "thousandeyes" and it annoyed me to no end. Then I re-read the text and I found that you can kick that word out of every 2nd paragraph and the text would still have merit.

Is this yet another marketing ploy where you post something with purposely-misleading title in order to attract traffic? I don't like the fact that word "thousandeyes" got stuck in my head, nor do I like the fact that I got clickbaited. This one is going to my list of "just like every other site since 2015, don't click".

It's called content marketing. Some do it better than others but it is always trying to attract attention to a company or service by blogging about related content.

> I find the title completely misleading

Why? BGP has always been insecure and repeating that fact doesn't make it any less so.

> Sooo... paint me stupid

why would somebody paint you stupid upon reading your totally spot-on observation? Even in case of not agreeing.

The talk about that clickbait made me click and see whats all the fuss about

Haha. I just refuse to click on anything like that. Much like with Flash, if they want me to look they'll have to use something proper.

It's SEO, they're always trying to produce content with the keywords on each paragraph.

Repeatedly including your company name, especially if it’s something unique like “thousandeyes” is essentially pointless when it comes to SEO. They own the domain thousandeyes.com. Google is going to recognize the intent for searches for thousandeyes are going to be about that company.

Other keywords, yes, those are important for SEO. But repeating your unique company name, not so much.


^ cryptocurrency marketing ^

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact