Can anybody help? Am I simply not technically competent enough to consume this article yet?
Most companies' networks have edge routers (which sit at the points where they connect to other networks) and core routers (which manage the flow of traffic inside the network. All these routers basically use a standard protocol called BGP (Border Gateway Protocol) which is defined by RFC 4271.
However, BGP was still designed from the view of individual machines making routing decisions and announcing routes to each other that collectively make up the whole Internet. This helps the Internet as a whole be quite resilient – if one network goes down, there are still ways to route traffic through to other networks. Also, since the protocol is standard, you can swap out one vendor's gear for another at will (in theory anyway) as long as you know how to configure it correctly.
But this leads to some inefficiency – for instance, it is very hard to say that a path with fewer hops will lead to lower latency. What Google seems to have done is to make their edge routers into one single "intelligent" network, where the edge routers don't make routing decisions on their own, but feed their data into a central server. This central server can then say stuff like "My peering router in NYC seems to be under heavy load, let me redirect some of my traffic to NYC destinations through the NJ datacenter instead", or something to that effect; while still doing the correct BGP announcements from the point of view of Level3 or whoever is peering.
In short, they built their internal network from the ground up since they are so big they can afford to build custom routing gear instead of using the standard off-the-shelf, standardized setup that a small or medium-sized company uses. The network consisting of their custom edge routers (all the green blobs) together is called Espresso and represented by a light grey circle.
AT&T used to try to avoid centralization, but ended up with routing controlled from Bedminster, NJ. An interesting comment from AT&T's NOC tour guide is that load doesn't vary much any more. AT&T used to have holiday calling surges and such, but now, in an always-on world, overall load is relatively steady.
Distributed things though, mesh networks, IPFS - that's what's giving me hope.
I think Google gets more freedom to try out some of these techniques because people still fundamentally think of them as a website (apart from Google Fiber, they don't serve end-users directly); whereas AT&T, being an ISP, is treated more like a water / power service, in that people expect them to be working by default, and going down is absolutely unacceptable.
What you don't want is software-defined networking where every new flow goes to Master Control for validation and routing. Some SDN systems do that, and they have a central point of failure and censorship.
Nearly every Google paper about Google networks in SIGCOMM is utter trash that would have been rejected in an instant due to lack of novel details if it had come from anyone else. They just brag about scale, utilization, and resiliency without actually giving anything back to the academic community.
Note: This does not apply to all Google papers (e.g. spanner and other DB ones are pretty good). Their networking publications seem to be crippled by either IP concerns or poor academic writers.
The essence of the article is the evolution of Google's SDN(software defined networking) architecture. They have added SDN to their edge whereas previously SDN existed mostly in their Core it sounds like. The Edge SDN architecture being "Espresso" and the Core SDN architecture being "Jupiter."
There is also some fishing about what exactly is Google's network hardware that runs all this.
The following article is far more brief but far more informative in my opinion:
This is a white paper about Jupiter mentioned in the article and infographic:
And there's some good resources here for just learning about SDN here:
I can't really search for Espresso, or Jupiter, or Andromeda without additional qualifiers that I may not know yet.
The names in order of popularity when it was originally announced (now they push for WSL)
1. Bash for Ubuntu on Windows
2. Windows Bash Shell
3. Windows Subsystem for Linux (WSL)
Trying googling WSL - it's world surf league. Windows Bash Shell is impossible, so is Bash for Ubuntu on Windows. It's getting better as more and more articles are being written, but god almighty. I wouldn't be surprised if google is lending a helping hand with searches for WSL stuff.
E.g. having windows 1st sort of inflates the 'primacy' of windows with respect to linux; it is linux that is being made compatible with windows rather than the other way around.
Best things that come to mind are things from the windows UI - like the network privacy zones, having the 'basic' and 'advanced' control panel, etc.. Some things say 'windows is updating your computer' rather than 'your computer is updating windows' or some such... To me they have this sort of psychological undertone of 'always remember how microsoft is helping you make this confusing thing much easier.. you couldn't do it without us'
a. Windows' Subsystem for Linux
b. Window's Subsystem for Linux
For the possessive form of Windows I would look at the Microsoft Manual of Style, which for me is the fourth edition.
Page 184: Possessive Nouns
Do not use the possessive form of Microsoft. Do not use the possessive form of other company names unless you have no other choice. And do not use the possessive form of a product, service, or feature name. You can use these names as adjectives, or you can use an of construction instead.
the Windows interface
Microsoft products, services, and technologies
That is correct (or, at least, nearly universally agreed to) for possessives of plural nouns ending in “s”, style guides are mixed when it comes to other nouns ending in “s”, though the most common rule seems to be to use “’s”; while “windows” is plural, “Windows” as the name of the operating system is a proper noun that is not treated as plural.
Q: It's a subsystem? What kind of subsystem?
A: It's a Windows subsystem?
Q: A subsystem for what?
A: It's the Windows subsystem for Linux.
As another commentator mentioned, there's probably a missing possessive ("'s"), but even if we take the proper name to be more of an adjective modifying/qualifying "subsystem", it's a Windows subsystem, it's purpose is running Linux... the Windows Subsystem for Linux.
Cygwin allows running GNU software on Windows, but it doesn't implement the Linux ABI. Therefore Cygwin requires software recompilation.
The big advantage of WSL is that you can run native linux binaries.
For decades, Richard Stallman has been calling it GNU/Linux, because the OS “aside from the kernel” was GNU, and Linux is the kernel. https://www.gnu.org/gnu/linux-and-gnu.en.html
Now, we have a Microsoft system that runs the Linux binaries, which are GNU according to Stallman, but not running them on the Linux kernel. You’re naming the entire system after the one component that is missing. By the same logic that normal Linux should be GNU/Linux, Linux containers on Windows should be GNU/Windows.
Despite that, it was still impossible to search for. Look for Ruby, get gemstones. Look for gem, get even more gemstones. Eventually Google figured it out and now it's almost impossible to search for gemstones. If I search for gem almost all search results are about Ruby things. Same when searching for Ruby.
Even "rails" now returns almost only Ruby on Rails stuff.
But yea, worth bearing in mind when naming stuff.
1. Download Ruby
2. Ruby entry on wikipedia (about the gemstone)
All the rest on the first page are about the programming language in some way or the other (Rails, various books, various tutorial resources, Stackoverflow, trending on Github)
...then again, WINE isn't exactly unique either.
That said WSL tends to work if you pair it with whatever your issue is.
IMO the most ridiculous name in this whole story is their github repo, Microsoft/BashOnWindows. I mean, bash has worked fine on Windows for decades, and it doesn't require an entire subsystem at all :-) WSL isn't half bad compared to that.
when i see someone mention ES6, I think its elasticserch but its not
Using common words for names is also ancient and inevitable. In the modern age there is a practical argument against it -- as made up nonsense is more googlable. But people making up names don't really care about that in any visceral way.
Especially if they take something with a somewhat awesome name. It waters the meaning down. My go-to example is Terraform, a glorified configuration manager that has absolutely nothing to do with the process of terraforming.
The cleverness here is that they effectively turned their routers in label switchers, with the label traffic is sent to determined by backend infra, not the router itself. They still have to run this over BGP of course, as thats how the internet works but it is an interesting approach for optimizing return traffic routes.
Edit: espresso is the name of the stack they use to make the label switch decision, its not a protocol or industry standard
I agree, article could be improved.
In a nutshell, Espresso system allows (logically) centralized, fine-grained control of how bits leave Google's network. Specifically, each server has a tailored (for that server) routing table that allows the server to select a specific port on a specific peering router for a given packet to exit to. The routing tables are updated very often and fast (e.g. to remove references to links that went down).
This makes it very easy to move bits around. You want to shift 42.2% of link X traffic to link Y? Just update some routing tables in end hosts. It also allows for different applications to the same destination to use different exit links (e.g. low latency - exit somewhere close, high bandwidth - exit where we have bandwidth). Bonus benefit: router can be simpler, it doesn't need to do any routing anymore, it just needs to follow instructions on each packet on which port to forward the packet to.
Does it make it more clear? Ask me anything.
There are definitely some Googlisms or something being thrown around though. Can anyone tell me what a "hypercaler" is?
Probably a typo for https://www.hyperscalers.com/
It will never be sufficient. A good backbone infrastructure doesn't compensate for the fact that the majority of users don't have ISP choices especially for fast speed fixed/mobile networks.
Such a mind blowing statement. Wonder when (if) they'll hit one-in-three bytes.
That said, those bytes still need to be delivered, Google really don't have peers in this stuff, apart from maybe Facebook.
So Google should get to 1 out of 3 pretty quickly.
My 7yo son is obsessed with the whole Let's Play phenomenon, and other associated minecraft content. I've looked up some of these people on Social Blade, and some of these people are likely making millions, but I can't fault the system. While the content of some isn't exceptional, it is professional, and some of these people keep up that pace for multiple content releases each day.
It's quite scary actually what the current and future generations are being conditioned to do.
It's true that you can binge on a series if you are new to the series (youtube or otherwise) but if you are up to date that is no longer the case.
I don't think I could have resisted the compulsion to be on YouTube for all of my free time. I am lucky that there is a ton of social pressure on me to spend my time doing other activities.
It could be a personal problem, unique to me and a small amount of other people.
'peak download internet traffic' whatever that means.
I think you misread that article with the 90%. That is talking about the proximity of Open Connect CDN nodes to customers, and how much traffic is served from "directly connected" Open Connect CDN nodes, and how much traffic is ... less than directly connected OpenConnect CDN nodes.. (sorry, I'm a kernel guy, not really versed in WAN stuff).
Of course, you could add enough coding theory to make it work.
Could they exist without the US market, Maybe, I don't know. But I can't imagine they want to go without it.
I guess if Netflix ever switched from AWS to Google Cloud Platform...
He also mentioned that in NZ miserly data caps are really not implemented any more, since people just aren't using expensive transits as much (any NZ people confirm?)
Of course you wouldn't know about any of these savings by following the pricing of internet services in Australia. And the government seems intent on making anyone who cares for even basic privacy needing to tunnel their entire connection outside Australia (not to mention content providers and their geoblock rules).
People often mistakenly associate the main cost of ISPs is BW. But it's the network equipment, the constant upgrading/replacement, an army of support (both physical and phone), and the last mile connectivity is the major costs. BW itself is a small part of the cost.
Getting both Akamai and Netflix cdn has similar issues, but are harder to qualify/get than GGC cache hardware.
Yes, all these CDN providers are providing us hardware and mostly excellent support in helping to set them up - in most cases, it's setup and forgets. But for them, it's a one time cost, but for ISPs it monthly recurring cost that we can't dump on users without losing customers, we can't charge the CDN providers, we just have to silently bear the cost and hope that in 15 years if we have enough customers, we might be profitable someday.
Everyone loves to hate ISPs, and I am sure a lot of it has some justification. But sometimes the cost and pain are real. That's why there is so little competition in this sector because it involves a huge amount of upfront and recurring investment and really long waiting time before you can hope to finally make some money. Then you have brain-dead regulations from the government (not all countries of course) and aggressive competition from well-connected existing companies with deep-pocket.
There is a reason why Google fiber doesn't roll out all over the country, because even with all their money cache and network expertise and well connection throughout the government - what a daunting task it will be for them to ever be profitable. Your Ad sales can only subsidize your last mile cost for so long.
"BBR: Congestion-Based Congestion Control" http://queue.acm.org/detail.cfm?id=3022184
All 3 are ways to connect your private datacenter or on-premise location (maybe your corporate office) directly to the cloud provider's network over a fast private link, like an industrial-sized VPN. This way you can access your cloud servers/services over this private connection rather than the public internet. Lots of companies do this if they have sensitive data or have their own "cloud" system running VMWare or something and want to augment with public providers.
This article is discussing lower-level infrastructure describing Google's global network components, from load-balancing, routing, internal datacenter connections to servers, etc.
Let me know if I'm way off base in my understanding.
You can access all Google Cloud services over this link and but you can also change your routes to point all internet traffic over this link too (you'll need some cloud VMs with public IPs to be the exit nodes). Not sure what you'd gain from doing that though.
Do you know if such a VPN is fundamentally different from something like OpenVPN, which can also be used site-to-site?
Espresso is a system for controlling BGP peering using application-fed metrics. It's basically complicated source routing.
Couldn't ExpressRoute (and not familiar with AWS VPC) in theory be configured at multiple global locations to handle internet traffic?
This quote stands out for me and makes me uneasy.
And while I agree about overcomplicated routers and box-centric thinking in computer networks, it's pretty much impossible to change things because of the monopolistic nature of the ISP industry. They are very far from competing on the levels of quality where SDN could matter.
This has pushed the decision back from the routers to the servers, where it scales with the number of hosts handling connections and can additionally be determined per /session/ instead of per packet. That state is held where sessions already need to be stored anyway instead of being duplicated in a router where it's entirely a burden.
Google still buy and use a ton of traditional mega (tera) high bandwidth networking gear. All of which will support regular label switching. But that doesn't sound cool and exciting so they always play that down. They can spin it that way if they like, network hardware vendors don't care because they are still selling them their boxes.
There is this danger that these pieces Google work on get joined up, e.g. Android apps on your Chromebook. With Google owning the network from the server 'encapsulated' over public networks and then on Google's own home router and on to Google gadgets like Chromebooks, there is a high likely hood that some glorified 'SPDY' that hooks in at OS level is going to happen. Therefore pieces of the puzzle may not have any obvious benefit, to take a train analogy, who would build a railway through Utah? On its own it might not make sense but as part of a transcontinental railroad it was an important part of the puzzle to get done. With Google there is some of this shrewd aspect with added 'moonshot' thrown in for good measure.