
TCP Over IP Anycast – Pipe Dream or Reality? - r4um
http://engineering.linkedin.com/network-performance/tcp-over-ip-anycast-pipe-dream-or-reality
======
paulsutter
We had the same experience at Quantcast. Better results with anycast within
continents, and DNS to distribute a per-continent anycast addresses.

Some know-it-all busybodies like to proclaim that anycast can't work for TCP
because the route can change during the session, but the reality is that:

\- route changes are infrequent

\- individual sessions are typically much shorter than route change intervals

\- we never experienced proposed problems with sustained route flapping

\- route changes are readily detected and handled as (rare) dead connections.
The nicest bit is that transferred live connections can get rejected and
closed immediately by the new server allowing for rapid recovery (unlike DNS
changes which leaves things hanging for a long time).

We had extremely fast datacenter failover using anycast (100% of traffic moved
as soon as 5 seconds). Contrasted to the minutes-to-hours nightmare delays
that characterize DNS failover. (The DNS time-to-live is an advisory at best,
and outright ignored by many clients)

It's too bad more hosting providers don't allow route announcements.

~~~
sspies
I can fully agree with your experience using DNS failover vs anycast failover.
Made a video of how I implemented anycast
[https://www.youtube.com/watch?v=tsXpQHi7Udo](https://www.youtube.com/watch?v=tsXpQHi7Udo)

~~~
_ao789
Interesting video, thanks for sharing

------
Animats
What is LinkedIn sending that generates so much traffic that they need this?
They're not Netflix.

(Never mind, I just looked at LinkedIn, which I don't use much. With their new
layout, I had to scroll through three screens of ads to get to anything that
affected me.)

~~~
forty
This is to improve latency, not throughput.

------
davidu
Having done TCP Anycast for more than 8 years in production now, with
extensive testing, and third-party monitoring, there has never been an issue
with it.

When a route flap happens that causes an AS path selection change, the user
almost universally routes back to the exact same node they would have routed
to prior to the flap.

The theoretical "IT person in texas doing per packet load balancing on a
Sprint T1 to Atlanta and an ATT T1 to Denver" doesn't actually exist.

------
transitorykris
Sometimes you don't have as much control over a client (many non-HTTP TCP
applications). A middle ground is to use anycast for authoritative DNS and
return a set of records for unicast IPs local to that POP. You get the
benefits of anycast and the stickiness of unicast. It's a short jump to
overriding decisions to correct certain clients, or to shed traffic from
servers or POPs.

~~~
jzwinck
Doesn't this leave you with the problem mentioned in the article where some
users have DNS hosts rather far away from themselves?

------
jakozaur
Would be interested to read how CloudFlare solve flapping problem with their
Anycast network: [https://blog.cloudflare.com/a-brief-anycast-
primer/](https://blog.cloudflare.com/a-brief-anycast-primer/)

~~~
samwillis
I may be wrong but the impression I get is that CloudFlare have many many more
POP much closer to the visitor, quite often in the major consumer ISPs own
data centres. This should reduce the probability of what LinkedIn saw
happening.

------
CodingGuy
Most important - how much has this work improved linkedin's site performance?
30%? 40%? 50%?

~~~
BillinghamJ
And perhaps they should also compare that improvement with the improvement
garnered by turning off javascript on LinkedIn (assuming it even works).

~~~
_ao789
And all so that you can read about people talking about themselves and how
amazing they think they are. Oh, what game changers the linkedin users are..

~~~
stephengillie
It's just another echo chamber. Don't hate the players, hate the game.

------
sspies
We are working on this as well at [http://datapath.io](http://datapath.io).
You can use multiple iaas providers using global elastic IP addresses that you
can announce from multiple locations at the same time. If you want to
participate in our beta please write to beta@datapath.io

Best, Sebastian

------
mcguire
" _Anycast has historically not proven to be useful for stateful protocols in
the internet because of the inherent instability of the internet._ "

Inherent instability meaning packet-switched networking, a.k.a. the most basic
mechanism of how the internet works.

------
t0mas88
This is exactly how EdgeCast CDN (now Verizon) has worked for many years, down
to the ip per region approach. So this is known for a long time to be a good
way to do global content delivery.

I'm curious why linkedin didn't just use a commercially available CDN service
for this? Could save them a lot of time and maintenance, while probably
getting a better result because CDNs have this down to the details.

------
pyvpx
it has been a reality for many folks for years. real-world anycast can be
tricky, but it is certainly deployable and usable.

------
SixSigma
Plan9 can turn any protocol into TCP via 9p.

Connect to a remote plan9 machine by whatever means using 9p, import /net from
the remote machine into your local namespace, et viola you now have all the
network connections of that remote machine available, including the tcp stack.

~~~
detaro
What does this have to do with the article?

~~~
SixSigma
Any protocol over any protocol. Never a pipe dream.

~~~
detaro
Anycast is not a specific protocol, but about how (normally IP-)packets are
routed. Transporting protocols in other protocols has nothing to do with it
and doesn't help.

