
Anycast TCP - fanf2
https://bill.herrin.us/network/anycasttcp.html
======
mitchs
In practice the "split path" scenario is only a problem when topology changes.
Routers typically use a hash of source and destination port, source and
destination IP address, and sometimes ingress port id to pick which of N equal
paths to take.

This is done not just to preserve end destination, but so every sequential
packet in a flow follows the same path, thus must arrive in the same order.
(Different paths have different latencies because of distance and buffer depth
on the routers.) Many tcp implementations will back way off as soon as they
see out of order packets, so network providers must do this.

~~~
userbinator
_Routers typically use a hash of source and destination port, source and
destination IP address, and sometimes ingress port id to pick which of N equal
paths to take._

...and sometimes even the TTL, as evidenced by this recent news item:
[https://news.ycombinator.com/item?id=17893594](https://news.ycombinator.com/item?id=17893594)

~~~
mitchs
Interesting, thanks for the link. Note to self, customers don't keep
consistent ttls... Yikes.

------
dagenix
The title is misleading - it makes it sound like this is a description of how
anycast works. However, this appears to be a proposal for how anycast could be
implemented. But, its not clear to me that it's actually implemented this way
in practice.

A big chunk of the proposal seems to involve maintaining state between
distributed servers - which is hard to do at all and even harder to scale.
There also seems to be some layering violations - if a transaction is going to
be too large, this proposes using an HTTP redirect. I might have missed it,
but, this is hard to do when working at the TCP layer. The redirect would have
to be done to a non-anycast address. Which means that every server needs to
know about all the other potential servers that are running.

------
jiveturkey
Interesting! with a few flaws. Out of order quoting:

> An Anycast TCP system can be achieved without any modifications to the TCP
> protocol.

Yet he describes a system that requires modification.

> With protocols that support redirects (like HTTP) ... use the initial
> anycast connection to determine which site is closest to the client and then
> send a redirect instructing the client to connect to a unicast address

> QUIC packets returned from the server to include a unicast address

These techniques already exist, patented by Google. It's not dissimilar from
anycast in ATM networks, which is where I believe anycast originated.

> Experiments with Anycast DNS show that in 99%+ cases Anycast devolves to
> unicast.

Too bad he doesn't say what experiments, but also he should have analyzed from
this that the primary (and only practical?) use of anycast is for wide area
gslb and HA. So let's not talk about toy implementations and lets talk only
about true global scale impact.

At this scale, he fails to acknowledge the difficult of sharing state
globally. He should make more of a point that only connections which exceed a
the high water mark within their local sequence number space need to have the
ownership info flooded. He needs to comment on the need for authentication of
this data (otherwise you have the makings of a synflood-like attack). He fails
to recognize the difficulty of draining connections and bringing up new nodes,
while others still operate and may have to have their sequence number space
re-partitioned.

It's too bad he doesn't discuss how well (or poorly) this could be handled
with client-side support (like http redirects, but at the TCP layer). Whilst
his approach does require stack changes, at least it's only server side.

He seems to think that DSR (as implemented by LVS in his example) performs
better than than termination by the load balancer. He's off on his knowledge
by 5 years if not a decade.

