
The State of NAT Traversal - api
https://www.zerotier.com/blog/?p=226
======
simmons
This a nice summary; I was actually thinking about doing my own writeup of
this very thing. I have a few thoughts beyond what was covered in the blog
post:

For many protocols and modes of communication, UDP is really lousy for the
developer -- we end up re-inventing TCP (congestion control, retransmission,
etc.). Unfortunately, if we want to traverse NAT, we often must rely on UDP
hole punching and thus use UDP. There's a TCP hole punching technique called
"TCP simultaneous open", but it seems too flaky to expect to work in the real
world. I've been thinking about trying UDT [1] as a user-mode TCP-like
connection layer over UDP.

In addition to the ugly UPnP technique for asking a router to automatically
port-forward, there have been attempts to standardize a much simpler solution
in NAT-PMP and it's successor, PCP. (PCP even works for IPv6 stateful
firewalls.) Many people do have security concerns about this capability. I
don't know to what extent NAT-PMP/PCP has made inroads into the router market
-- it would be interesting to conduct some sort of survey.

[1] [http://udt.sourceforge.net/](http://udt.sourceforge.net/)

~~~
latitude
You don't need to reinvent TCP. You can simply put TCP over UDP. Works like
magic :)

TCP hole punching works just fine in practice, but it's certainly a part of
the "long tail" of empirical hacks that allow bumping NAT traversal rates into
95+% range. Detecting NATs that _decrement_ ports is another. Ports that are
incremented/decremented by more than 1 is yet another. And there's also UPnP
that gives you another percent or so.

I was doing this as a part of the Hamachi project [0], almost 10 years ago,
and this long tail was absolutely instrumental to being able to serve hundreds
of thousands of clients off just few measly Dell servers for $300/mo in
hosting costs.

[0] [http://swapped.cc/hamachi](http://swapped.cc/hamachi)

~~~
jon-wood
Its somewhat OT, but thanks for Hamachi! It made many a late night gaming
session possible in my teens and early twenties, and was always an
astoundingly simple piece of software to use considering everything it did.

~~~
latitude
My pleasure, glad to have been of some help :)

------
zurn
So true in the kittens dept. The worst part is that while this situation has
been developing, Stockholm syndrome has taken hold and half the battle is
convincing people to be receptive to having real IP addresses again! Few
people know the difference between a NAT box and firewall anymore, and
"router" now means NAT box to many people.

~~~
jmspring
In the case of many consumer/desktop machines, being behind a NAT is a very
useful first tier security barrier. Every machine being fully reachable -- say
via IPv6? I foresee many IT headaches and the Nortons/McAffees of the world
raking in more money.

~~~
zrm
NAT != firewall. You can have a firewall without NAT.

Probably the sensible thing to do for IPv6 is to make the default to block the
ports of specific applications people don't canonically expect to be visible
from the internet, like Windows file sharing or telnet or HP JetDirect, but
allow everything else so that VoIP and other P2P apps will all work properly.

~~~
Wilya
Of course you can do without a NAT. But the NAT has the advantage to make the
firewall rules trivial to write.

Put "DROP all packets with destination 192.168.0.0/16 coming from the external
interface", and a few other boilerplate rules of the same kind, and poof: you
just secured 99.9% of the NAT-ed networks. No configuration required.

To do the same without a NAT, you need at the very least to adapt the rules to
the IP addresses of the machines behind the firewall, which add complexity and
increase the likelihood someone will screw up the configuration.

And, hu, please don't open ports by default. That's just a recipe for
disaster. If apps really want to open a globally accessible port, make them
explicitly signal that to the firewall so it opens access.

~~~
Danieru
You know you could just drop all packets with destination of any valid address
coming from an external interface. This isn't BGP, you're not going to really
be routing the entire internet.

------
zrm
> Some NAT devices support various methods of intentionally opening ports from
> the inside. The most common of these is called Universal Plug-and-Play
> (UPnP).

UPnP should be avoided whenever possible. It's a hideous protocol. NAT-PMP
(originally from Apple, now RFC6886) is dramatically simpler and ideal for
small networks. Its successor NAT-PCP (RFC6887) is only slightly more
complicated in order to better deal with large enterprise networks.

------
yry4345
> "Lots of people think NAT is a show-stopper for peer to peer communication,
> but it isn’t. More than 90% of NATs can be traversed, with most being
> traversable in reliable and deterministic ways."

All the traversal methods require coordination with a 3rd party (ie:
centralized) server so - yes - this is a show stopper for P2P.

As public addresses become more scarce, and carrier NAT becomes common, the
problem of finding that intermediary will only get worse.

IPv6 should be a solution, but it won't get off the ground if carrier NAT gets
priority, for example. (Or if ISPs just put firewalls everywhere, and other
"best practices"...)

~~~
zrm
> All the traversal methods require coordination with a 3rd party (ie:
> centralized) server so - yes - this is a show stopper for P2P.

Third party and centralized aren't the same thing. Any peer with a real public
address or even a manual port mapping or a router that supports NAT-PMP or PCP
can play the role of the "server" in this context.

~~~
yry4345
In order to boot strap a connection to a P2P network, one must contact a well-
known server. It doesn't matter if the well known server is a "peer" that
happens to be running the same software, or if it's the BitTorrent DHT
bootstrap servers, what matters is that that peer has a disproportionate
amount of authority and influence over the network, amounting to a single
point of censorship or failure.

NAT (and firewalls anywhere except the networked computer - even at the
subscriber's router) contribute to/create this asymmetry, where one side has
to beg to connect to another, and so everyone winds up settling on the most-
memorable 3rd party. Everyone has heard of the king (or Verisign), and so the
rich get richer...

~~~
zrm
There is no requirement that it be a single node or that they all be operated
by the same party.

~~~
yry4345
It's not a requirement, it's just that it tends toward it naturally due to the
asymmetric addressability. As long as there are two or more global addresses
available to the public on which to run STUN, UPnP, etc, there will be
"competition" but it is immeasurably weak compared to what would be possible
with direct (non-NAT) addressability. In an environment without those
obstacles, systems are naturally designed in a P2P way - simply from the need
for scalability.

Case in point A: Skype leveraged an initial P2P design at a time where direct
addressability was the norm (and there were many freeware alternatives that
allowed direct dialing)... Now that Skype has become dominant, it has switched
to a centralized infrastructure 1) because it's owners can (it makes
administration, censorship, and surveillance easiest), and 2) because a P2P
model no longer makes sense with most users relying on their centralized
bootstrap servers.

Case in point B: Dropbox and similar services have replaced self-hosted FTP, I
would argue, simply because noone wants to maintain static port mappings and
Dropbox is easier.

Even without other incentives, the presence of NAT is a centralizing force
that - taken to the extreme (such as with carrier NAT) outright precludes P2P
- and that is undesirable. In an Internet with NAT (or any other violation of
the end-to-end principal) all systems suffer the same fate: centralization
(the antithesis of P2P).

~~~
zrm
Is your argument that we need to adopt IPv6? Because you'll get no
disagreement from me. But something has to be done in the meantime.

I guess I'm going to have to plug my software:
[http://trustiosity.com/snow](http://trustiosity.com/snow)

The idea is that it doesn't actually eliminate the horrors of NAT traversal,
it just makes it my problem instead of yours.

The current solution is to use other nodes as relays using DHT-style routing
and then I put a VM on AWS to bootstrap. The interesting thing is the
bootstrap peer is only required for the first connection. Once there is an
existing path A-B-C-D, it doesn't matter that zero of them have a public
address, you can still use it to send a hole punch message from A to D.

The real problem is that trusting random peers to relay messages allows them
to DoS you by filling up the network with Sybils and then not forwarding your
messages. So I'm in the process of coming up with solutions to that, probably
something along the lines of allowing particular nodes to be designated as
trustworthy and preferring those.

~~~
yry4345
Very cool.

Thanks for the link and thanks for taking a stab at a hard problem! Snow looks
very promising so far... (I can ping nodes on my LAN over it too, which is
usually a sticking point for traversal-oriented software - one is doubly-
NAT'd, and there's an SSH server with an Ubuntu banner reachable from afar
with UDP packets cutting through both brick walls nicely.) I'm impressed! :)

(FYI, building snow on a fresh Debian Squeeze 686-pae (packages: make g++
libssl-dev libminiupnpc-dev libnatpmp-dev) fails for me at dht.cpp line 220
(ambiguous function call) though; I'll have to read the source more to find
the right cast or ::namespace to fix it but it compiles fine on amd64 with an
identical set of packages.)

I'll definitely be reading the code more closely!

~~~
zrm
Thanks.

I can see the bug: The function is overloaded as taking uint64_t or a pointer
and I'm passing "0UL" to it, which on 64-bit is an exact match for uint64_t
but when 0UL is 32-bit it doesn't know whether to convert it to a uint64_t or
a NUL pointer. It probably just needs a cast to uint64_t.

------
captainmuon
> Another difficult situation arises if two peers are actually on the same
> local network behind the same NAT router.

...

> It might be tempting for peers to encode their private IP address and send
> it to the intermediate server and/or to the other peer. I thought about this
> when writing ZeroTier. On the plus side, this would work even on large
> segmented networks where UDP broadcasts don’t make it everywhere. But the
> problem is that this exposes internal configuration details about the
> network to potentially random external peers. Many network security
> administrators are not going to like that, so I decided against it. I tried
> to think of a way to anonymize the data but couldn’t, since the IPv4 private
> address space is so small that no form of hashing will adequately protect
> against exhaustive search.

I wonder if it would be possible to encrypt the local address with the
internal MAC address of the router/gateway. Then only nodes seeing the same
router could decode the information. I also wonder how Dropbox does this (it
can tell when two connected computers are on the same LAN, and transfers files
directly).

~~~
Maakuth
Dropbox does the UDP broadcast thing. Try dumping packet in pretty much any
LAN today and you see their broadcasts flying around.

~~~
nly
I see Dropbox traffic hitting my servers. People evidently rent Windows dedis,
install Dropbox on them, and they spew out broadcast to the local /24

------
cdoxsey
Actually this is entirely achievable right now, with very little code. You can
use WebRTC in chrome / firefox, which implements all the nasty details of
peer-to-peer communication for you (and uses TLS-esque security). All you need
is a STUN / TURN server and something to do the initial handshake and you're
off to the races.

I've not gotten around too it, but someone should setup a webserver that
speaks the protocol (perhaps with this:
[https://code.google.com/p/libjingle/](https://code.google.com/p/libjingle/))
then you serve your site as a static HTML file on AWS or similar, spin up
WebRTC, and then communicate with your server that way. Now you can serve a
whole website sitting behind a firewall.

------
2bluesc
If only I had someone else to talk to on IPv6 other than Gmail or YouTube.

+1 to Google for caring more than most and thanks to Comcast for giving me a
IPv6 /60 prefix delegation natively. :)

------
2bluesc
I think the vast majority of my IPv6 traffic is to sites who also care about
security (or care about networking in general?) and consequently use TLS where
possible.

Anyone know on any stats regarding popular sites reachable over IPv6? For
example, what percentage redirect the requests to TLS by default?

------
julie1
NAT may kill kittens, but IPv6 is killing polar bears massively.

I recommend the lecture of this:
[https://www.nanog.org/meetings/nanog46/presentations/Sunday/...](https://www.nanog.org/meetings/nanog46/presentations/Sunday/RAS_Practical_ipv6_61309_N46.pdf)
(deployment document of ipv6) then proceed to the glitches section.

I recommend reading nanog mailing list for having a glimpse on the added
complexity of dealing with IPv6 in real life.

Just peak random mails, it pops up once a month. Like for instance provider
turning to LS (Large Scale) NATing or Carrier Grade NATING rather than going
IPv6 for rationnale reason: IPv6 engineers are hard to find and LSNAT (+dual
stack eventually) since China has proven valid has both a way to handle the
scarcity of IPv4 addresses and the need to control user's traffic.

[http://mailman.nanog.org/pipermail/nanog/2014-July/068892.ht...](http://mailman.nanog.org/pipermail/nanog/2014-July/068892.html)

IPv6 is not bad, it is kind of bloated and requires a tinge more training than
IPv4. Plus there are the concerns about privacy (but some IPV6 guru will come
up with ULA or another obscure trick. It is IPv6 realm: everything is clear on
the paper but is seems confusing to quite a few real life engineers since it
requires a lot of RFC stacking in the brain)

~~~
api
> and the need to control user's traffic.

Some of us think that's a bad thing :)

> Plus there are the concerns about privacy

IPv6 has privacy extensions that allow address randomization, and its address
space is so large it makes it easy to just make up addresses. Besides,
tracking IPs is really only one of hundreds of ways of tracking you on the
Internet.

------
Zarathust
But as a side bonus, nearly every home computer now have a poor man firewall.
I find this to be insanely important for the global internet security.

~~~
anderiv
This doesn't change with the vast majority of residential IPv6 deployments.
Many people conflate having a public IP address with not having a firewall.
This is not the case at all. Yes, internal clients will get public, routable
IPv6 addresses. They are still behind a firewall, though, and you have just as
much control over ingress and egress traffic as you did before.

~~~
userbinator
The difference is in the isolation factor: there is absolutely NO way a packet
with an internal IP can get anywhere on the public Internet, and likewise a
packet with a public IP will never be accepted by a device that has been
configured with an internal one. The failure modes are different - although
firewalls can be configured to provide an "outgoing connections only" default
like NAT, they are also software and thus not immune to bugs; a buggy firewall
letting packets through which shouldn't be is far worse than a buggy NAT.

~~~
oasisbob
_there is absolutely NO way a packet with an internal IP can get anywhere on
the public Internet_

That very much depends on your definition of "public internet."

My service provider doesn't properly filter RFC1918 space at their border.
From my home on the west coast, I can hit 10.0.0.0/8 devices on the east coast
in a separate AS.

Is a reachable printer web interface 3000 miles away in New York on the public
internet? The "Internet" is a very tough entity to define succinctly, but I'd
argue yes.

~~~
graylights
Why would you need to filter an internal IP as a destination IP, routing
should handle that.

I understand filtering the source address as internal IP because that could be
malicious spoofing of internal host (and destination can't legitimately
reply.)

------
dxhdr
It's surprising that the number of non-traversable hosts is only 4-8%. Aren't
most cellular (3G/4G) users behind symmetric NAT, or is this considering only
PC users?

~~~
callesgg
All the cellar stuff I have used in the last 5 years have given me public open
ip addresses.

So i don't think that is the case.

~~~
dmm
As of at least summer 2013 Sprint no longer offers public ips with new phones.
It's all carrier-grade NAT. T-Mobile is the same but they provide ipv6, though
with just a single ipv6 address which doesn't make any sense.

I'm not sure about the other carriers.

------
JoeAltmaier
Its lots more complicated than ZeroTier would have you believe, especially for
Enterprise clients. That can add wrinkles due to VPNs, firewall filtering,
non-symmetric packet forwarding. And there's the case of two clients inside
the same subnet - to go P2P using the described algorithm requires something
called Router Hairpinning which has spotty support.

There are ways around all that, and Sococo supports them all. (Caveat: I work
at Sococo).

~~~
api
Two hosts in the same subnet is best achieved using LAN locator beacons that
allow direct connectivity without involving the server at all.

ZeroTier, Dropbox, and I think Skype all do that, though ZeroTier could do it
better (there's an issue to improve it).

~~~
JoeAltmaier
Sure; but again assumes a simple subnet configuration. Lots of enterprise
customers have multiple subnets with gateways, internal routers etc. They
probably won't respect/forward LAN locator beacons. We (Sococo) gave up on
that approach, particularly since Enterprise IT guys like to get excited about
broadcast UDP wandering around their subnet.

A straightforward approach is, when talking with the 'rendezvous' server, to
include in the message your discovered local subnet address(es). Then the
server can inform your peer of the entire set of candidate IP addresses you
might be available at (NATted address + subnet addresses). P2P connection
pinging then works the same way, with the additional constraint that the 1st
P2P message must include a unique id to identify the peer. Because non-
routable subnet addresses can be re-used (10.x.x.x or 192.168.x.x style). You
don't want to try to connect to Alice and discover that XRay has the same
subnet address, XRay is available on your subnet, and XRay is also running the
same P2P app.

Anyway like I mentioned, its complicated and full of wrinkles.

~~~
idlewords
The article takes up this very idea and dismisses it because it leaks info
about private networks.

~~~
JoeAltmaier
I guess they dismissed it too soon. None of our enterprise customers have a
problem with it. And some demand it - the alternative is to have their P2P
traffic travel outside the firewall and back in again, 'hairpinned' through
the public router. Talk about a security risk!

~~~
JoeAltmaier
And in our product, the sharing is over encrypted links to qualified peers
(members of your working group).

------
asdffdsajkl
I found this twitter post interesting - the dude did a writeup of how to use
metasploit when both the attacker and the victim are behind nat and you are
able to get code execution some how on the victim:
[https://twitter.com/b1tripper/status/383085600040947712](https://twitter.com/b1tripper/status/383085600040947712)

------
cornewut
I'm somehow not convinced IPv6 is the solution - someone somewhere will just
decide to nat IPv6.

~~~
login01
Cisco has been working on pushing NAT as a standard for IPv6 - Why? Sell's
more boxes and training!

~~~
mprovost
Or maybe their customers are asking them for it?

------
unwind
_" The terminology used is somewhat confusing… I’m not really sure what is
meant by a “cone.”"_

I think the linked-to Wikipedia page makes it pretty clear, it's simply a
reference to the conical (funnel-like) shape of the right-hand side of the
illustration
([http://en.wikipedia.org/wiki/Full_cone_NAT#Methods_of_port_t...](http://en.wikipedia.org/wiki/Full_cone_NAT#Methods_of_port_translation)).

Full-cone NAT allows inbound connections from any external host, i.e. it's an
unrestricted funnel (or cone).

------
IgorPartola
This is why I IPv6:
[https://www.google.com/intl/en/ipv6/statistics.html](https://www.google.com/intl/en/ipv6/statistics.html)

~~~
simmons
I'm also anxiously awaiting the rise of IPv6, but I'm guessing that -- in a
sane world -- consumer routers will still default to a stateful firewall for
IPv6, thus necessitating the continued need for hole punching. :/

~~~
richardwhiuk
Hole punching will be very reliable though - each end picks a port and starts
talking to the other end at roughly the same time. Once they both start
talking to each other, the flow will be fine.

------
maqr
> Ziggy then sends a message to both Alice and Bob. The message to Alice
> contains Bob’s public IP and port, and the message to Bob contains Alice’s.

Is this done by modifying the source address on the outgoing UDP packet? If
so, wouldn't modern filtering prevent the spoofed packets from getting to
their destination?

~~~
anemic
no, ziggy just acts as a directory of public ip addresses. Once alice and bob
know each other's public ip they can start communicating each other. Suppose
alice is at A.A.A.A:1234 and bob is at B.B.B.B:5678. Alice sends

A.A.A.A:1234 -> B.B.B.B:5678

and bob simulataneously sends

B.B.B.B:5678 -> A.A.A.A:1234

When the packets arrive to the routers the NAT already has an opposite rule
because of the sent packet. As this is UDP there is no SYN-ACK handshake or
other state tracking information in the packets and they traverse the NAT.

------
danbruc
In some situations Teredo [1] may get the job done.

[1]
[http://en.wikipedia.org/wiki/Teredo_tunneling](http://en.wikipedia.org/wiki/Teredo_tunneling)

------
lectrick
More like "Where the hell is IPV6 already?" :)

