Hacker News new | past | comments | ask | show | jobs | submit login
How NAT Traversal Works (tailscale.com)
162 points by signa11 on Aug 22, 2020 | hide | past | favorite | 28 comments

The article (like many others) distinguish between easy NAT (endpoint-independent) and hard NAT (endpoint-dependent symmetric NAT), but does not discuss how real-world NAT devices behave (whether they are predominantly easy NATs or hard NATs).

In reality, home routers are generally Linux-based and share netfilter NAT implementation. And how this implementation behaves? Well, it is a symmetric NAT masquerading as a port-restricted cone NAT. While it is symmetric NAT, it tries to reuse internal port as external port, and therefore often ends with the same port on different NAT mappings to different external machines.

Therefore, NAT traversal often works by accident in simple cases, but suddenly stops to work when it becames more complicated (e.g. when two clients behind a NAT try to connect to the same server).

Linux netfilter has an option to randomize ports in NAT mappings to avoid this appearance of working, but it is unlikely that home routers allow to configure this.

The article makes the distinction, because it's a useful empirical distinction to draw :). While you're correct that linux is, strictly-speaking, an endpoint-dependent implementation, it goes to sufficient effort to reuse mappings in an endpoint-independent way that you can do useful things against it by treating it as an EIM NAT.

Similarly, commercial systems like JunOS try to stick to EIM behavior for as long as they can, and only shift to EDM behavior when load makes it infeasible to do otherwise. There again, strictly speaking that means you should abandon all hope... But in practice, NAT traversal still works much more often than not, and we can use that to build useful systems while we wait for IPv6 adoption to continue.

This is the kind of content that makes me visit hn, long & insightful

I got lost at the point where an easy Nat connects to a hard one. Ok, the hard one can reach the easy one with the info obtained from the stun server, but why does the easy one have to do port scanning on the hard one? if the easy one receives the packet from the hard one, just send a packet back to the source IP:port? the hard one should have created an entry in its Nat map and firewall at that point since it just sent data to the easy one. What am I missing?

It's not the NAT part on the easy side, it's the firewall on the easy side that needs to be open.

Even if the easy side knows (through STUN) that its public address is, that doesn't mean it'll accept any packets addressed to

For it to accept packets from e.g., it first needs to send a packet there, to create the connection table entry in the firewall.

But... it doesn't know what port is going to be sending from (since that's the whole point of having a "hard", i.e., destination-dependent-mapping, NAT), so it can't preemptively send a packet to open the firewall.

At least, this is my understanding.

Author here. You are correct, the challenge in an easy/hard pair is the stateful firewall built into the easy side. Even thought the _NAT_ is easy (we can discover our public ip:port), the firewall still wants to see transmissions to/from the correct ip:port on the hard side in order to let the traffic through. The hard side defeats that by not letting us discover what the correct ip:port should be, so we can't punch through the firewall component.

So, we're left with two options: make one of the NATs vanish via the port-mapping protocols, or use the birthday paradox to find a usable ip:port in reasonable time.

Thanks to both of you. So it would only work the way I assumed if the easy one is that full cone type that doesn't have a firewall at all.

Yup, exactly right. In the article, I tried to call those "trivial" NATs, because from a NAT traversal POV they might as well not exist. You do need to do a tiny bit of work to get them up and running, but once that's done, you effectively have an open port on the internet that anyone can connect to.

That was my understanding as well.

What I got was that "hard" NAT is hard because neither peer is able to obtain the hard NAT's public IP/Port pair: The peer behind the hard NAT can't ask a STUN server because that server would see a different pair while peer behind the easy NAT can't receive any packets.

I think it's interesting that the peer behind the hard NAT could actually produce a packet that would contain all necessary information by sending something to the easy NAT's address - but neither peer could access this packet. You'd have to call the NSA to monitor the packet in-flight...

I wonder if you could somehow fiddle with the max-hop header to have the packet be sent back to the hard NAT's peer though, sort of like traceroute works.

You're on the right lines, yeah. On the wire, out on the internet, there is a packet with all the information we need to get through the easy/hard pair, but we can't access it.

According to the author of dublin-traceroute, it used to be that you could use ICMP TTL Exceeded errors to your advantage, because buggy NATs didn't translate ICMP payload (which contains the IP+UDP header of the packet whose TTL hit zero).

IOW, you could send a probe packet to the easy ip:port, with a TTL low enough that the packet expires somewhere between the two NATs, and listen for the returning ICMP error. Its payload would contain the public ip:port for your session on the hard NAT, which effectively gives you an accidental STUN server that gives you correct answers, even for a hard NAT.

I _believe_ this trick no longer works reliably, because the "NAT Behavioral Requirements" family of RFCs started laying down rules for what well-behaved NATs should do, and afaict, inexplicably, vendors listened. REQ-4 of RFC 5508 says that NAT devices MUST translate the payload of ICMP errors according to the appropriate NAT mapping, to preserve the end device's illusion that no NAT exists. So, these days, the ICMP error you get back contains the unhelpful LAN source ip:port :(

I still want to experiment with that some more, because if it's unusual but not unheard of... Well, it's another technique to grab a little bit more of the long tail :)

> and afaict, inexplicably, vendors listened

That got a good laugh out of me.

That idea sounds really clever, but even if NATs wouldn't mangle the ICMP reply, don't you need elevated privileges to receive it in your application? And I wouldn't be too surprised if a bunch of devices would just drop it entirely.

Another thing that just came to my mind: Do any of those hard NATs assign the outgoing port in a predictable pattern for some inexplicable reason? I'd guess no, since those already seem to be security focused and not randomizing the port sounds stupid, but you never know... Would be a cool last resort for hard/hard.

"it depends". There used to be NATs that did linear allocation or some other weird schemes (e.g. linear, but matching the lower bit of the WAN port to the LAN port, for reasons to do with SIP weirdness). These days, due to the misguided narrative that NATs are security devices, random port selection dominates. Certainly it does in linux (which accounts for most home routers), and all the commercial vendors (Juniper, Cisco, etc.). Some of the latter might let you configure it if you have esoteric needs, I haven't dug into that. Tailscale just assumes that port allocation is random.

This is a really nice explanation! I was also considering the birthday paradox to guess through a hard nas. It seems that the cgnat I'd like to traverse uses certain port ranges which could ease the guessing (I've read about nat traversal helpers to predict the ip address and ports). It would also be nice to do some research on NAT ALGs, it would be cool to fake a FTP connection to open an incoming port on the NAT gateway!

Natural follow up question - does NAT traversal actually work?

I've tried to replicate the NAT traversal paper[1] a couple of times in python, but it never really got it to work.

Maybe my ISP's network topology just inlcudes multiple NATs which is messing things up.

[1] https://bford.info/pub/net/p2pnat/

author here. NAT traversal definitely still works, but it's very situational. I'd estimate the simple techniques get you successful p2p in 90% of cases, but those 90% are unevenly distributed: for a particular pair of endpoints, the success rate is either 100% or 0%, and it's little consolation to the 0% folks to know that they're the exception :)

The paper you linked seems to have reasonably modern techniques for UDP traversal, so it should work. One thing that paper doesn't make clear, IMO, is that you need to STUN on both ends to discover your WAN ip:port, and you need to run the STUN traffic from the same socket as the one you're using for the NAT traversal.

If you have Tailscale, you can run `tailscale netcheck` to characterize your network a little bit, which might give you some answers as to why the traversal doesn't work. Successful traversal depends on the properties of both endpoints, so you'd want to netcheck from both ends. The usual culprit is MappingVariesByDestIP=true, which indicates a hard NAT.

Thanks for the insider info.

I didn't run a STUN server, but I exchanged the WAN ip:port via a server and some simple TCP messages.

Yes, it works quite well.

Especially if the port prediction part is driven by a dedicated "rendezvous" server rather than delegated to the clients to do. This plus taking care of the long tail (weird NAT combos) and adding a TCP traversal as a fallback used to get you a success rate of 95-97%.

Has anybody here ever used something like XMPP as a sidechannel? Sounds interesting and I kind of want to try setting up something really silly just for fun.

As I said in the article, many years ago I played with using XMPP as the side channel for NAT traversal software. XMPP was handy for this because you can just define your own extension that chat programs will ignore, but XMPP itself will still carry those extensions end to end. The idea I had at the time was to have the p2p software signed into your XMPP account, and it would use that as a federated message bus between all your devices, and the devices of your friends.

Honestly, the idea only made sense back when Google Talk ruled the earth, because it gave XMPP a network effect I could utilize. Sadly, since then, chat has fallen into a dozen different silos, so the "piggybacking" benefits are zero. And if you're not piggybacking on an established network, you can build a side channel protocol that's much simpler than what you'd make with XMPP.

Thanks for the explanation! XMPP may not be the flavor du jour but it's got the old gears turning in new ways...

If I were trying this again today, I'd probably poke at what Matrix can do in terms of non-user-visible messages. It seems to be a successor to XMPP in terms of the communication model, so if it can do broadcast+unicast of non-visible messages, and supports sending to yourself as well as others, it might be a good modern replacement!

I'm actually just now gearing up to homelab a Matrix set up! I'll keep this in mind.


There's no special treatment, and we've banned you more than once for using multiple accounts to spread this false claim.

Edit: actually, neither submission of this article made HN's front page. I'm reupping this one (using the process described at https://news.ycombinator.com/item?id=11662380 and the links back from there) because it's obviously a solid, substantive article. No, that's not special treatment, because we do it for a wide assortment of such articles when we see them. There's a partial list at https://news.ycombinator.com/invited, if anyone wants to see what I mean by wide assortment.


User vote data is personal and private to those users. Of course we don't publish it.

It wouldn't help anyhow. Long experience with imaginary accusations shows clearly that more data doesn't persuade anyone to drop their fixed idea. On the contrary, it just spawns even more accusations.

(In case anyone's wondering, the reason I post these replies is to reassure the general reader, who might be concerned whether there's anything to this, that there isn't anything to this.)

You're lying again since this post is obviously in the homepage right now and was in the 2nd page since yesterday which is suspiciously much longer than any post would stand generally.

Wouldn't be awesome if everyone was friends with you? we all would have been millionaires by now. Everybody is equal here but those who have 100% hit rate to the homepage were obviously just born lucky, right?

The post is on the front page now for the reason I explained upthread: https://news.ycombinator.com/item?id=24246018.

Please stop this now.

So HN rediscovers NAT traversal yet again...

Please don't post unsubstantive comments, and especially not supercilious ones, which are poisonous in addition to noise.

If you know more than the rest of the community, it would be great to share some of what you know so we all can learn. If you don't want to do that, that's fine, but then please don't post.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact