This is not due to congestion control, but because of the ordered byte stream semantics of TCP. You can't deliver a packet until the retransmission dance is done for any previous lost packets.
As an aside, these days it's worth noting that a 3G, WLAN or 4G link layer will try very hard not to drop any packets. So when you're doing UDP, watch out for those 40000 millisecond old packets that will burst to your lap once the rain cloud moves away from the cell tower or your co-worker's cocoa blings in the microwave.
It sounds like it would be a waste to tunnel it in UDP, which is the alternative...
(I'm not sure how much of the redundancy is shaved off--does anyone know if WebRTC's tries to encapsulate "normative" SCTP, or whether it uses SCTP without checksum/port information?)
There is also QUIC protocol also by Google, which is yet another crack at solving the problem. Wonder if those 2 groups sat down and discussed merging or re-using code.
This describes 95%+ of home users. NAT may do bad things to IP, but if you are an app developer that depends on no NAT, you're product is going to have a very limited market of people that can use it.
Or are you predicting the rise of IPv6 NAT?
Teredo is a Microsoft-developed (but IETF-standadized) NAT-traversing IPv6-over-IPv4 tunneling method.
In our setup, NAT's done on a GNU/Linux machines in a simplest possible way, almost like `iptables -t nat -A POSTROUTING -s 10.0.0.0/8 -o eth1 -j MASQUERADE` (we offer dynamic globally-routeable IPv4 addresses first, but when our pools are drained, we have no choice but do NAT for v4). So, it seems, almost every ISP who uses software routers (they're stable, performant and cheap) should be SCTP-friendly.
No idea about Ciscos and alikes, though. None we have does NAT, and given I'm not a Cisco guy I'm not going to reconfigure one just to test things.
Linus should just make STCP NAT support the default with a comment "router manufacturers - please don't disable", so the next time router vendors update their kernels, it gets built into.
Then just wait for 3-4 years.
SCTP has had 14+ years - SCTP made standards track in 2000 (rfc2960) and was in use before that. Chicken and egg.
This is why NAT is bad. It's has introduced a systemic freeze in the Internet, breaking the inert router/smart host model that used to enable deployment of new protocols.
On the other hand, STCP NAT support is just one kernel config option away and no further configuration's needed. A complete no-brainer - just persuade those building routers' firmware to not tick it off.
So yes, selective acks preserve network bandwidth but don't solve the real time problems. With UDP the client application can receive all packets when they arrive even if there are some missing.
Isn't that a good thing?
I still play a game that's almost 20 years old (the original Descent), which has a small but active multiplayer community . I regularly play against players in my own house as well as players from as far away as Brazil, Qatar, and Australia -- so I have to deal with various connection consistency issues. Given the game physics, I would much rather have out-of-order packets (resulting in an opposing ship briefly flashing in a wrong location and firing a shot from there) than having a delay on in-order packets (resulting in an opposing ship completely freezing, and then completing several seconds of movement and several seconds of shots in a fraction of a second).
There are other games that behave differently, where TCP would be preferable.
 descentrangers.com for anarchy, cooperative, and team games; descentchampions.org for competitive 1v1s.
If your protocol is stateless, you might choose to lose the data, because you know that the loss doesn't matter as soon as you get a new update. This might be applicable in a games, if the state you need to synchronize (say, position) is quite small.
Even if you need your protocol to be stateful: you might choose to design your protocol such that loss results in a temporary degradation in the client view of state, but which will converge back on the correct state as it receives new data. This is a very common pattern; the quake net protocol referenced in OP is an example, as are most streaming A/V codecs.
If you absolutely cannot tolerate the loss of any data, and are also latency sensitive, you can also choose to add redundant data so you can recover the lost data without retransmitting.
And you can make intermediate trade-offs: A little bit of extra overhead for error-correction, and in the now-rare cases where error rate overwhelms your correction ability, you accept a degradation / retransmit round-trip. Or you accept a certain amount of degradation before you need retransmission to recover. Etc.
We use TCP for sending real-time vessel track data to the central correlation servers, because they need to know if the connection has dropped and change their behavior accordingly. Traffic volume is low, so congestion throttling and re-tries are not a problem. The system is able to cope with reports being delayed and then coming in much later (and catching up) because reports are time-stamped and the correlator works in fuzzy 6D phase-space.
We use UDP for sending live radar data because, frankly, if you've missed one sector of data you really, really don't want to delay subsequent sectors while waiting for the data that will be irrelevant in 2 seconds anyway. The compression and encryption schemes are specially crafted to allow for lost packets while still using solid implementations of off-the-shelf algorithms where appropriate, and lovingly hand-crafted algorithms for the non-security critical aspects.
And I can't tell you what we use our locally defined reliable UDP for.
But it doesn't seem to talk about the virtues of congestion control - the primary one being it prevents the collapse of the Internet. Reacting to packet loss by just trying harder results in nothing but a network filled with doomed packets. This is not theoretical - before Van Jacobson that was the Internet.
I'm not defending the status quo nor TCP - TCP has deep problems figuring out what is really loss. and it speeds up too slowly but still doesn't manage to slow down when it should resulting in induced delay. But the mere fact that is worried about congestion (not just unreliability - one result of congestion) and sharing the channel isn't its problem.
have a look at QUIC, ledbat, Minion and the recent IETF TAPS BoF for work going on improving transport options in these areas. Its just the wrong takeaway to read that article and say "UDP is better because it doesn't have congestion control" - you need to consider CC in any reliable UDP based transport you roll too.
rfc 5405 provides some advice (http://tools.ietf.org/html/rfc5405) - its a little dated but still useful.
> Because congestion control is critical to the stable operation of the Internet, applications and upper-layer protocols that choose to use UDP as an Internet transport must employ mechanisms to prevent congestion collapse and to establish some degree of fairness with concurrent traffic.
It's a shame that the internet requires this kind of good behaviour from individual hosts, because it leaves it vulnerable to buggy or malicious nodes. But as long as it does, it's essential that users of UDP know about this.
(Or at least enabled it for ~25 years until NAT put a damper on it - let's all keep our thumbs up for IPv6).
TCP is really pretty reasonable. Fast retransmit/fast recovery lets TCP eat up small packet losses without entering congestion control. Plus everybody recently upped their intial congestion window to 10.
Also, there was an attempt to deploy ECN (explicit congestion notification) for TCP, but net equipment vendors and providers resisted deploying it. It was specced as an official standards track TCP feature in 2001.
There are smaller problems (but still worth fixing), like TLS over TCP doing a few more round trips more than necessary. QUIC tackles this in particular.
Current radio link layers (3g, wifi) are designed to play nicely with TCP. Their below-IP layers do their own retransmissions to make up for radio link issues. They have to take into account radio channel congestion - it doesn't help if everyone on the channel just starts shouting louder and repeating themselves.
In the virtual world platform my team wrote, we started using UDP exclusively, then moved to mixed TCP and UDP, then moved to using TCP for everything except P2P voice connections. with UDP I had always dreaded going into see a potential corporate client, and not knowing if my networking was going to make it through the firewall.
This is a good discussion of the subject. http://stackoverflow.com/questions/992069/ace-vs-boost-vs-po...
That is : One can easily rewrite a TCP-like on top of UDP.
This isn't baked into the protocol, but in practice everyone relies on it.
We've been poking UDP holes in firewalls for 5 years; its the best (most frequently successful) option for enterprise by far.
Failures to communicate on the same subnet with UDP are the wellknown 'hairpin' issue, where some routers will not recognize their own outside IP address and short-circuit a packet back to another subnet inside their domain. Not a lot you can do about it, except use TURN or another UDP proxy.
One of the cute things it did was improve reliability for critical messages by preemptively sending them twice, in successive datagrams. That allowed the protocol to avoid an ack-miss-retransmit cycle for those packets, as long as it was only a single datagram that was dropped.
we're seeing it come back into vogue in TCP and modern protocols. QUIC has a FEC component and TCP Tail Loss Probe (http://www.ietf.org/proceedings/84/slides/slides-84-tcpm-14....) is a subtle variation on the theme too
the million dollar question around FEC is how correlated are different loss events which has a lot to do with how well it can work.
But, yes, sending duplicates in another datagram does allow errors to be corrected without being reported to the sender, so it corrects errors in a forward direction, which i suppose makes it FEC.
That Google tail loss stuff is nice, thanks for the link. It's cool that thirty years after RFC793, we're still making significant improvements to TCP.
> Our solution was simple and surprisingly effective. Every packet would send copy of the last packet. This way if a packet were dropped, a copy of it would arrive with the next packet, and we could continue on our merry way. This would require nearly twice as much bandwidth, but fortunately our system required so little bandwidth that this was acceptable. This would only fail if two consecutive packets were dropped, and this seemed unlikely. If it did happen, then we would fall back on the re-sending code.
The protocol i remember specifically send some parts of datagrams, the most critical messages, twice.
"Hi, I'd like to hear a TCP joke."
"Hello, would you like to hear a TCP joke?"
"Yes, I'd like to hear a TCP joke."
"OK, I'll tell you a TCP joke."
"Ok, I will hear a TCP joke."
"Are you ready to hear a TCP joke?"
"Yes, I am ready to hear a TCP joke."
"Ok, I am about to send the TCP joke. It will last 10 seconds, it has two characters, it does not have a setting, it ends with a punchline."
"Ok, I am ready to get your TCP joke that will last 10 seconds, has two characters, does not have an explicit setting, and ends with a punchline."
"I'm sorry, your connection has timed out. Hello, would you like to hear a TCP joke?"
Things that challenged us:
64K message size limit (varied slightly from linux/solaris/hpux)
you can't tell if the other side is listening or got the message.
multicast requires use of certain ip ranges which people seem to forget frequently.
I had the UNIX Network Programming by W. Richard Stevens tome I borrowed from my boss. Good for networking.
But the good news is, you CAN do those things yourself. And avoid the TCP lockups, slowdowns and endless retransmissions.
as an aside, its generally a mistake to use IP-Size > PMTU (commonly ~1500).. the result is IP fragmentation and IP stacks commonly have very limited resources devoted to IP reassembly (otherwise its a DoS attack) - so its very easy for your fragmented UDP message to get "lost" even when the network transport does not have an error.
When it seems to work ok its basically because your app is the only one doing it.. if it were common practice you would all fight for the same reassembly buffer space and the OS would have to start dropping things. Your application also becomes trivially dosable at low bandwidths by an attacker intentionally using up the reassembly buffer.
Yes, firewalls can be configured to restrict communications. And you can have a combined NAT+firewall that is so configured.
Just likes routers vs firewalls. Routers forward packets as well as they know how, firewalls selectively drop them, router-FW combos can configured either way.
Reliance on restrictive centralized firewalls is a pretty 1990s mindset, and doesn't lead to good security outcomes in the current world where users constantly suffle devices between networks, and vpn in to your network, etc. You just end up making your internal network unusable for production work.
I suppose if you view NAT to "facilitate communications" then mapping a port for all inbound IPs instead of symmetric makes sense.
Also, these days with fast retransmit TCP isn't as bad as it used to be when there are network problems.
I've been using TCP for over 20 years for multi-player games, VOIP and web conferencing, and it works incredibly well. Even if you do all the legwork to get UDP working reliably (which TCP give you for free), you still have the problem of firewalls.
Wireless still has interesting(?) packet-loss behaviors. It can lose single packets, bursts of packets or even too-large packets (MTU exceeded). It depends on the AP and your local radio noise characteristics.
Simple, yet robust.
I'll defend UDP.
UDP is the honey badger of the internet protocol suite.
UDP is all about the transaction. UDP is standing on a cliff yelling, "Come at me, bro", whether you're there or not.
UDP is a man's protocol doing real shit like bootstrapping your ass and slapping an IP on you. Get up, motha fucka!
UDP will talk shit to one of you or all of you. UDP ain't scared. UDP brought the fear.
UDP understands that you may be slow sometimes. So UDP will wait for your sorry ass. UDP grew up without a father, too.
UDP sends a message and couldn't give a fuck if you got it or not.
UDP got a message from you saying that you got his messages and guess what? UDP didn't even open it! Not one fuck given.
Don't try to shake UDP's hand! You crazy?
And, when UDP dies because you weren't available, UDP doesn't shed a tear. UDP is hardcore. He's going out even if he knows you ain't there. UDP is a goddam one-man slaughter house. Why?
Because UDP doesn't give a fuck.
I mean if you want to ship a product. If your goal is to learn networking, then by all means, roll your own.
I am surprised that UDP is not used more often when the business cost of loss data is small.
The TCP fallback only consistently works well (in terms of QoE) in cases in which the TCP legs are short.
 http://www.ietf.org/rfc/rfc3093.txt (funny, an April 1 RFC applicable to a serious post on April 1)
Advantage of STRAW over other reliable-udp protocols: multiple connections with a single port/NAT pinhole. Multiple channels per path.
Since the signaling is not part of the standard, on purpose, you can choose whatever you like (XMPP, SIP...) or create you own solution.
On the server side, TURN is implemented to go through the NATs, the firewalls, that helps.
Yup, it's April Fools on the Internet, and everyone's a comic genius!
Socket servers are so 80's.
For example, the multiplayer version of asteroids in this article uses TCP sockets:
But what would this magical protocol do differently that would make it work better?
In my opinion, most of TCP's semantics arise not out of the network, but rather the data itself. I can't have packets getting lost in the middle of an SSH session: it just doesn't make sense. My keystrokes are a stream of data that must be in order, and must be delivered: thus TCP.
I commute daily, and use 4G on my commute. Another poster has problems very similar to mine:
> As an aside, these days it's worth noting that a 3G, WLAN or 4G link layer will try very hard not to drop any packets. So when you're doing UDP, watch out for those 40000 millisecond old packets that will burst to your lap once the rain cloud moves away from the cell tower
You can see this just by pinging 220.127.116.11 in the background. The loss of good signal will result in many packets getting dropped, but then when signal resumes, ping will receive the packets that it had presumed lost, and you'll get things like:
64 bytes from 18.104.22.168: icmp_seq=65 ttl=46 time=37324.3 ms
64 bytes from 22.214.171.124: icmp_seq=66 ttl=46 time=36324.3 ms
64 bytes from 126.96.36.199: icmp_seq=67 ttl=46 time=35324.3 ms
64 bytes from 188.8.131.52: icmp_seq=68 ttl=46 time=34324.3 ms
... and so on ...
For all the ads make you think speed is the determining factor in who has the better 4G, I'd say simple packet loss or connectivity loss makes much more of a difference to me day-to-day. "Coverage", you might call it, except typically according to the phone there is a signal, it is 4G, but the strength is just so poor as to be unusable. I use an app to get this info (the bars in the corner just aren't fine-grained enough), and it reports things like "Net. type: NSDPA * 7.2 Mbps", and that'd be okay, except: "Net strength: -99 dBm * 7 ASU" — too low for connectivity; in my experience, I require >-80 dBm for actual data to transmit successfully. (Ping packets to round trip, etc.)
Mosh is basically (a better) SSH over UDP. It fares a lot better than SSH on mobile connections. It does away with hanging connections and such nonsense.
Where a lost packet did not matter, there would still be a new player position packet a half a second later.
Then for chat and score and stuff like that tcp was used.
Mixing reliable and unreliable updates for game logic would seem to result in a lot of complexity as things can be out of sync in a variety of ways now.
"His next iteration involved using UDP with both reliable and unreliable data, pretty much what many would consider a standard networking architecture. However standard mixed reliabled/unreliable implementations tend to generate very hard to find bugs, e.g. sequencing errors where guaranteed messages referenced entities altered through unreliable messages."
Yes, but UDP is choosing unreliable transport, not throwing away data entirely. No matter the transport, once you start dropping a large enough number of packets it will degrade (or break) the experience. The use case being discussed is when getting the next packet fast is more important than the overhead of TCP. You can build retries or various other forms of reliability into your UDP protocol when outdated information is still useful.
Mixing reliable and unreliable updates for game logic would seem to result in a lot of complexity
Yes and no. It's complex because game state is complex to begin with, but less daunting than you may think because you can compartmentalize to different components of the game. You're not likely to using multiple transports for the same information. See the parent's chat vs player position example.
If you have cool transit providers, you can have them place upstream filters to block UDP. That might be a pipedream though, I don't know of any large providers who will do that for you these days.
Kids these days.
We all say essentially the same, at different levels of complexity. I have a live demo :) But you should read all three to get a complete picture.